+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190505/A1c --output-run DEEP/20190505/A1c/science#step0/20240309T144841Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.7066 seconds; current memory usage: 0.179 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0186 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0252 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/pipeline_orig.qgraph DEEP/20190505/A1c/science#step0/20240309T144841Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 7.8254 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 24.8262 seconds; current memory usage: 0.338 Gibyte, delta: 0.159 Gibyte, peak delta: 0.146 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.338 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.4019 seconds; current memory usage: 0.338 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.338 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 5022 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190505_A1c_science#step0_20240309T144841Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 1.2658 seconds; current memory usage: 0.338 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.338 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 5024 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 1.6175 seconds; current memory usage: 0.342 Gibyte, delta: 0.004 Gibyte, peak delta: 0.004 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.342 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 03875176-889f-4a45-b542-3da436024a9c parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-09 14:49:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-09 14:49:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 14:49:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-09 14:49:49 proc_lsst.multi:170 [INFO] [multi] job_id 77954 proc_lsst.multi INFO: [multi] job_id 77954 2024-03-09 14:49:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:49 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 77954 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 77954 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 77954 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash f92a5490ba85f93f00f0233eed719f9c parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8418eec2-61a0-4f28-944b-279b9a4686a5_isrForCrosstalkSources_855449_46.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8418eec2-61a0-4f28-944b-279b9a4686a5_isrForCrosstalkSources_855449_46.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 20680f1d85fa696d0fabc4bf60cee66f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/1a958d77-8bf9-425a-86d4-802881926d34_isrForCrosstalkSources_855390_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/1a958d77-8bf9-425a-86d4-802881926d34_isrForCrosstalkSources_855390_35.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash a2089c2fd78b9627b6d8c87769f35e9f parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3f410c72-2667-4e61-a5b7-3cc997248cf6_isrForCrosstalkSources_855448_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3f410c72-2667-4e61-a5b7-3cc997248cf6_isrForCrosstalkSources_855448_6.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 4028de1093ff467cfe5770721655bddf parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/83fc7c42-4a60-4eeb-a50c-32a027ff8897_isrForCrosstalkSources_855413_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/83fc7c42-4a60-4eeb-a50c-32a027ff8897_isrForCrosstalkSources_855413_37.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 19034cc880f5b1c8da06eb82e7068765 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d783b8e7-8d12-4ef3-a53e-458f878d0843_isrForCrosstalkSources_855457_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d783b8e7-8d12-4ef3-a53e-458f878d0843_isrForCrosstalkSources_855457_41.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash d3a268b34276d8aaf369b59f5c3b3ed0 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b009a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d090e80f-fc19-4680-bd1e-73fdbc2b4be0_isrForCrosstalkSources_855383_44.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d090e80f-fc19-4680-bd1e-73fdbc2b4be0_isrForCrosstalkSources_855383_44.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 7b8a736e8e3bfa2947557c599212857f parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a894fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1919a9dc-5e4a-44e9-b39e-c67fa67658c9_isrForCrosstalkSources_855419_40.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1919a9dc-5e4a-44e9-b39e-c67fa67658c9_isrForCrosstalkSources_855419_40.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 828f3c0d5d891b2cc91724ff364d6d4a parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2b7c3f03-16c4-4095-b008-99548cabfb66_isrForCrosstalkSources_855384_22.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2b7c3f03-16c4-4095-b008-99548cabfb66_isrForCrosstalkSources_855384_22.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash f6da97dcb2c71c9418f8530362d72a0f parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/eedc41b2-f0b4-47dd-a978-0db5d08ec0f5_isrForCrosstalkSources_855433_30.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/eedc41b2-f0b4-47dd-a978-0db5d08ec0f5_isrForCrosstalkSources_855433_30.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash d97668dbfac40b0802e9bc2ddd51902c parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e1afd0c2-8ab6-437b-82d7-7b1ef48be40a_isrForCrosstalkSources_855423_39.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e1afd0c2-8ab6-437b-82d7-7b1ef48be40a_isrForCrosstalkSources_855423_39.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash aad47b592411aeb101ffac4274c6e8c1 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c7366544-d8e4-400f-bdd8-51d2e1737f90_isrForCrosstalkSources_855404_13.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c7366544-d8e4-400f-bdd8-51d2e1737f90_isrForCrosstalkSources_855404_13.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a619fa75e14eba3371f32b6358b829e8 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/08c15e73-0ab4-4038-a094-1a47f9c2fe81_isrForCrosstalkSources_855422_7.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/08c15e73-0ab4-4038-a094-1a47f9c2fe81_isrForCrosstalkSources_855422_7.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash e954c7d7c1f32e4314d01139a7e43bb0 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9c626707-fa5b-4831-adc6-1711699c3e9d_isrForCrosstalkSources_855397_51.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9c626707-fa5b-4831-adc6-1711699c3e9d_isrForCrosstalkSources_855397_51.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash e010b781bfdbe137c035fe8fbb1a23f6 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e0a56c3a-b340-4d97-b6e4-1f764cfb1362_isrForCrosstalkSources_855411_56.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e0a56c3a-b340-4d97-b6e4-1f764cfb1362_isrForCrosstalkSources_855411_56.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 06dc7b49a309a692358364024e552dec parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5f608c1d-46df-4c43-8926-f85071c81e32_isrForCrosstalkSources_855381_43.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5f608c1d-46df-4c43-8926-f85071c81e32_isrForCrosstalkSources_855381_43.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 49c8bdf22e83efa7c6225fdaea4e7b60 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b00fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/3acda221-f75d-4abb-95c4-3ed31777aa7d_isrForCrosstalkSources_855406_38.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/3acda221-f75d-4abb-95c4-3ed31777aa7d_isrForCrosstalkSources_855406_38.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash e78e00c796502efe34f04a4d6442b96f parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b014e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/74415227-741f-456c-b042-6f029a0cd4ef_isrForCrosstalkSources_855395_57.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/74415227-741f-456c-b042-6f029a0cd4ef_isrForCrosstalkSources_855395_57.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash fbff304fe8b9a0e83ef6b13462759d01 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b016c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/edceabb3-7ed1-42d7-9c60-c854ce19697f_isrForCrosstalkSources_855410_3.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/edceabb3-7ed1-42d7-9c60-c854ce19697f_isrForCrosstalkSources_855410_3.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 6a363909203120fd680aaea16780b77b parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c36c11e5-9e6c-4b04-8ba4-cf94262ba824_isrForCrosstalkSources_855435_7.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c36c11e5-9e6c-4b04-8ba4-cf94262ba824_isrForCrosstalkSources_855435_7.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 58ee2f46733c51b3e4017dc24cae0a0e parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81b82c2d-eb84-43d1-80e9-594431a9e1ce_isrForCrosstalkSources_855398_22.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81b82c2d-eb84-43d1-80e9-594431a9e1ce_isrForCrosstalkSources_855398_22.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 29ca7a996b9242ca171f19f4db28042e parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0595f7d5-6cf7-4329-b490-de818f0e6743_isrForCrosstalkSources_855385_45.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0595f7d5-6cf7-4329-b490-de818f0e6743_isrForCrosstalkSources_855385_45.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 07c1b6c6164878cb8c890b745f37eb0c parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/51dbf996-c0ef-4405-9892-e7bb4dff6053_isrForCrosstalkSources_855402_16.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/51dbf996-c0ef-4405-9892-e7bb4dff6053_isrForCrosstalkSources_855402_16.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash c1b682dce7ac1f56bbba0281a9a6ffff parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/96dbfa9e-0797-4897-bf95-90d45c783bb4_isrForCrosstalkSources_855443_24.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/96dbfa9e-0797-4897-bf95-90d45c783bb4_isrForCrosstalkSources_855443_24.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash b671f59e43654371eb78ce40983607fb parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3f1534a1-b5d9-4315-9567-b16ebfade7e1_isrForCrosstalkSources_855454_43.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3f1534a1-b5d9-4315-9567-b16ebfade7e1_isrForCrosstalkSources_855454_43.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash c6baa5878ae5e23ac3801548df319308 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fa9abeac-08f3-469f-bae0-af58c9a34b65_isrForCrosstalkSources_855413_27.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fa9abeac-08f3-469f-bae0-af58c9a34b65_isrForCrosstalkSources_855413_27.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 07571ba26a1f4790b9b95aefce29b7b6 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d5c7e745-5541-42c6-99e6-f5a2a1e16a41_isrForCrosstalkSources_855435_1.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d5c7e745-5541-42c6-99e6-f5a2a1e16a41_isrForCrosstalkSources_855435_1.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 4f134c616a06e86b177b41be31bdd868 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c9477436-68f7-4461-ac8b-2d4ece8edd6e_isrForCrosstalkSources_855436_28.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c9477436-68f7-4461-ac8b-2d4ece8edd6e_isrForCrosstalkSources_855436_28.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 237014cbbc13ee69866655ed3aaa3def parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bb182782-bb17-4a0f-8077-1ae22fd4b004_isrForCrosstalkSources_855412_38.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bb182782-bb17-4a0f-8077-1ae22fd4b004_isrForCrosstalkSources_855412_38.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 007729bd26d58ff047a432ccb6ae358d parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c3e45e64-c049-4821-bab6-8138e817bf71_isrForCrosstalkSources_855415_62.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c3e45e64-c049-4821-bab6-8138e817bf71_isrForCrosstalkSources_855415_62.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 2ac2abdc54376b82c28d2837b41e592f parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec36a123-bad2-42ec-81f8-7ed94362e912_isrForCrosstalkSources_855391_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec36a123-bad2-42ec-81f8-7ed94362e912_isrForCrosstalkSources_855391_2.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash b8c110d3963cc2b894c8cecd2af7cd59 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/879884ae-8d84-4871-8528-cd8ca1babd05_isrForCrosstalkSources_855430_47.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/879884ae-8d84-4871-8528-cd8ca1babd05_isrForCrosstalkSources_855430_47.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 8fe2fc251e67a4c1872303622a2b550b parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e1133b7d-3099-40e3-8428-9bfac04ef96a_isrForCrosstalkSources_855441_1.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e1133b7d-3099-40e3-8428-9bfac04ef96a_isrForCrosstalkSources_855441_1.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 79430137afdfa2218b77f3c591babd8f parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/bbf8b392-9510-4b8b-b20e-40282fb0a452_isrForCrosstalkSources_855415_26.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/bbf8b392-9510-4b8b-b20e-40282fb0a452_isrForCrosstalkSources_855415_26.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash c646d30df5c5abaf961a9b7fd25ad71a parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/59be1e37-9104-4258-a842-13482fb6ee07_isrForCrosstalkSources_855415_19.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/59be1e37-9104-4258-a842-13482fb6ee07_isrForCrosstalkSources_855415_19.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash cb46a85be4c13cb079d64d11ccde07b8 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b020c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f653bc54-0004-4c21-a524-0df884431f99_isrForCrosstalkSources_855409_52.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f653bc54-0004-4c21-a524-0df884431f99_isrForCrosstalkSources_855409_52.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash ecb2ce2b8b02a4a77847e9c3fb4dc3ce parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a6b6e051-f44c-48fa-b78c-4e40926eb8e8_isrForCrosstalkSources_855455_57.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a6b6e051-f44c-48fa-b78c-4e40926eb8e8_isrForCrosstalkSources_855455_57.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 28cd26354445bc8b7191785dca445449 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5e67b706-c429-4fbf-922d-c84b84ef02d6_isrForCrosstalkSources_855439_36.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5e67b706-c429-4fbf-922d-c84b84ef02d6_isrForCrosstalkSources_855439_36.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 053ca3e76cfce1c2f7cca2203fc7273f parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b022a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bcca0090-4ea9-42b4-8235-338868fd6b34_isrForCrosstalkSources_855396_9.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bcca0090-4ea9-42b4-8235-338868fd6b34_isrForCrosstalkSources_855396_9.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash c3455ea119eef5598ad5738cec7640bf parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/5d8dd0ac-8162-4917-a416-284334efb83a_isrForCrosstalkSources_855437_12.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/5d8dd0ac-8162-4917-a416-284334efb83a_isrForCrosstalkSources_855437_12.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 12ea10079c88793914b09285ac546bbd parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/711ffe4c-604e-448c-aec3-4bb98b825c98_isrForCrosstalkSources_855382_17.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/711ffe4c-604e-448c-aec3-4bb98b825c98_isrForCrosstalkSources_855382_17.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 12d6b1ea8af0d065a1b54c0addd0712a parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/94fc10d9-9397-4d33-90dc-77f9b839975a_isrForCrosstalkSources_855434_54.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/94fc10d9-9397-4d33-90dc-77f9b839975a_isrForCrosstalkSources_855434_54.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash c0622f94ab343f670f9e5d1883a30ca2 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bdbb9eb5-e48d-479d-95e7-382d943fc3ff_isrForCrosstalkSources_855420_12.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bdbb9eb5-e48d-479d-95e7-382d943fc3ff_isrForCrosstalkSources_855420_12.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 52572a212fb501cbf9715e089c73061c parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/57f1a0fe-e744-4673-99c9-b0dc2ccf4d6f_isrForCrosstalkSources_855395_47.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/57f1a0fe-e744-4673-99c9-b0dc2ccf4d6f_isrForCrosstalkSources_855395_47.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 4da9ba1f2e7c59cb7d761a09a4c2075d parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/11015b11-2f75-4d0b-a02c-f2759bff18d2_isrForCrosstalkSources_855417_15.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/11015b11-2f75-4d0b-a02c-f2759bff18d2_isrForCrosstalkSources_855417_15.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7d5d1b12ca3e4997bf5f1c2bfeb32d04 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7ee20a5b-c053-42fe-8506-1ed792996921_isrForCrosstalkSources_855446_6.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7ee20a5b-c053-42fe-8506-1ed792996921_isrForCrosstalkSources_855446_6.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash fc056b43b1f14c3c4c3595e2a0083772 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/676d8b5d-b321-473f-a231-dc3001974725_isrForCrosstalkSources_855406_12.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/676d8b5d-b321-473f-a231-dc3001974725_isrForCrosstalkSources_855406_12.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash d05f7d005740cf4541525585fd552071 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3bb82c5f-8e7c-45ba-a91e-eb517e84535e_isrForCrosstalkSources_855387_46.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3bb82c5f-8e7c-45ba-a91e-eb517e84535e_isrForCrosstalkSources_855387_46.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 29224622adb208294893572e298633f5 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0c2a7950-45ff-47b5-a565-20873c31c3a5_isrForCrosstalkSources_855425_29.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0c2a7950-45ff-47b5-a565-20873c31c3a5_isrForCrosstalkSources_855425_29.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 7f8698971eadf711f67f44ff57637171 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0245fed-668e-45ac-be65-a03f3331d3d0_isrForCrosstalkSources_855388_24.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0245fed-668e-45ac-be65-a03f3331d3d0_isrForCrosstalkSources_855388_24.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash a09740dce4ab9860819ba967fdd5e43e parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cece0a50-f82b-4939-bffd-6a69d4d9b76f_isrForCrosstalkSources_855409_21.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cece0a50-f82b-4939-bffd-6a69d4d9b76f_isrForCrosstalkSources_855409_21.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 8e3be45bcd49934e89d9ae5f8f013141 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8a36f9d7-cf0d-4782-8853-5ce0c1f5453c_isrForCrosstalkSources_855406_25.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8a36f9d7-cf0d-4782-8853-5ce0c1f5453c_isrForCrosstalkSources_855406_25.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash a0ea67d43bc5109ee5afd7030e52db2c parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3ad48710-90a3-429c-b3b7-13367f656c83_isrForCrosstalkSources_855457_53.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3ad48710-90a3-429c-b3b7-13367f656c83_isrForCrosstalkSources_855457_53.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 34d38d17f363013118cd1a43b093f349 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ea27ad5f-2308-4707-9d89-11196a52c900_isrForCrosstalkSources_855416_47.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ea27ad5f-2308-4707-9d89-11196a52c900_isrForCrosstalkSources_855416_47.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash cebfa6ca5ecc85c888cb9eadd0a7b6e7 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a59e67f1-9f14-4bf6-b15e-d688a2f6219e_isrForCrosstalkSources_855434_15.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a59e67f1-9f14-4bf6-b15e-d688a2f6219e_isrForCrosstalkSources_855434_15.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash caa393a7c1230f88f7d0a726d4eaf725 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fe4ada47-c2c7-41a6-a999-ec317e9e7414_isrForCrosstalkSources_855442_13.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fe4ada47-c2c7-41a6-a999-ec317e9e7414_isrForCrosstalkSources_855442_13.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash c374d6c954dbcef0194d19af79fae844 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/12b9c846-6d82-458e-bf47-24938027d510_isrForCrosstalkSources_855457_35.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/12b9c846-6d82-458e-bf47-24938027d510_isrForCrosstalkSources_855457_35.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 274cf1fe899dad91ae8740c29043781b parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/60c6f57c-5037-4d7b-8838-7445c24c159a_isrForCrosstalkSources_855428_32.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/60c6f57c-5037-4d7b-8838-7445c24c159a_isrForCrosstalkSources_855428_32.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash a25a0013227ee5ddd2cb4674c08c8f6e parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/10687160-3f29-4068-9916-84e5fe45fe4c_isrForCrosstalkSources_855401_37.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/10687160-3f29-4068-9916-84e5fe45fe4c_isrForCrosstalkSources_855401_37.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 8f098efa7771d599dedb0a3c0263616b parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7bbcdcc6-f6b5-497d-a2c9-4b15c27f3590_isrForCrosstalkSources_855440_57.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7bbcdcc6-f6b5-497d-a2c9-4b15c27f3590_isrForCrosstalkSources_855440_57.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 3155f4930706de2d41805e85098f74f9 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b028e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/500cc5f7-b8ec-4148-83c2-379041637744_isrForCrosstalkSources_855430_49.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/500cc5f7-b8ec-4148-83c2-379041637744_isrForCrosstalkSources_855430_49.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash b140f164b696b1b3348d7c9c9b3b5d4d parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/341f4089-1e42-46db-91ca-288da28dcac6_isrForCrosstalkSources_855452_49.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/341f4089-1e42-46db-91ca-288da28dcac6_isrForCrosstalkSources_855452_49.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 270d48a0a1faf7d9cc758c9edf59f23a parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b822200b-c46f-41a4-bc29-45be0deb746c_isrForCrosstalkSources_855416_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b822200b-c46f-41a4-bc29-45be0deb746c_isrForCrosstalkSources_855416_27.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 5357a08326808915125e8222718a1952 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b027a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a0d75c98-36f0-45a6-99de-7c102f1cd47b_isrForCrosstalkSources_855441_6.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a0d75c98-36f0-45a6-99de-7c102f1cd47b_isrForCrosstalkSources_855441_6.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash c9355d5e45a724f3d12448266ca8bd7c parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/f0a22717-b76d-457a-9975-945f2361d5cf_isrForCrosstalkSources_855406_1.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/f0a22717-b76d-457a-9975-945f2361d5cf_isrForCrosstalkSources_855406_1.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash dd3f44417537f24c73de7423f764bd1e parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b025c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e6928c95-26c9-4943-9fe2-cb2045d2b78e_isrForCrosstalkSources_855450_6.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e6928c95-26c9-4943-9fe2-cb2045d2b78e_isrForCrosstalkSources_855450_6.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 415ac16c7b4d47a7681fae994f022777 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c9d0a1c6-1ad7-4eef-bc79-207ab5490a4f_isrForCrosstalkSources_855385_49.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c9d0a1c6-1ad7-4eef-bc79-207ab5490a4f_isrForCrosstalkSources_855385_49.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash f6c529506d8074def65ad4eb3e7df36b parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2e53abfc-2a33-4c1e-88cd-b8502deea102_isrForCrosstalkSources_855444_12.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2e53abfc-2a33-4c1e-88cd-b8502deea102_isrForCrosstalkSources_855444_12.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash e6e146ab17e7d0f054cdfbc5575a5642 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a8038f37-c6f7-45f1-bb0c-82ece3ea524b_isrForCrosstalkSources_855457_44.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a8038f37-c6f7-45f1-bb0c-82ece3ea524b_isrForCrosstalkSources_855457_44.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 02dd53f80392c0b742e7434e4bfe3926 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b023e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b89bcc1c-853a-4a69-b9bd-adb530f12e8a_isrForCrosstalkSources_855419_23.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b89bcc1c-853a-4a69-b9bd-adb530f12e8a_isrForCrosstalkSources_855419_23.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 36414fa91943bb2aed3ecf6472dee93e parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3e2556c6-e1c4-4d4f-bf73-8dfd0521177a_isrForCrosstalkSources_855382_15.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3e2556c6-e1c4-4d4f-bf73-8dfd0521177a_isrForCrosstalkSources_855382_15.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 283f9b8a6a337bf8a9ec70da15060eca parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b02340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e0b55214-6a28-47e9-aa43-e604f892428b_isrForCrosstalkSources_855443_36.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e0b55214-6a28-47e9-aa43-e604f892428b_isrForCrosstalkSources_855443_36.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 9e8ea2ad10ef1a4c04ae15eb44bcb8ec parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f76035d3-20ff-46f3-87ff-20ff80164a3e_isrForCrosstalkSources_855430_61.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f76035d3-20ff-46f3-87ff-20ff80164a3e_isrForCrosstalkSources_855430_61.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash e2e628d83db5bdc2fd7a66445b6f4070 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d294132c-4929-41bc-b2b6-7f868f627409_isrForCrosstalkSources_855433_43.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d294132c-4929-41bc-b2b6-7f868f627409_isrForCrosstalkSources_855433_43.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 3bd86a80eff622afba6731e9c0ea3a1e parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a7b154b2-51af-41f8-ab75-d5070c31723a_isrForCrosstalkSources_855448_31.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a7b154b2-51af-41f8-ab75-d5070c31723a_isrForCrosstalkSources_855448_31.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 975f113b7057d8c2bff36b6087e292bb parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/2903731f-694e-4190-b37f-52c355faf5a1_isrForCrosstalkSources_855454_53.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/2903731f-694e-4190-b37f-52c355faf5a1_isrForCrosstalkSources_855454_53.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash ecc156ea81fb568296dd3281239bbb56 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ec5ec741-aaf1-4f41-af89-bf88d52107ea_isrForCrosstalkSources_855400_6.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ec5ec741-aaf1-4f41-af89-bf88d52107ea_isrForCrosstalkSources_855400_6.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash e85a72a9e82f6dcaa051d0f227ece183 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/efd3de75-89f6-4171-a6cf-a0fc3c40f669_isrForCrosstalkSources_855406_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/efd3de75-89f6-4171-a6cf-a0fc3c40f669_isrForCrosstalkSources_855406_2.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash a75cef6ab6e51e7bd0e5696bac66d87b parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4f126955-6e46-4ec4-baea-2bac8a8d8c08_isrForCrosstalkSources_855421_61.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4f126955-6e46-4ec4-baea-2bac8a8d8c08_isrForCrosstalkSources_855421_61.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 92b26c28dc55b3ea9ffb7847e1f64c2f parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/65d887cc-b8a5-4910-b23b-7e765782b9a7_isrForCrosstalkSources_855446_44.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/65d887cc-b8a5-4910-b23b-7e765782b9a7_isrForCrosstalkSources_855446_44.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash ac4ba10e95c824de60ab63dc7363598a parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d6b6318-fe1a-4a0f-b255-f7a53b61b1c3_isrForCrosstalkSources_855400_54.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d6b6318-fe1a-4a0f-b255-f7a53b61b1c3_isrForCrosstalkSources_855400_54.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 8794e04b20985bff4c62ac4024e5b214 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/654c9797-6340-48de-b23e-dd1e696cbb3a_isrForCrosstalkSources_855379_23.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/654c9797-6340-48de-b23e-dd1e696cbb3a_isrForCrosstalkSources_855379_23.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 42a37b75a39b353fe60ba907160fe010 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/94890e15-b2c9-4d76-aad0-eaf87a691673_isrForCrosstalkSources_855379_54.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/94890e15-b2c9-4d76-aad0-eaf87a691673_isrForCrosstalkSources_855379_54.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 48279bb21fc60bdb6f4c5ece5423454e parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2d77db6f-b069-496e-a0cf-a2ae5b2a36cf_isrForCrosstalkSources_855404_52.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2d77db6f-b069-496e-a0cf-a2ae5b2a36cf_isrForCrosstalkSources_855404_52.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash db6cfbb94c1b3f015e7181ced3b9d5a7 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b01620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bba1f562-0ac8-4e8c-9f99-a88cbd130dc2_isrForCrosstalkSources_855450_48.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bba1f562-0ac8-4e8c-9f99-a88cbd130dc2_isrForCrosstalkSources_855450_48.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 5c92e78be4198a7485c394ea573ad757 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f77a25e-3cd6-4b0a-a5b6-7212dc21aa01_isrForCrosstalkSources_855409_46.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f77a25e-3cd6-4b0a-a5b6-7212dc21aa01_isrForCrosstalkSources_855409_46.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash bcf421f4f842669c40c7fe45082aaddc parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c480e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8ca45d1b-b832-49b1-86e7-395da5ecb4be_isrForCrosstalkSources_855422_1.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8ca45d1b-b832-49b1-86e7-395da5ecb4be_isrForCrosstalkSources_855422_1.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash b06489eaf0887c26d823db045067ae6e parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/78e49f2c-c790-4d42-85a2-3bda14c2183c_isrForCrosstalkSources_855442_56.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/78e49f2c-c790-4d42-85a2-3bda14c2183c_isrForCrosstalkSources_855442_56.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 499300cb7606c4552d09d19bedc2608d parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/94fcddcc-67f8-4d83-8285-90b8d75a962e_isrForCrosstalkSources_855408_25.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/94fcddcc-67f8-4d83-8285-90b8d75a962e_isrForCrosstalkSources_855408_25.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 9f5315963947ae2b31f46799cbacd251 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/30b22380-6ecd-4c33-86db-caa72e7f20cb_isrForCrosstalkSources_855381_21.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/30b22380-6ecd-4c33-86db-caa72e7f20cb_isrForCrosstalkSources_855381_21.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 4a5d9748e83b2af65b6a3150c047e262 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2ada34f3-3c10-456a-aaa4-56b51246d719_isrForCrosstalkSources_855384_37.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2ada34f3-3c10-456a-aaa4-56b51246d719_isrForCrosstalkSources_855384_37.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 5b6df7b00a62d5c7c61408a7f6aeb843 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9f95f1ee-93c1-412d-90c2-2ec65b7578f7_isrForCrosstalkSources_855395_23.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9f95f1ee-93c1-412d-90c2-2ec65b7578f7_isrForCrosstalkSources_855395_23.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash cabe3788987602504f3388a736e7e02d parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4435830b-a058-470d-9459-09dcc2bdca13_isrForCrosstalkSources_855379_8.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4435830b-a058-470d-9459-09dcc2bdca13_isrForCrosstalkSources_855379_8.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 3f4be53e7db643e1ae10452dd867866a parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/12e671f0-ea70-4f90-afb0-7a460381860c_isrForCrosstalkSources_855434_27.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/12e671f0-ea70-4f90-afb0-7a460381860c_isrForCrosstalkSources_855434_27.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 597cbe14f2e5c9d3dd335470d92e5738 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6d3e9d06-6f73-4f12-bb2d-3c1486d789f0_isrForCrosstalkSources_855410_48.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6d3e9d06-6f73-4f12-bb2d-3c1486d789f0_isrForCrosstalkSources_855410_48.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 6eef7980a82e47bcb509160154b8727b parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e38c929a-fcef-4f87-aee8-2440a92d0636_isrForCrosstalkSources_855390_51.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e38c929a-fcef-4f87-aee8-2440a92d0636_isrForCrosstalkSources_855390_51.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 50066869a46f4748424864434f68713a parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d8a47d61-5578-47e3-902b-029fdfd14362_isrForCrosstalkSources_855383_11.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d8a47d61-5578-47e3-902b-029fdfd14362_isrForCrosstalkSources_855383_11.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 9f835085fad3a4e74572055f309bb642 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e79aeb1c-11b1-4716-9c68-089c1ef2c9c7_isrForCrosstalkSources_855438_62.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e79aeb1c-11b1-4716-9c68-089c1ef2c9c7_isrForCrosstalkSources_855438_62.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 1379f3cd1f83c55e1153701b3a8b9d13 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/90342eb7-944d-4590-a07f-05f38937d614_isrForCrosstalkSources_855453_29.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/90342eb7-944d-4590-a07f-05f38937d614_isrForCrosstalkSources_855453_29.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 1e7b0fe4d05f2b8f495f90ad8220d6e3 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/05ca043e-ca98-4d7c-9f3b-0eb65c9796dc_isrForCrosstalkSources_855388_26.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/05ca043e-ca98-4d7c-9f3b-0eb65c9796dc_isrForCrosstalkSources_855388_26.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 59534fd95e65e7f677d36630fe01cce0 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1cadedca-87f2-4800-99d3-26ba45ed42bd_isrForCrosstalkSources_855442_62.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1cadedca-87f2-4800-99d3-26ba45ed42bd_isrForCrosstalkSources_855442_62.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash fec3e05117d8a0eec97b80f9d6b3283a parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:50 proc_lsst.multi:146 [INFO] found job 77954 in provider local parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b3a156e5-db54-470c-9f6e-20c7c624ca38_isrForCrosstalkSources_855415_54.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b3a156e5-db54-470c-9f6e-20c7c624ca38_isrForCrosstalkSources_855415_54.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 5cac43c6c0ddb18e8139f454e5356c63 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: found job 77954 in provider local parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2520b55b-5326-471e-a882-cdd2eeea49d0_isrForCrosstalkSources_855422_13.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2520b55b-5326-471e-a882-cdd2eeea49d0_isrForCrosstalkSources_855422_13.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 609e46bacb71111b9e3e20fc16c63a56 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/a436dd41-b62b-49e8-a549-035f274b3365_isrForCrosstalkSources_855402_51.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/a436dd41-b62b-49e8-a549-035f274b3365_isrForCrosstalkSources_855402_51.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 9a49da6315192775ad99d8468e52c943 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bdb11fcb-1aca-432e-9401-b4dbea4fe933_isrForCrosstalkSources_855426_39.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bdb11fcb-1aca-432e-9401-b4dbea4fe933_isrForCrosstalkSources_855426_39.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 4c19bc66b4dc6c0802f87b79a8059dc5 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2cdf767f-a379-4ce7-9fa8-d1284f449b90_isrForCrosstalkSources_855414_33.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2cdf767f-a379-4ce7-9fa8-d1284f449b90_isrForCrosstalkSources_855414_33.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 2cb20e1302004d73e13095262a61e70c parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bd269dbd-8a6d-4b6a-bc2c-b025639b2755_isrForCrosstalkSources_855431_61.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bd269dbd-8a6d-4b6a-bc2c-b025639b2755_isrForCrosstalkSources_855431_61.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash ceac414a4e8524609238a634c79f6ccc parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a206943f-07aa-44e3-86b8-6cf3f7805baf_isrForCrosstalkSources_855396_39.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a206943f-07aa-44e3-86b8-6cf3f7805baf_isrForCrosstalkSources_855396_39.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash bc73d24009ab5a2efe89f43d4c03ec35 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ee457bf2-8e07-494a-ba70-93c9da0cb863_isrForCrosstalkSources_855440_29.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ee457bf2-8e07-494a-ba70-93c9da0cb863_isrForCrosstalkSources_855440_29.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 6e0b71a5ef0e33261972d93b475c30b1 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3a1fb120-6d62-48e1-89ba-0dc3519bbbdb_isrForCrosstalkSources_855381_23.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3a1fb120-6d62-48e1-89ba-0dc3519bbbdb_isrForCrosstalkSources_855381_23.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash a9f45d61a0f369d7d55e34eb540f1e5d parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 109 parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed5b9281-bf42-4ea0-824f-b8bafde70dbc_isrForCrosstalkSources_855427_50.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed5b9281-bf42-4ea0-824f-b8bafde70dbc_isrForCrosstalkSources_855427_50.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 50cb983923d2db41c1183758d5c9493a parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/678ae8e1-b153-4867-b973-c2c5f1ad3fa5_isrForCrosstalkSources_855408_30.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/678ae8e1-b153-4867-b973-c2c5f1ad3fa5_isrForCrosstalkSources_855408_30.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 820243d04d1c0c510ecbdb0bd4c6a73e parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Executor multi has 109 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 27 more blocks parsl.executors.status_handling INFO: Scaling out by 27 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-09 14:49:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-09 14:49:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 14:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/97c76f00-e7e6-450b-911e-2b1f96995a7b_isrForCrosstalkSources_855392_44.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/97c76f00-e7e6-450b-911e-2b1f96995a7b_isrForCrosstalkSources_855392_44.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 6c14b7962587db55cfa7c848b729b79c parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3f0bb869-b42e-48ac-bf38-a1e079813d89_isrForCrosstalkSources_855407_45.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3f0bb869-b42e-48ac-bf38-a1e079813d89_isrForCrosstalkSources_855407_45.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 28d0f7322c36f92f9ae931298cb282f9 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/64182dd3-9340-4086-8242-0bf78e117eb4_isrForCrosstalkSources_855392_53.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/64182dd3-9340-4086-8242-0bf78e117eb4_isrForCrosstalkSources_855392_53.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash bbf7715212f53535dbbbe02930e81643 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a20e8c6d-5502-4bc4-a979-dd1b447d24b9_isrForCrosstalkSources_855435_12.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a20e8c6d-5502-4bc4-a979-dd1b447d24b9_isrForCrosstalkSources_855435_12.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b4c94a823aa4ee3034c7041d6b558ad8 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c494e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d9da83-fa3c-4bc5-b63a-033c0b1f24bc_isrForCrosstalkSources_855426_48.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d9da83-fa3c-4bc5-b63a-033c0b1f24bc_isrForCrosstalkSources_855426_48.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 0b7d150660f28eceda89a0dea98a3d8b parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b5abb156-f24a-4a91-9ada-c25403942e58_isrForCrosstalkSources_855397_30.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b5abb156-f24a-4a91-9ada-c25403942e58_isrForCrosstalkSources_855397_30.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash bc354ca22f8c32ff533c813a99f18c48 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f45ff313-5be1-4231-864f-78306c92f152_isrForCrosstalkSources_855453_12.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f45ff313-5be1-4231-864f-78306c92f152_isrForCrosstalkSources_855453_12.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 85dd6c62f60e209e8642fdc8662b5cfd parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ded99ff1-adf4-4ae1-bf63-0a51ad24abd8_isrForCrosstalkSources_855424_29.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ded99ff1-adf4-4ae1-bf63-0a51ad24abd8_isrForCrosstalkSources_855424_29.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 690870ac5d3e534f6055e94f54569b7b parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/38f8e3d2-5c23-4b3a-8571-2f8128e32947_isrForCrosstalkSources_855395_43.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/38f8e3d2-5c23-4b3a-8571-2f8128e32947_isrForCrosstalkSources_855395_43.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 43586e16767f1a54bcddfda0f37bd3a2 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d092a12d-eb32-4f69-a2f0-86cfd7f687c2_isrForCrosstalkSources_855386_22.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d092a12d-eb32-4f69-a2f0-86cfd7f687c2_isrForCrosstalkSources_855386_22.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 8056fa7ce42557659312e3f5ffe8d8d4 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ceb88ed0-6670-4b41-b6ea-352c00fe2903_isrForCrosstalkSources_855434_48.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ceb88ed0-6670-4b41-b6ea-352c00fe2903_isrForCrosstalkSources_855434_48.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 2e80d41d1023d5fe86ff68f74b40f433 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/72725a73-e6a9-4f9d-8915-899d94c00649_isrForCrosstalkSources_855387_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/72725a73-e6a9-4f9d-8915-899d94c00649_isrForCrosstalkSources_855387_61.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 61986706d30b37abee29789df9c7404c parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/81b2baf5-8cd5-4ebf-b2c6-23184f060d33_isrForCrosstalkSources_855386_34.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/81b2baf5-8cd5-4ebf-b2c6-23184f060d33_isrForCrosstalkSources_855386_34.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash f968c0df8ae2e989b0865c97b7806fb8 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f19f2bc5-f453-46bb-9529-3df745494d3a_isrForCrosstalkSources_855395_19.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f19f2bc5-f453-46bb-9529-3df745494d3a_isrForCrosstalkSources_855395_19.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 97214ae8b979df536b62494ac6df1e2e parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/6cf600cf-3bed-45da-ad5a-464db0dc546b_isrForCrosstalkSources_855403_37.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/6cf600cf-3bed-45da-ad5a-464db0dc546b_isrForCrosstalkSources_855403_37.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash e84ca320771f73ba286e77f0f073b992 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/e40f36d6-805c-4582-8817-3c40a2a3d714_isrForCrosstalkSources_855426_60.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/e40f36d6-805c-4582-8817-3c40a2a3d714_isrForCrosstalkSources_855426_60.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 4c2c133d699617b1d79cc360d330d563 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/8e3fa845-c8a1-409d-865a-4bc7c7cd7e5b_isrForCrosstalkSources_855390_23.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/8e3fa845-c8a1-409d-865a-4bc7c7cd7e5b_isrForCrosstalkSources_855390_23.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 4ae8c690e1d71e41d5a0b02838c7e428 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/815de729-47e5-4861-a301-010813a6403f_isrForCrosstalkSources_855389_28.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/815de729-47e5-4861-a301-010813a6403f_isrForCrosstalkSources_855389_28.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 888a04031b8b481f0da47d89c17c4fe5 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2347db10-d335-4336-b6ce-3f552480e9b6_isrForCrosstalkSources_855429_50.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2347db10-d335-4336-b6ce-3f552480e9b6_isrForCrosstalkSources_855429_50.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 574c6494627f1ae8892c394ce13acf04 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6cf79e8c-ff5f-473f-ad0f-00d027cfc95e_isrForCrosstalkSources_855458_54.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6cf79e8c-ff5f-473f-ad0f-00d027cfc95e_isrForCrosstalkSources_855458_54.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 54dbef74b877970e62c6e009a0954921 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/00884cfb-f47c-4adb-a764-69569df32c9c_isrForCrosstalkSources_855382_56.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/00884cfb-f47c-4adb-a764-69569df32c9c_isrForCrosstalkSources_855382_56.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash b70a731926700baa1e94306bfb8ab3de parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/da3e6e76-fdda-4547-87c1-d9001a475908_isrForCrosstalkSources_855446_58.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/da3e6e76-fdda-4547-87c1-d9001a475908_isrForCrosstalkSources_855446_58.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 3e21e9a4a3556c84ee88ec78a478e821 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e8ae782f-351d-48fb-aa0d-5a6bafad3b63_isrForCrosstalkSources_855451_37.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e8ae782f-351d-48fb-aa0d-5a6bafad3b63_isrForCrosstalkSources_855451_37.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 59349211e8f905c32d531ab94161c4e8 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3aa504f5-6954-4f3e-ae97-5a36f335097e_isrForCrosstalkSources_855407_16.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3aa504f5-6954-4f3e-ae97-5a36f335097e_isrForCrosstalkSources_855407_16.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 60e761d03f73deff01e2ef5101d157fc parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/de1df16e-7580-404b-aba1-7cc2069d00f1_isrForCrosstalkSources_855416_56.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/de1df16e-7580-404b-aba1-7cc2069d00f1_isrForCrosstalkSources_855416_56.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 10efabc5c87744473df0b205c063347e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/88f62bb3-3c19-437b-a5eb-7b58455b6786_isrForCrosstalkSources_855396_57.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/88f62bb3-3c19-437b-a5eb-7b58455b6786_isrForCrosstalkSources_855396_57.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 5b37082f97105925d5776f83826097f9 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:50 proc_lsst.multi:170 [INFO] [multi] job_id 16964026 proc_lsst.multi INFO: [multi] job_id 16964026 2024-03-09 14:49:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964026 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964026 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16964026 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-09 14:49:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-09 14:49:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1caed2e4-b98c-4b09-945e-94009ee52b3b_isrForCrosstalkSources_855450_32.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1caed2e4-b98c-4b09-945e-94009ee52b3b_isrForCrosstalkSources_855450_32.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 91b9faf945d4628db21da8151a1f6a29 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f23539ef-f7bb-499a-a675-717f48a6b68a_isrForCrosstalkSources_855436_31.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f23539ef-f7bb-499a-a675-717f48a6b68a_isrForCrosstalkSources_855436_31.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash ee6dbdfb20c93d97d20d4eba616ea88d parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7a87aa4c-f5c8-4066-a9b3-3d83bb61004a_isrForCrosstalkSources_855392_10.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7a87aa4c-f5c8-4066-a9b3-3d83bb61004a_isrForCrosstalkSources_855392_10.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 9fb04471057474de3f7460b4ab9b3d59 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9119ade5-7faa-4cf0-92d1-9bd6c4c79bf7_isrForCrosstalkSources_855444_53.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9119ade5-7faa-4cf0-92d1-9bd6c4c79bf7_isrForCrosstalkSources_855444_53.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash b0594deafae0aad3b111687d2b766bc2 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0bcdbeb7-86de-4898-b485-183f79bc3ed8_isrForCrosstalkSources_855395_1.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0bcdbeb7-86de-4898-b485-183f79bc3ed8_isrForCrosstalkSources_855395_1.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash ae7b0bb8d4e63283347c08ff4f79d009 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/7ec35235-e7ab-40ce-9f7d-73075ba7fd10_isrForCrosstalkSources_855454_58.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/7ec35235-e7ab-40ce-9f7d-73075ba7fd10_isrForCrosstalkSources_855454_58.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash a6f645a7350c6485be159677cda9b22b parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2df251d6-0be4-4ed7-bc2d-f6c0dfb88cdf_isrForCrosstalkSources_855422_60.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2df251d6-0be4-4ed7-bc2d-f6c0dfb88cdf_isrForCrosstalkSources_855422_60.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash c50bcc00cc702ee763917221cec9aea6 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/75fe732a-57fb-4594-9c77-863ba215d6f6_isrForCrosstalkSources_855399_22.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/75fe732a-57fb-4594-9c77-863ba215d6f6_isrForCrosstalkSources_855399_22.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 8c9a49f8f8ae4aef38883f3a3e20e66e parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b7a0aede-7218-44f2-b832-202a79f05bf8_isrForCrosstalkSources_855442_36.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b7a0aede-7218-44f2-b832-202a79f05bf8_isrForCrosstalkSources_855442_36.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 05c9889dd602dcb35abff08000f91427 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/63a24297-926d-419e-ae0a-2287b2a6f6af_isrForCrosstalkSources_855413_20.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/63a24297-926d-419e-ae0a-2287b2a6f6af_isrForCrosstalkSources_855413_20.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash c398190e64fdcf2c9cb9c74a62a8cfdc parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f04fcea5-c999-4f0b-9beb-c9f0a85f8b2b_isrForCrosstalkSources_855448_49.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f04fcea5-c999-4f0b-9beb-c9f0a85f8b2b_isrForCrosstalkSources_855448_49.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 4a0d180a65032dd52d6dcb4bac3833ea parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e448432d-30c1-4794-a41c-8f9ca313c6ea_isrForCrosstalkSources_855392_56.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e448432d-30c1-4794-a41c-8f9ca313c6ea_isrForCrosstalkSources_855392_56.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 10e8c77056bb36a9f5b146daa1b8cb26 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/95a92f3e-45c6-4083-945f-1a0381bcff8d_isrForCrosstalkSources_855447_33.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/95a92f3e-45c6-4083-945f-1a0381bcff8d_isrForCrosstalkSources_855447_33.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 54787dae9b6e95041b0d719e433dbae5 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/947ec90b-957b-4c03-86c1-e577a2bb2393_isrForCrosstalkSources_855394_12.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/947ec90b-957b-4c03-86c1-e577a2bb2393_isrForCrosstalkSources_855394_12.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash fd145774920193c18c90e37ebd84a04c parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/51a4c46e-21bf-44fa-b19a-4f8f8dd05bec_isrForCrosstalkSources_855423_50.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/51a4c46e-21bf-44fa-b19a-4f8f8dd05bec_isrForCrosstalkSources_855423_50.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash a1169d9d1d0e77578b78e319f92adcc0 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d48ec79c-04fd-4f59-93a7-c2c7425f88d9_isrForCrosstalkSources_855445_14.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d48ec79c-04fd-4f59-93a7-c2c7425f88d9_isrForCrosstalkSources_855445_14.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 1fc764fae48905e8457842ef56b58b7e parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0f1a2025-5e9a-4d26-a6ea-2dcc28e31aab_isrForCrosstalkSources_855384_40.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0f1a2025-5e9a-4d26-a6ea-2dcc28e31aab_isrForCrosstalkSources_855384_40.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash c959cab582a5a7b4c9bcc4e02019b4e7 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/446e0923-53e0-4442-b7f2-951183a361bd_isrForCrosstalkSources_855443_27.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/446e0923-53e0-4442-b7f2-951183a361bd_isrForCrosstalkSources_855443_27.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash c2f11d81d923e07a1bf8fa462f9e433f parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:51 proc_lsst.multi:170 [INFO] [multi] job_id 16964027 proc_lsst.multi INFO: [multi] job_id 16964027 2024-03-09 14:49:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:49:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964027 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964027 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16964027 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-09 14:49:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-09 14:49:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a1f1e329-2a96-4e56-abaa-32a27f24110b_isrForCrosstalkSources_855429_30.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a1f1e329-2a96-4e56-abaa-32a27f24110b_isrForCrosstalkSources_855429_30.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 8d64e29e0abe60b2467e434028f9af52 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d56687c1-fae7-473a-8c69-e3e99594c9e2_isrForCrosstalkSources_855383_47.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d56687c1-fae7-473a-8c69-e3e99594c9e2_isrForCrosstalkSources_855383_47.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 55f12d500b447eacb0ac77cf214e8398 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/76c6b48b-02e7-4333-b0dd-99f18d70ea3a_isrForCrosstalkSources_855381_47.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/76c6b48b-02e7-4333-b0dd-99f18d70ea3a_isrForCrosstalkSources_855381_47.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 62da8ba7ca5ee5c1470c4d6cca44fc48 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8b03ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c44e0032-3ed4-48bc-93f3-fd2f6c893ded_isrForCrosstalkSources_855387_7.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c44e0032-3ed4-48bc-93f3-fd2f6c893ded_isrForCrosstalkSources_855387_7.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash e7b39b4a91c24709c64d38d1f3ed35a0 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3ef22b8e-088e-498c-b2c8-fe7e293bb78d_isrForCrosstalkSources_855404_35.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3ef22b8e-088e-498c-b2c8-fe7e293bb78d_isrForCrosstalkSources_855404_35.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash dc14f2952ee33110f2ebbe4513327455 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/773e3c0d-b1d5-4a87-88eb-80c19858afb3_isrForCrosstalkSources_855410_25.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/773e3c0d-b1d5-4a87-88eb-80c19858afb3_isrForCrosstalkSources_855410_25.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 1ade03d73bd3295dead53a5995e400b8 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fb262b4e-e3bd-4d6f-b12c-aba6f74d9855_isrForCrosstalkSources_855449_3.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fb262b4e-e3bd-4d6f-b12c-aba6f74d9855_isrForCrosstalkSources_855449_3.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 04ad3be52f5a1e116c47008703807fcf parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c493a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c36a562e-cd5c-41fc-ae18-f585233d7409_isrForCrosstalkSources_855426_21.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c36a562e-cd5c-41fc-ae18-f585233d7409_isrForCrosstalkSources_855426_21.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash fc48b3ca2f7fad715b2a839aeba476c5 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c7ec8907-dbf7-49fa-b0df-6d61a1b23418_isrForCrosstalkSources_855417_53.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c7ec8907-dbf7-49fa-b0df-6d61a1b23418_isrForCrosstalkSources_855417_53.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 50ddd1ed3ddecebba3f42e054572f9a8 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a9305eb6-5ac7-42b4-9542-3a8689986e53_isrForCrosstalkSources_855397_17.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a9305eb6-5ac7-42b4-9542-3a8689986e53_isrForCrosstalkSources_855397_17.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 522d4fa73bc0ca8b74b89c28cbbe8ac7 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e421317-f91e-4a51-9450-67d7a5bc317c_isrForCrosstalkSources_855449_29.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e421317-f91e-4a51-9450-67d7a5bc317c_isrForCrosstalkSources_855449_29.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 6912b1a9d43ef85be2d46469b7d37ac7 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ad99dc1d-6480-4821-9bc4-3dc27e64c2e9_isrForCrosstalkSources_855447_1.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ad99dc1d-6480-4821-9bc4-3dc27e64c2e9_isrForCrosstalkSources_855447_1.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash c610e18cd185ff14d8d8e19eb76901c8 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/053f7b10-d523-4cc3-8226-3ed5a5851e1c_isrForCrosstalkSources_855406_26.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/053f7b10-d523-4cc3-8226-3ed5a5851e1c_isrForCrosstalkSources_855406_26.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash e73da8aa755ea934601e4c177f7468a7 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5db58ed1-f632-4362-85fc-87a62823e911_isrForCrosstalkSources_855416_57.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5db58ed1-f632-4362-85fc-87a62823e911_isrForCrosstalkSources_855416_57.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash a8a1ffe0ccbc68ceadbe723297ea8e1d parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/89452317-4c20-425c-9a34-ccdac4c6f150_isrForCrosstalkSources_855434_44.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/89452317-4c20-425c-9a34-ccdac4c6f150_isrForCrosstalkSources_855434_44.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 8b9e20836e414f4eeb825fea042cde1d parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/56733b72-7a28-4124-b6ca-6bb6b2497b1d_isrForCrosstalkSources_855382_10.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/56733b72-7a28-4124-b6ca-6bb6b2497b1d_isrForCrosstalkSources_855382_10.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash b9ead4cbccd89422ac73616a3b645674 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/41b25252-2e75-411c-b7a7-f055ed8dfb62_isrForCrosstalkSources_855385_50.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/41b25252-2e75-411c-b7a7-f055ed8dfb62_isrForCrosstalkSources_855385_50.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash d5da861e86f602ab3f7a03be834688ca parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6f2640e1-9c2d-4fa0-8d96-2afb8d0c8743_isrForCrosstalkSources_855445_31.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6f2640e1-9c2d-4fa0-8d96-2afb8d0c8743_isrForCrosstalkSources_855445_31.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash c2ac86e9931c22ef461f36e3c9ccbf5d parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3ab1f7a6-9d8b-470d-bad7-729b4483cbf0_isrForCrosstalkSources_855403_59.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3ab1f7a6-9d8b-470d-bad7-729b4483cbf0_isrForCrosstalkSources_855403_59.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 561d3266cfac23c38a5a68e5b746db64 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5d035bec-230b-4312-9e98-7081ef6b6bc3_isrForCrosstalkSources_855441_23.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5d035bec-230b-4312-9e98-7081ef6b6bc3_isrForCrosstalkSources_855441_23.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash e8fe6273aab45ad5000179dcf36da704 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:51 proc_lsst.multi:170 [INFO] [multi] job_id 16964028 proc_lsst.multi INFO: [multi] job_id 16964028 2024-03-09 14:49:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:49:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964028 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964028 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16964028 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-09 14:49:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-09 14:49:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/370c6fbc-2b39-43a3-b9b8-101b56e76746_isrForCrosstalkSources_855422_9.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/370c6fbc-2b39-43a3-b9b8-101b56e76746_isrForCrosstalkSources_855422_9.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 5e89ef0466ffd89426f657c2abfb9c59 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/1273549f-7a51-4e9e-a2af-ba8d29a70f78_isrForCrosstalkSources_855414_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/1273549f-7a51-4e9e-a2af-ba8d29a70f78_isrForCrosstalkSources_855414_59.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 67a108ce3e9d9b14e7e98d64d9ae6bfe parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/102dca8d-a803-4926-9e37-22c5479fbb44_isrForCrosstalkSources_855455_1.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/102dca8d-a803-4926-9e37-22c5479fbb44_isrForCrosstalkSources_855455_1.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 3dfded93159157b9225aca2781b19031 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/785e1362-3196-4404-b220-da5cd889643b_isrForCrosstalkSources_855454_22.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/785e1362-3196-4404-b220-da5cd889643b_isrForCrosstalkSources_855454_22.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 4956e3da7685a683faacddd19ed49a66 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/38440223-ee29-4cb5-af44-2d0e818c9ec3_isrForCrosstalkSources_855458_4.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/38440223-ee29-4cb5-af44-2d0e818c9ec3_isrForCrosstalkSources_855458_4.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 534c3ed8348eda1f390147ee3be2cb8c parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/9cfc6a3d-a04c-46c9-b61e-88074c43ce1f_isrForCrosstalkSources_855404_31.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/9cfc6a3d-a04c-46c9-b61e-88074c43ce1f_isrForCrosstalkSources_855404_31.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 3364df049f080bbe1aebba076d9cf9fa parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/46fb774e-9eab-4de7-b444-6e4573739549_isrForCrosstalkSources_855417_49.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/46fb774e-9eab-4de7-b444-6e4573739549_isrForCrosstalkSources_855417_49.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 8e8e5af84220fa521ab1d0d7af25d840 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9d6d25bc-25ea-47ce-902f-a2440481c620_isrForCrosstalkSources_855428_38.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9d6d25bc-25ea-47ce-902f-a2440481c620_isrForCrosstalkSources_855428_38.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 0e36ac5dbb114f745f6991021b8eaf4b parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e2a2e292-dc77-488a-a796-14f58d8ec03b_isrForCrosstalkSources_855424_32.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e2a2e292-dc77-488a-a796-14f58d8ec03b_isrForCrosstalkSources_855424_32.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash d6bb9283dda0fa4da542818af94afd60 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcac0e1-be24-4934-b97d-cf532d4a3b24_isrForCrosstalkSources_855409_16.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcac0e1-be24-4934-b97d-cf532d4a3b24_isrForCrosstalkSources_855409_16.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 5f337705f49747a66e802767692acbdb parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e4531a74-8ca4-423d-893e-7da425fef2ad_isrForCrosstalkSources_855416_39.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e4531a74-8ca4-423d-893e-7da425fef2ad_isrForCrosstalkSources_855416_39.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 0bc37832687e6a57794a9512f000405c parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/540cc22f-e2f4-4257-80e4-2e5d0d1df20f_isrForCrosstalkSources_855399_27.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/540cc22f-e2f4-4257-80e4-2e5d0d1df20f_isrForCrosstalkSources_855399_27.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 52cc3dc1683ce876c1ca3840f02c0a6d parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8e4e9671-3f3a-4b77-8709-28a129c62908_isrForCrosstalkSources_855443_39.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8e4e9671-3f3a-4b77-8709-28a129c62908_isrForCrosstalkSources_855443_39.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 715cfac95dbf2457bc68cff4c1870f32 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca5e15ae-7c0e-4990-b408-61410b7a383f_isrForCrosstalkSources_855400_38.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca5e15ae-7c0e-4990-b408-61410b7a383f_isrForCrosstalkSources_855400_38.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash b5e1892311fa7d41e95c5faaa45f1ec0 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/aac80d6f-1831-4513-8b22-af24c6c6dc32_isrForCrosstalkSources_855454_2.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/aac80d6f-1831-4513-8b22-af24c6c6dc32_isrForCrosstalkSources_855454_2.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash f69ec3f6450da9667fc1b850030fbfb4 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0cfb8404-c35a-4c46-90a4-549ce15422c2_isrForCrosstalkSources_855420_57.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0cfb8404-c35a-4c46-90a4-549ce15422c2_isrForCrosstalkSources_855420_57.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 937e1037db8671de1275f4fc3d42423b parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/53246840-1222-48a8-83a3-2588fe4f3fa5_isrForCrosstalkSources_855418_8.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/53246840-1222-48a8-83a3-2588fe4f3fa5_isrForCrosstalkSources_855418_8.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 9765f48ac6972ac90605d88fbaa72526 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf75224-ef9c-4a8f-98e0-a1cde723b061_isrForCrosstalkSources_855388_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf75224-ef9c-4a8f-98e0-a1cde723b061_isrForCrosstalkSources_855388_13.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 11ed16a7f786d354ff0cab8b59741723 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/eb355cfc-b734-44ea-bbae-977a7f195d82_isrForCrosstalkSources_855449_8.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/eb355cfc-b734-44ea-bbae-977a7f195d82_isrForCrosstalkSources_855449_8.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash ae68f88c0ac9fc778a9e2a41be798e3c parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1ca5c878-5636-4658-9952-cedaa718524b_isrForCrosstalkSources_855383_59.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1ca5c878-5636-4658-9952-cedaa718524b_isrForCrosstalkSources_855383_59.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 66fcc9bc866d6dbfcab8b1ae7956e291 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8f5161dd-31c6-48c0-b682-291c0241bef7_isrForCrosstalkSources_855414_32.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8f5161dd-31c6-48c0-b682-291c0241bef7_isrForCrosstalkSources_855414_32.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 68cdffb739cab136e540aca5e05f1945 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/27c45629-4d61-41fd-94b6-38ad218b752a_isrForCrosstalkSources_855418_42.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/27c45629-4d61-41fd-94b6-38ad218b752a_isrForCrosstalkSources_855418_42.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 568a441063b38582692718aa05cd8c6a parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:52 proc_lsst.multi:170 [INFO] [multi] job_id 16964029 proc_lsst.multi INFO: [multi] job_id 16964029 2024-03-09 14:49:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:49:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964029 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964029 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16964029 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-09 14:49:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-09 14:49:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:49:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ae517807-0780-4d35-bff8-5453c2a71b47_isrForCrosstalkSources_855435_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ae517807-0780-4d35-bff8-5453c2a71b47_isrForCrosstalkSources_855435_35.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 5331c8f6ba74d7fd8939ae9fc3dae32e parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eecf7047-af51-423b-bb6e-be8ca98cf30e_isrForCrosstalkSources_855381_46.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eecf7047-af51-423b-bb6e-be8ca98cf30e_isrForCrosstalkSources_855381_46.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 329f5ae81dddf928982f3529de10d545 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bc631c22-5dd2-4fdd-b151-a9ca1df2060d_isrForCrosstalkSources_855442_3.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bc631c22-5dd2-4fdd-b151-a9ca1df2060d_isrForCrosstalkSources_855442_3.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash de9d4a2e7a4a224d33ba999629ab66e6 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c49580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/03a43c11-1b90-40c3-be56-86d4e00c7f0b_isrForCrosstalkSources_855428_14.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/03a43c11-1b90-40c3-be56-86d4e00c7f0b_isrForCrosstalkSources_855428_14.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash edf2f57f2df04eed670d837b6c4390da parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/17740517-fe19-437e-a820-5d4f29f4692e_isrForCrosstalkSources_855398_51.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/17740517-fe19-437e-a820-5d4f29f4692e_isrForCrosstalkSources_855398_51.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash b99a57e2ecf69f081a73298ffb3346ee parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/74cb720b-34d1-44cc-9232-20cfb424deaf_isrForCrosstalkSources_855394_48.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/74cb720b-34d1-44cc-9232-20cfb424deaf_isrForCrosstalkSources_855394_48.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 622d35ab7a95cd1cb9db9d27f312ce56 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4e3792d2-27da-47e9-9a79-9a89a1c4c5d1_isrForCrosstalkSources_855388_59.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4e3792d2-27da-47e9-9a79-9a89a1c4c5d1_isrForCrosstalkSources_855388_59.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash b04505f5695fa8f21be88eaeebda2e96 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/712bbbfc-aea2-4896-b4c2-08e775ccc05f_isrForCrosstalkSources_855450_62.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/712bbbfc-aea2-4896-b4c2-08e775ccc05f_isrForCrosstalkSources_855450_62.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash b222a0f29fc9d0b062f93f4d12e267f3 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c00f6760-63b6-4498-b042-709af639604a_isrForCrosstalkSources_855441_9.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c00f6760-63b6-4498-b042-709af639604a_isrForCrosstalkSources_855441_9.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 7ab563e9d097ede93020e4f29c7945cc parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/976916db-1f5d-42f5-bc78-a595a95eed6c_isrForCrosstalkSources_855393_40.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/976916db-1f5d-42f5-bc78-a595a95eed6c_isrForCrosstalkSources_855393_40.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash c0091bc7489cf3bb18f20def541be080 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e127f447-54a8-464f-9b51-31176cb5d292_isrForCrosstalkSources_855413_39.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e127f447-54a8-464f-9b51-31176cb5d292_isrForCrosstalkSources_855413_39.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 74931db9111aabee8646f4bd92124a66 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/39ed7f4a-856d-4301-9f94-e177c80c6d33_isrForCrosstalkSources_855433_62.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/39ed7f4a-856d-4301-9f94-e177c80c6d33_isrForCrosstalkSources_855433_62.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 88a05a2c51635b8fe8f49da46b060905 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f33fc030-10f8-44b2-9d5e-a543f45f5f7e_isrForCrosstalkSources_855455_37.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f33fc030-10f8-44b2-9d5e-a543f45f5f7e_isrForCrosstalkSources_855455_37.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash ac16e74e831458b11cc7d1bb499c39a2 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/772d07ba-9301-4545-9f3d-8424ef4f5b09_isrForCrosstalkSources_855451_58.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/772d07ba-9301-4545-9f3d-8424ef4f5b09_isrForCrosstalkSources_855451_58.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 15d08d5eae9b76953df171b95f6d9e2e parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/6b0493e2-d321-4233-8912-29c63a81f39d_isrForCrosstalkSources_855441_22.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/6b0493e2-d321-4233-8912-29c63a81f39d_isrForCrosstalkSources_855441_22.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 077e16f5e3c8168168c45faf06bb7af1 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4069fa53-eabd-4c94-ba0c-1cbb5c2fa0a2_isrForCrosstalkSources_855405_46.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4069fa53-eabd-4c94-ba0c-1cbb5c2fa0a2_isrForCrosstalkSources_855405_46.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 007bed86f82a7baedd011f8afc5f78e0 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/397ef0e1-4a1a-46c2-9f91-ff520b284f39_isrForCrosstalkSources_855419_28.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/397ef0e1-4a1a-46c2-9f91-ff520b284f39_isrForCrosstalkSources_855419_28.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 4dea5290d03ab6dbd2bb5d9064eb05be parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:52 proc_lsst.multi:170 [INFO] [multi] job_id 16964030 proc_lsst.multi INFO: [multi] job_id 16964030 2024-03-09 14:49:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:49:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964030 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964030 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16964030 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-09 14:49:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-09 14:49:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:49:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9f7b8-7f72-49e7-a17c-679505b035ed_isrForCrosstalkSources_855429_41.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9f7b8-7f72-49e7-a17c-679505b035ed_isrForCrosstalkSources_855429_41.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 95be5ff5adbb75503f00704bf06ca129 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32e8ca45-f9dd-4785-913d-893e6b1e492e_isrForCrosstalkSources_855447_16.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32e8ca45-f9dd-4785-913d-893e6b1e492e_isrForCrosstalkSources_855447_16.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 8506a5bf9ae1bb323a6972f21674c60f parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/05c6f1e7-3bd7-43e1-a9ef-dea1defbe65d_isrForCrosstalkSources_855421_53.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/05c6f1e7-3bd7-43e1-a9ef-dea1defbe65d_isrForCrosstalkSources_855421_53.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 7419a599081981e4536153962a538e47 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8708523-3410-4545-a5d5-e725424f5b17_isrForCrosstalkSources_855412_53.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8708523-3410-4545-a5d5-e725424f5b17_isrForCrosstalkSources_855412_53.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 27b8912bad0ab61ece3c3c8ea13e0bce parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ced4ed47-012e-4925-b944-db17aea0bc17_isrForCrosstalkSources_855379_57.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ced4ed47-012e-4925-b944-db17aea0bc17_isrForCrosstalkSources_855379_57.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 065453a78a84e737f38557709c8aabc2 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4fefc08-b460-4945-bfa9-c01d6c68bb1f_isrForCrosstalkSources_855393_58.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4fefc08-b460-4945-bfa9-c01d6c68bb1f_isrForCrosstalkSources_855393_58.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 4337bccadc0c6f92fa542248f1c8973b parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e65d006-6493-4183-9478-ca0f24607960_isrForCrosstalkSources_855408_43.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e65d006-6493-4183-9478-ca0f24607960_isrForCrosstalkSources_855408_43.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash e5d4acb67df0d6b379f60648485534a7 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a423c639-7cd3-4539-98ff-49bc26b55b8e_isrForCrosstalkSources_855384_18.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a423c639-7cd3-4539-98ff-49bc26b55b8e_isrForCrosstalkSources_855384_18.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash e64baac1fc2872204e720985c98b35a2 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e6102aad-580a-4694-b6a2-b11f38dc50a5_isrForCrosstalkSources_855420_29.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e6102aad-580a-4694-b6a2-b11f38dc50a5_isrForCrosstalkSources_855420_29.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 48aa5da189eef77b9cce2742efb32687 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a94cf063-0889-4910-8909-457d2a64bcd8_isrForCrosstalkSources_855449_16.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a94cf063-0889-4910-8909-457d2a64bcd8_isrForCrosstalkSources_855449_16.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 0b08b625ab1ef73d2a184016dbfe1849 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/2f5130f2-e7c3-4ce3-b456-d267b1511e61_isrForCrosstalkSources_855435_18.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/2f5130f2-e7c3-4ce3-b456-d267b1511e61_isrForCrosstalkSources_855435_18.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 6e2f61b2b17fadcfd5e673829c53a8bb parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/691b38e4-f919-462c-a475-ad0c0cb9d40d_isrForCrosstalkSources_855379_20.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/691b38e4-f919-462c-a475-ad0c0cb9d40d_isrForCrosstalkSources_855379_20.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash f8935e9c6fce79eadc1d255379a3262d parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/bac87108-fa39-4a72-9418-9423aa2e31d8_isrForCrosstalkSources_855452_59.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/bac87108-fa39-4a72-9418-9423aa2e31d8_isrForCrosstalkSources_855452_59.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 098c0e3f9266482d4ade78e064a9a06b parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/78b13cb2-68d9-4e1a-a1f2-372ef14794ab_isrForCrosstalkSources_855413_14.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/78b13cb2-68d9-4e1a-a1f2-372ef14794ab_isrForCrosstalkSources_855413_14.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 28a92f70593430d815e99bbeb7030999 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/77e17cde-47b9-4825-bf7c-af3a0063c465_isrForCrosstalkSources_855411_52.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/77e17cde-47b9-4825-bf7c-af3a0063c465_isrForCrosstalkSources_855411_52.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 4c2acc64a343856de7a7fd4bd4936912 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/39b2a43d-b75b-42b2-b1c7-dcef2cfdf011_isrForCrosstalkSources_855379_34.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/39b2a43d-b75b-42b2-b1c7-dcef2cfdf011_isrForCrosstalkSources_855379_34.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 30e1df0269a1ca43b679091c408c0100 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54b634f9-a38e-4e15-a65f-77dd5e2f94f5_isrForCrosstalkSources_855382_7.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54b634f9-a38e-4e15-a65f-77dd5e2f94f5_isrForCrosstalkSources_855382_7.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash b37b9e6c8bc2c0d7db2e9797663359d6 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9b262aa1-6f44-48e6-aace-9d238b40be64_isrForCrosstalkSources_855438_7.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9b262aa1-6f44-48e6-aace-9d238b40be64_isrForCrosstalkSources_855438_7.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 5d5613b4e1161d56f380e4594c9ed560 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c2437ef3-aeba-4f52-9a69-771c2cd98b06_isrForCrosstalkSources_855451_30.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c2437ef3-aeba-4f52-9a69-771c2cd98b06_isrForCrosstalkSources_855451_30.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 96c2b84d45e0ec401d2bfa336728aa19 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/bf458f0a-e311-4217-9af2-8488489674e4_isrForCrosstalkSources_855446_27.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/bf458f0a-e311-4217-9af2-8488489674e4_isrForCrosstalkSources_855446_27.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 8d0a7691c8498685d7a12fb4634bff34 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7b13b6a5-521f-45b4-b241-225cc3775812_isrForCrosstalkSources_855406_5.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7b13b6a5-521f-45b4-b241-225cc3775812_isrForCrosstalkSources_855406_5.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 5abeeb953dee93c0f5dd2e6905b7864a parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d1f1779d-476d-4329-ab8f-30d03ba56c6a_isrForCrosstalkSources_855421_18.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d1f1779d-476d-4329-ab8f-30d03ba56c6a_isrForCrosstalkSources_855421_18.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 9ac9af901db7a5f2f0dcf789a680f29a parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/2ea068c1-b746-4086-a578-ba88d8df01b4_isrForCrosstalkSources_855385_2.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/2ea068c1-b746-4086-a578-ba88d8df01b4_isrForCrosstalkSources_855385_2.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash a20435a79be801ed3146752bf534bec9 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f1956a3b-a300-449f-8f9e-c3e073c2e0c2_isrForCrosstalkSources_855432_60.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f1956a3b-a300-449f-8f9e-c3e073c2e0c2_isrForCrosstalkSources_855432_60.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 9ba3af1482ce52f70ca124c254d4fa8d parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/112b75d2-a1f1-4fa7-8868-20f66a4d08ba_isrForCrosstalkSources_855434_40.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/112b75d2-a1f1-4fa7-8868-20f66a4d08ba_isrForCrosstalkSources_855434_40.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 3b34a5ff145382a05fb51f2ff25c62e0 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 2024-03-09 14:49:52 proc_lsst.multi:170 [INFO] [multi] job_id 16964031 proc_lsst.multi INFO: [multi] job_id 16964031 2024-03-09 14:49:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 14:49:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964031 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964031 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/df839e70-fd29-4534-ba66-202c996f5c1e_isrForCrosstalkSources_855383_20.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/df839e70-fd29-4534-ba66-202c996f5c1e_isrForCrosstalkSources_855383_20.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash e5b1d4448e2daf055579b5c674debe23 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16964031 parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e7b77c0f-afc6-43fb-b7c6-cce6da95e5f2_isrForCrosstalkSources_855441_16.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e7b77c0f-afc6-43fb-b7c6-cce6da95e5f2_isrForCrosstalkSources_855441_16.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 52df9e1db34664e5d5f50e70b1e5698f parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c4bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-09 14:49:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-09 14:49:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 14:49:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b0cb3b7c-3f71-47d7-992a-cb74d7914727_isrForCrosstalkSources_855414_61.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b0cb3b7c-3f71-47d7-992a-cb74d7914727_isrForCrosstalkSources_855414_61.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash ad87428e4c385bf8ef6b62a13705e3a7 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/73827ce2-5851-47bb-b44c-b9042299481b_isrForCrosstalkSources_855395_39.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/73827ce2-5851-47bb-b44c-b9042299481b_isrForCrosstalkSources_855395_39.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 73020f9c8aafee408067544b6dddb936 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c36d5678-9ebf-4038-a835-fc2f04618420_isrForCrosstalkSources_855438_10.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c36d5678-9ebf-4038-a835-fc2f04618420_isrForCrosstalkSources_855438_10.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 10e33ff3b776d8a5b8f65d4d5ca520af parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/054c6952-15e1-4eed-a77f-342495918eaf_isrForCrosstalkSources_855415_8.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/054c6952-15e1-4eed-a77f-342495918eaf_isrForCrosstalkSources_855415_8.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 8fff53cdf04bedf4712ee34a413c9d32 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/ed3594e3-fdfe-4470-a30c-3d7bd236e265_isrForCrosstalkSources_855422_44.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/ed3594e3-fdfe-4470-a30c-3d7bd236e265_isrForCrosstalkSources_855422_44.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash b0471056c62bf0906164ef830ab7557a parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/eacba2aa-dfba-4fd3-93ee-10c240450754_isrForCrosstalkSources_855457_1.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/eacba2aa-dfba-4fd3-93ee-10c240450754_isrForCrosstalkSources_855457_1.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash df2c75458afe5633e6757f16cdcaeb4f parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d813064-593d-4087-8187-3ff0371901eb_isrForCrosstalkSources_855380_59.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d813064-593d-4087-8187-3ff0371901eb_isrForCrosstalkSources_855380_59.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 6208d7408e8e0a64e81e9f4511e0287c parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b42578a2-70c8-4e10-863e-c9da0a1f958f_isrForCrosstalkSources_855457_24.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b42578a2-70c8-4e10-863e-c9da0a1f958f_isrForCrosstalkSources_855457_24.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 998c39b7cb700194b71a2ea1a99bc9b4 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3f50c789-238c-4490-ba83-ef154fbfe1ab_isrForCrosstalkSources_855408_38.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3f50c789-238c-4490-ba83-ef154fbfe1ab_isrForCrosstalkSources_855408_38.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 5fe94bdd9c25887514ae73180f8c3696 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c00f8ee9-1b28-40fe-904d-26d159387e1c_isrForCrosstalkSources_855388_18.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c00f8ee9-1b28-40fe-904d-26d159387e1c_isrForCrosstalkSources_855388_18.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash c6cbd2ac20bf49e63fbdfe05ad95eacb parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c06ad283-a872-41cc-a3bb-645aac23f7b6_isrForCrosstalkSources_855380_22.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c06ad283-a872-41cc-a3bb-645aac23f7b6_isrForCrosstalkSources_855380_22.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 63082eb999790c02492c082192dad41d parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/361258bc-5db2-425a-8fa4-fd639b47f38a_isrForCrosstalkSources_855430_32.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/361258bc-5db2-425a-8fa4-fd639b47f38a_isrForCrosstalkSources_855430_32.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 15fa15e9f1ea1e10173063be0011f1d2 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c66a3270-5685-4cdd-a439-0b0e2eaf1f53_isrForCrosstalkSources_855436_27.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c66a3270-5685-4cdd-a439-0b0e2eaf1f53_isrForCrosstalkSources_855436_27.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 5e68bebeebad790e962cf580e941d985 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/431e4453-cbf6-460e-ae20-c8bdf45ed5f5_isrForCrosstalkSources_855459_17.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/431e4453-cbf6-460e-ae20-c8bdf45ed5f5_isrForCrosstalkSources_855459_17.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 7ebb9ebdb8cddfbbe9f7ae370b83241e parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/9554fd02-8269-484d-bb25-4d92e11143ef_isrForCrosstalkSources_855429_17.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/9554fd02-8269-484d-bb25-4d92e11143ef_isrForCrosstalkSources_855429_17.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 192362c3a9dd6f5e006dee4a88786120 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be2b72bb-93e9-4e2e-9da6-99bf342594e5_isrForCrosstalkSources_855425_16.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be2b72bb-93e9-4e2e-9da6-99bf342594e5_isrForCrosstalkSources_855425_16.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash e7ede6bd9f84631d3196be66d7c67c95 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/11299445-a97b-49bd-aeed-49571f17b89e_isrForCrosstalkSources_855450_16.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/11299445-a97b-49bd-aeed-49571f17b89e_isrForCrosstalkSources_855450_16.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash fb8daf38235073560903f778bc0bdec1 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/286e38d4-aece-4b84-b69a-df21c0302a72_isrForCrosstalkSources_855453_41.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/286e38d4-aece-4b84-b69a-df21c0302a72_isrForCrosstalkSources_855453_41.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash cb0096d34ba2cec20e25ec4d7f2f77d3 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/20e56840-32d7-42e6-afeb-205bf415ef6c_isrForCrosstalkSources_855403_56.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/20e56840-32d7-42e6-afeb-205bf415ef6c_isrForCrosstalkSources_855403_56.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 0b2675408bf315f2c1f1a1db11b98d0b parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a04179ca-0eff-4b4e-ba43-cb487aa8ac19_isrForCrosstalkSources_855453_33.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a04179ca-0eff-4b4e-ba43-cb487aa8ac19_isrForCrosstalkSources_855453_33.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 2624183d66f8ecd7ded506be66357116 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5924b07e-c0e6-44c5-827c-82fbfdbd6081_isrForCrosstalkSources_855454_24.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5924b07e-c0e6-44c5-827c-82fbfdbd6081_isrForCrosstalkSources_855454_24.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 9c3073e66221165daaac671b6422cfde parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e6089ac2-c25b-4691-8e08-66c412fe5201_isrForCrosstalkSources_855433_2.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e6089ac2-c25b-4691-8e08-66c412fe5201_isrForCrosstalkSources_855433_2.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 514597f0fe96bf62b63983a7dcef2c63 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b133a2c6-2f8c-4caf-aec3-135c7cf0211f_isrForCrosstalkSources_855423_22.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b133a2c6-2f8c-4caf-aec3-135c7cf0211f_isrForCrosstalkSources_855423_22.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash d8ebfbe40efc34b8f76d8e16c6028b34 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/57f871ff-f46b-4167-9982-6a3d74027b7f_isrForCrosstalkSources_855456_40.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/57f871ff-f46b-4167-9982-6a3d74027b7f_isrForCrosstalkSources_855456_40.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 1d2cd0938e34da127eedb0ba9d89e6fc parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/95b0323a-2656-4f78-aeba-44a4111f8aec_isrForCrosstalkSources_855405_56.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/95b0323a-2656-4f78-aeba-44a4111f8aec_isrForCrosstalkSources_855405_56.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 4877de6b68fdcda28e293daf13e78f30 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e4a5d1b3-2875-40f9-93a5-7dd8cacdfed8_isrForCrosstalkSources_855398_33.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e4a5d1b3-2875-40f9-93a5-7dd8cacdfed8_isrForCrosstalkSources_855398_33.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash e2194659c97ee49fd308da982731ede4 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/137f03c5-ce71-41a1-a78d-6723b06779cd_isrForCrosstalkSources_855418_5.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/137f03c5-ce71-41a1-a78d-6723b06779cd_isrForCrosstalkSources_855418_5.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash c3ce7d2c45f6db38afcef4a9c7b68f04 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f14284f8-8630-4d86-8816-30c4393a1a76_isrForCrosstalkSources_855381_39.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f14284f8-8630-4d86-8816-30c4393a1a76_isrForCrosstalkSources_855381_39.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 955653f72d87415974831d36980c27e6 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dc0b69b8-bba5-449d-8991-bb80fa7161db_isrForCrosstalkSources_855447_39.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dc0b69b8-bba5-449d-8991-bb80fa7161db_isrForCrosstalkSources_855447_39.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash efe87e2eff1562c016a8c4a3acd1b325 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f4c01a5c-745e-4933-a021-826db741bd2b_isrForCrosstalkSources_855439_27.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f4c01a5c-745e-4933-a021-826db741bd2b_isrForCrosstalkSources_855439_27.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash aa65d51c9a6115f8433bc1164df842f7 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5015968a-c083-4106-804d-5d316be1febf_isrForCrosstalkSources_855397_38.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5015968a-c083-4106-804d-5d316be1febf_isrForCrosstalkSources_855397_38.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash b6630bbfb01314696996f28604389174 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a47b955-1676-47b4-b2d1-a9303368c958_isrForCrosstalkSources_855425_13.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a47b955-1676-47b4-b2d1-a9303368c958_isrForCrosstalkSources_855425_13.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash feb756b438752c33e6819125ac52dbbd parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:53 proc_lsst.multi:170 [INFO] [multi] job_id 16964032 proc_lsst.multi INFO: [multi] job_id 16964032 2024-03-09 14:49:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 14:49:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964032 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964032 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16964032 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-09 14:49:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-09 14:49:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 14:49:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e24d56e0-09c2-4f9a-ab42-cdd20bb78a00_isrForCrosstalkSources_855391_56.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e24d56e0-09c2-4f9a-ab42-cdd20bb78a00_isrForCrosstalkSources_855391_56.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 188feb473dc555cea717b1d64d4e6d49 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/58369c1e-e131-4696-8d47-fa956fd0d244_isrForCrosstalkSources_855399_2.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/58369c1e-e131-4696-8d47-fa956fd0d244_isrForCrosstalkSources_855399_2.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 8e4ede53bb3b4f925217848b4c4a6702 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9eb8d864-b490-4f61-929f-549eef900be5_isrForCrosstalkSources_855423_4.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9eb8d864-b490-4f61-929f-549eef900be5_isrForCrosstalkSources_855423_4.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 5556012ada06c9e935a2a9f8dd843c64 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/fe631fb3-fa1e-4f2c-9b75-87af4ab60d78_isrForCrosstalkSources_855398_62.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/fe631fb3-fa1e-4f2c-9b75-87af4ab60d78_isrForCrosstalkSources_855398_62.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 427a46e58184dcf0cb3feda67b9fc2e5 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/060ef822-6bb3-4c97-bff2-8c4b33352262_isrForCrosstalkSources_855413_45.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/060ef822-6bb3-4c97-bff2-8c4b33352262_isrForCrosstalkSources_855413_45.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 459e16b891afa56557ebd128d8fc67a5 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7b325447-964a-47a6-ab13-69109dcac9cd_isrForCrosstalkSources_855453_11.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7b325447-964a-47a6-ab13-69109dcac9cd_isrForCrosstalkSources_855453_11.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 42a02dd1e7868134932bbf2865677dec parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bc4d8770-bb2a-4fb5-9040-ec4e67d8894c_isrForCrosstalkSources_855458_22.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bc4d8770-bb2a-4fb5-9040-ec4e67d8894c_isrForCrosstalkSources_855458_22.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash bedb98c057c7c392e4eeae7e85c88289 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/34eecb6e-f21c-4bc9-b2cf-3e8ad526a8d0_isrForCrosstalkSources_855380_52.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/34eecb6e-f21c-4bc9-b2cf-3e8ad526a8d0_isrForCrosstalkSources_855380_52.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 0958a97cce6f8479ea5ede8f4fdd2d83 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fa180342-ae9d-4aa2-a184-c907040f31db_isrForCrosstalkSources_855438_37.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fa180342-ae9d-4aa2-a184-c907040f31db_isrForCrosstalkSources_855438_37.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash b26484a10e3486caf541814daa047a0d parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9104e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/aa969765-d7d5-4f02-bcfd-8eab2fb98ef1_isrForCrosstalkSources_855391_13.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/aa969765-d7d5-4f02-bcfd-8eab2fb98ef1_isrForCrosstalkSources_855391_13.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 65d40061bc617011dd77b76b060bf6d1 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/698f92cb-cfc1-406a-bbf6-dbcebe9283c6_isrForCrosstalkSources_855445_37.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/698f92cb-cfc1-406a-bbf6-dbcebe9283c6_isrForCrosstalkSources_855445_37.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 520bf8664b6bd9f558202020f63ba801 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a91053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/63a095ff-6d9a-4baf-8f97-ae8cc0e3a9a2_isrForCrosstalkSources_855379_14.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/63a095ff-6d9a-4baf-8f97-ae8cc0e3a9a2_isrForCrosstalkSources_855379_14.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 2713aa7935b77e7d673e3c5f2ed13581 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/289d0cef-335c-4b6e-9b9e-cafe867aabc4_isrForCrosstalkSources_855433_38.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/289d0cef-335c-4b6e-9b9e-cafe867aabc4_isrForCrosstalkSources_855433_38.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 89aa57d617d2c16e30a122a9b6dee7ca parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/085d84ee-6df1-4a03-98f4-d9596201b736_isrForCrosstalkSources_855391_34.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/085d84ee-6df1-4a03-98f4-d9596201b736_isrForCrosstalkSources_855391_34.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash d4ab5006ba370cdbc00fdecf641daadb parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/48953295-bdb2-4e40-8cab-0c60644a9554_isrForCrosstalkSources_855435_48.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/48953295-bdb2-4e40-8cab-0c60644a9554_isrForCrosstalkSources_855435_48.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash d0692dad9289f650bea5c5ecd8a804fd parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/e2a0305c-3514-4973-bf1a-47026e94bb99_isrForCrosstalkSources_855452_6.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/e2a0305c-3514-4973-bf1a-47026e94bb99_isrForCrosstalkSources_855452_6.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 71c91f15103d3f8c25121c7f7759a759 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0d494811-b4d8-4e66-9a46-d3d13c04143b_isrForCrosstalkSources_855451_33.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0d494811-b4d8-4e66-9a46-d3d13c04143b_isrForCrosstalkSources_855451_33.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash bb9f9d1e749556e39520849d349cdf40 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/64f33b78-2366-47b5-8b7b-f9eeac3bc9b1_isrForCrosstalkSources_855454_32.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/64f33b78-2366-47b5-8b7b-f9eeac3bc9b1_isrForCrosstalkSources_855454_32.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 4afc1a0a14518eda4f4798f7c9302480 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6eb8134e-d231-4227-992b-baf1ac987625_isrForCrosstalkSources_855453_5.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6eb8134e-d231-4227-992b-baf1ac987625_isrForCrosstalkSources_855453_5.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 4de3acd361dbcce3058416d1ddd111fa parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/74b480ce-26bc-4fa7-b2e0-a0d60a3e8a28_isrForCrosstalkSources_855452_7.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/74b480ce-26bc-4fa7-b2e0-a0d60a3e8a28_isrForCrosstalkSources_855452_7.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 081c49cc880d3ca4a81b028f240b8213 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/a3946805-3bb5-4b82-bf46-6c0113e5ab4b_isrForCrosstalkSources_855417_50.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/a3946805-3bb5-4b82-bf46-6c0113e5ab4b_isrForCrosstalkSources_855417_50.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 5bc7a01f21a7149d158a701df151f5ba parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5a7ade19-74de-4c0a-b187-85de3a59272b_isrForCrosstalkSources_855449_57.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5a7ade19-74de-4c0a-b187-85de3a59272b_isrForCrosstalkSources_855449_57.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash f7003d70f5c9a65592682ee0c071b02f parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/72e6ea9d-75e9-4c12-9db9-a60236e226e8_isrForCrosstalkSources_855384_51.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/72e6ea9d-75e9-4c12-9db9-a60236e226e8_isrForCrosstalkSources_855384_51.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 1ccd8358dda532b40f6a44d60c137b06 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/aceb60a0-40b5-424e-b3b7-bbae4694f4fc_isrForCrosstalkSources_855443_8.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/aceb60a0-40b5-424e-b3b7-bbae4694f4fc_isrForCrosstalkSources_855443_8.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 81aa09f8c93627c81995c2c50448e2ed parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c0acadf4-497d-41ff-a992-e8bc83427770_isrForCrosstalkSources_855399_6.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c0acadf4-497d-41ff-a992-e8bc83427770_isrForCrosstalkSources_855399_6.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 2029458be2114a30436cf0dac9076844 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a530cf7-984d-4135-afbd-51c35189fdee_isrForCrosstalkSources_855393_2.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a530cf7-984d-4135-afbd-51c35189fdee_isrForCrosstalkSources_855393_2.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash c352ee5ad22e4a3474337660275d453c parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1beec3e7-7172-4b5c-a776-900fa2f8acb6_isrForCrosstalkSources_855442_38.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1beec3e7-7172-4b5c-a776-900fa2f8acb6_isrForCrosstalkSources_855442_38.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 57bcc044aeb63a86a02e65afca5c851d parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/eee6febc-9114-4cc1-be4f-3d2afda0215d_isrForCrosstalkSources_855430_38.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/eee6febc-9114-4cc1-be4f-3d2afda0215d_isrForCrosstalkSources_855430_38.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash e5862c5803815e301478befe72b0b4fc parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c54733a9-ee71-4bb8-b9bd-4f3c20d6a9b2_isrForCrosstalkSources_855443_6.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c54733a9-ee71-4bb8-b9bd-4f3c20d6a9b2_isrForCrosstalkSources_855443_6.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash bf1673bb2373590124c50dbe88abaff1 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d9a91b7d-2012-4be8-b1f9-6c44b8342830_isrForCrosstalkSources_855415_7.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d9a91b7d-2012-4be8-b1f9-6c44b8342830_isrForCrosstalkSources_855415_7.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 424f0c8e7377c2017c7e46a22ff1c4f7 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/43cbe0ca-3b78-45e4-b785-704d4ad1d878_isrForCrosstalkSources_855434_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/43cbe0ca-3b78-45e4-b785-704d4ad1d878_isrForCrosstalkSources_855434_19.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash a750ed23b47c9991f8851bbe1326acea parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/9ff45d83-3f49-4e70-853b-29c18af91409_isrForCrosstalkSources_855391_59.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/9ff45d83-3f49-4e70-853b-29c18af91409_isrForCrosstalkSources_855391_59.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash d373f80873937134308a600eae12ad9e parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1c7242b1-7008-4dc9-92a4-a9e4c3d61962_isrForCrosstalkSources_855408_16.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1c7242b1-7008-4dc9-92a4-a9e4c3d61962_isrForCrosstalkSources_855408_16.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 2f19e30309087fcc6f214bb85715d5a0 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/0f4c324b-44e5-4f5a-aa2c-235ff3dd8cdd_isrForCrosstalkSources_855399_7.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/0f4c324b-44e5-4f5a-aa2c-235ff3dd8cdd_isrForCrosstalkSources_855399_7.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash a8851fdc5710565b15a3fa299d07ef13 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8bb63a92-abaf-420d-ab13-24510e397729_isrForCrosstalkSources_855404_7.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8bb63a92-abaf-420d-ab13-24510e397729_isrForCrosstalkSources_855404_7.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash e5c15815182e643ab4fe22224fc9d3dc parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ee37b39a-d458-462d-8789-34cff3ddd614_isrForCrosstalkSources_855383_53.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ee37b39a-d458-462d-8789-34cff3ddd614_isrForCrosstalkSources_855383_53.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash c76b7a502b2e2f88ada48d86b7245cb2 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b79689f0-c4e3-4ea2-99c7-752ecc739288_isrForCrosstalkSources_855433_22.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b79689f0-c4e3-4ea2-99c7-752ecc739288_isrForCrosstalkSources_855433_22.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 3e5ede7e49345bb5afc8682733eea174 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b1c57994-424a-4836-8f69-12504cc8c604_isrForCrosstalkSources_855379_17.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b1c57994-424a-4836-8f69-12504cc8c604_isrForCrosstalkSources_855379_17.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash b73f23fd04044f4721d7b4032e23dcdd parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/de7f6216-cf0d-47ff-96ab-a0baa21e47b3_isrForCrosstalkSources_855435_31.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/de7f6216-cf0d-47ff-96ab-a0baa21e47b3_isrForCrosstalkSources_855435_31.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 95f75fb9a97c90dfcc00485088facd15 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3223be1e-cdbd-484b-8b3c-98ca3e0e4582_isrForCrosstalkSources_855395_26.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3223be1e-cdbd-484b-8b3c-98ca3e0e4582_isrForCrosstalkSources_855395_26.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 89d7be050fd7b3f434dc9af0f262694d parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/cecb02c7-a7ae-460c-a202-c144aeca5753_isrForCrosstalkSources_855381_16.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/cecb02c7-a7ae-460c-a202-c144aeca5753_isrForCrosstalkSources_855381_16.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 6a0ee6d0648571c120cc969e524e828c parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/b776f1ac-0eb7-4ea2-ba3f-49c05e7a12ab_isrForCrosstalkSources_855445_28.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/b776f1ac-0eb7-4ea2-ba3f-49c05e7a12ab_isrForCrosstalkSources_855445_28.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 9fa57f16e685b53e7076993f4cd5eeb5 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2694c71d-bc8d-4ebc-b11b-75dddba2920b_isrForCrosstalkSources_855444_18.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2694c71d-bc8d-4ebc-b11b-75dddba2920b_isrForCrosstalkSources_855444_18.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 4525e0122ff6ee7fabce7a16553cdc1d parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4a556679-8a81-41ec-971e-2e7080e5f88b_isrForCrosstalkSources_855412_14.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4a556679-8a81-41ec-971e-2e7080e5f88b_isrForCrosstalkSources_855412_14.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 18e05027c83b9a9e7301f8d1362b5ac6 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/702e16f8-52b0-43f1-8bbe-60be4478685c_isrForCrosstalkSources_855389_2.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/702e16f8-52b0-43f1-8bbe-60be4478685c_isrForCrosstalkSources_855389_2.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash ec0712062872770a19939e37b2db94a1 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/85eb9b59-7bfa-414a-9bf7-ba3340e57d94_isrForCrosstalkSources_855438_5.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/85eb9b59-7bfa-414a-9bf7-ba3340e57d94_isrForCrosstalkSources_855438_5.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash a2614b24a8fc188f2198da36c05112c6 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ddc34dc5-914c-4fea-bda2-37e5218d0882_isrForCrosstalkSources_855379_35.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ddc34dc5-914c-4fea-bda2-37e5218d0882_isrForCrosstalkSources_855379_35.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 0554d703423165cca0a66918da104ab3 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/36608357-0891-482d-8532-d79e5a714698_isrForCrosstalkSources_855406_18.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/36608357-0891-482d-8532-d79e5a714698_isrForCrosstalkSources_855406_18.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 83f264537a7bd7ae55822499ad7b2525 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/30a43159-a185-4b27-874f-ebd509cbaec9_isrForCrosstalkSources_855423_44.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/30a43159-a185-4b27-874f-ebd509cbaec9_isrForCrosstalkSources_855423_44.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 3c6035ec823e2428e07d625c4efa0ee7 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dde47ee4-51bd-429e-9cdd-1d6eafe772e9_isrForCrosstalkSources_855458_32.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dde47ee4-51bd-429e-9cdd-1d6eafe772e9_isrForCrosstalkSources_855458_32.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 5a52d186d772819871bc9dd27875ab3e parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a8e320e-018e-417f-b24a-271456defc16_isrForCrosstalkSources_855452_2.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a8e320e-018e-417f-b24a-271456defc16_isrForCrosstalkSources_855452_2.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash b1977ba673ccbdcfe999cad1528e0bc2 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b4ea3c07-95e4-414a-acb9-06572ff77d9c_isrForCrosstalkSources_855442_21.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b4ea3c07-95e4-414a-acb9-06572ff77d9c_isrForCrosstalkSources_855442_21.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 6d33957d6be096836f825b1eb06b0b65 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e1784fd8-e44f-4691-a5ac-d7a819c5d35b_isrForCrosstalkSources_855402_56.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e1784fd8-e44f-4691-a5ac-d7a819c5d35b_isrForCrosstalkSources_855402_56.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 66abb658593a5341bea37b59b43b231c parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0c9bfa7d-3230-412c-ae98-3ffdeb977eb8_isrForCrosstalkSources_855444_54.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0c9bfa7d-3230-412c-ae98-3ffdeb977eb8_isrForCrosstalkSources_855444_54.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash aa3852712cf4d86f8a18430579f97f79 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/91fd216a-f8f1-433e-8004-d0e5f3f16e0c_isrForCrosstalkSources_855449_11.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/91fd216a-f8f1-433e-8004-d0e5f3f16e0c_isrForCrosstalkSources_855449_11.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 544c97ed733d5a098ff4e85370daf40f parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53553d90-274e-4115-9d38-6d47566a7f6a_isrForCrosstalkSources_855389_47.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53553d90-274e-4115-9d38-6d47566a7f6a_isrForCrosstalkSources_855389_47.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 8e8eb0031d5a7030454dc900fc115571 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e64fffb0-e180-4ff4-8c2c-e04278ac885e_isrForCrosstalkSources_855403_17.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e64fffb0-e180-4ff4-8c2c-e04278ac885e_isrForCrosstalkSources_855403_17.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 2df2b91aea0f26254a5115ebde8480bc parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2baaa614-7be9-4456-8a0c-1bf632f4e57c_isrForCrosstalkSources_855424_11.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2baaa614-7be9-4456-8a0c-1bf632f4e57c_isrForCrosstalkSources_855424_11.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 281c23e0ef7191627f5df6115f7d7956 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8772ef1d-239c-465b-966c-37bd15c1d545_isrForCrosstalkSources_855397_28.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8772ef1d-239c-465b-966c-37bd15c1d545_isrForCrosstalkSources_855397_28.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash cbb7637020bbadd6c179bd3a0f2ea3b1 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/169f296c-b55c-47ff-b5b1-3a08b9774840_isrForCrosstalkSources_855406_55.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/169f296c-b55c-47ff-b5b1-3a08b9774840_isrForCrosstalkSources_855406_55.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 6fc30f673a92203d27b3c9633d6e1ca0 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e70c7127-6e14-49fd-b03e-f1ea0f7e381f_isrForCrosstalkSources_855418_18.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e70c7127-6e14-49fd-b03e-f1ea0f7e381f_isrForCrosstalkSources_855418_18.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 7c3425973303f5c7498a82d2ad5504ec parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/17e1e1b8-3fc3-4ca4-978e-01faa596a326_isrForCrosstalkSources_855438_12.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/17e1e1b8-3fc3-4ca4-978e-01faa596a326_isrForCrosstalkSources_855438_12.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 7d9bce13b2346dcaf09490079b1a7fff parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/583c0134-f8ca-4b5e-9b1d-cd913ebbad53_isrForCrosstalkSources_855406_15.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/583c0134-f8ca-4b5e-9b1d-cd913ebbad53_isrForCrosstalkSources_855406_15.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 8f71c5fa7f36fdd08f6cc24564ecf393 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/6c7e4911-4864-4659-a2ee-c7f0bb47d57e_isrForCrosstalkSources_855435_15.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/6c7e4911-4864-4659-a2ee-c7f0bb47d57e_isrForCrosstalkSources_855435_15.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 30864e491b6f937bfa46354efe834a1d parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f0795439-b226-474f-9d9b-be54370e4dba_isrForCrosstalkSources_855429_2.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f0795439-b226-474f-9d9b-be54370e4dba_isrForCrosstalkSources_855429_2.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 6077f8728d73b33dd9b4f6fc1a17cf16 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e7a890b9-cc29-4b33-b663-d6c093ccae9d_isrForCrosstalkSources_855428_13.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e7a890b9-cc29-4b33-b663-d6c093ccae9d_isrForCrosstalkSources_855428_13.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 464f0092d5061f1939676469d4ff31cc parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b80468c8-8547-4f54-9de0-a65e6b020da9_isrForCrosstalkSources_855421_55.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b80468c8-8547-4f54-9de0-a65e6b020da9_isrForCrosstalkSources_855421_55.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 9d571a0c4a59ff720fd7d083045976dd parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4863cfa-6646-4d7e-959d-a0d385721c62_isrForCrosstalkSources_855434_18.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4863cfa-6646-4d7e-959d-a0d385721c62_isrForCrosstalkSources_855434_18.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 6e7bc506cbd462ea07a2b2ebfadb0d93 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/574e3960-6912-48b5-a42d-a82acebe42c2_isrForCrosstalkSources_855433_34.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/574e3960-6912-48b5-a42d-a82acebe42c2_isrForCrosstalkSources_855433_34.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash f4a7cb0b430a8ad7c43df9df43e840a7 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e60b8503-494d-4b25-8d0e-0d4f37ee2f23_isrForCrosstalkSources_855412_43.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e60b8503-494d-4b25-8d0e-0d4f37ee2f23_isrForCrosstalkSources_855412_43.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash e3237caa1af44e66fbbf044678586e03 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/32cc831c-a1e1-4e12-9f15-9e8dd35617f1_isrForCrosstalkSources_855390_30.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/32cc831c-a1e1-4e12-9f15-9e8dd35617f1_isrForCrosstalkSources_855390_30.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 48e4bd002fd164017043fc65a6ec2052 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ccb0799b-565d-4b71-9800-f6b394a3acd7_isrForCrosstalkSources_855455_53.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ccb0799b-565d-4b71-9800-f6b394a3acd7_isrForCrosstalkSources_855455_53.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 6ffe1a7d9a0e13a3ebf6cfcfb7e96f3d parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e3b90e00-db19-47c1-b4d5-252bd2b14a56_isrForCrosstalkSources_855419_15.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e3b90e00-db19-47c1-b4d5-252bd2b14a56_isrForCrosstalkSources_855419_15.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash a5e4a73ec795330d267ee6ae458f0ba0 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2c325436-9b4d-493b-9a95-b9048512beba_isrForCrosstalkSources_855393_56.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2c325436-9b4d-493b-9a95-b9048512beba_isrForCrosstalkSources_855393_56.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 50bb850727873d92d1a7f58dc435651c parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1418cdd2-8cad-44ce-a65c-0e430102e229_isrForCrosstalkSources_855454_45.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1418cdd2-8cad-44ce-a65c-0e430102e229_isrForCrosstalkSources_855454_45.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash de3a5d9a0721d2436ebf8ecc4d8819b5 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/06fb6fd3-77c8-4245-b22d-7082dfaaf675_isrForCrosstalkSources_855411_51.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/06fb6fd3-77c8-4245-b22d-7082dfaaf675_isrForCrosstalkSources_855411_51.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 7e1e5fd629baf861d33923f6f99d9552 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/299ecb10-15b2-4e70-964a-4bed5dc61525_isrForCrosstalkSources_855426_43.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/299ecb10-15b2-4e70-964a-4bed5dc61525_isrForCrosstalkSources_855426_43.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 9669f7e48e1a76c23199b852528bf6e1 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/25a8a36a-6870-405a-8501-8fb9c89085d8_isrForCrosstalkSources_855444_3.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/25a8a36a-6870-405a-8501-8fb9c89085d8_isrForCrosstalkSources_855444_3.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash f50138552c28c9f921ccfc4e70d30483 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1ce20eeb-3b09-4a80-b85a-104360ec34a8_isrForCrosstalkSources_855423_25.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1ce20eeb-3b09-4a80-b85a-104360ec34a8_isrForCrosstalkSources_855423_25.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 1e94c3a92c9157f10da8c3af2f451832 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/49e0f6e4-2402-4939-aceb-4d26ef8bbeda_isrForCrosstalkSources_855444_32.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/49e0f6e4-2402-4939-aceb-4d26ef8bbeda_isrForCrosstalkSources_855444_32.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 8f3f478a9680073fe93e81943826c332 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a2320da4-3305-48dd-90db-fd961d1341b2_isrForCrosstalkSources_855388_10.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a2320da4-3305-48dd-90db-fd961d1341b2_isrForCrosstalkSources_855388_10.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash b908de86fb7cae4590396a70e2416e44 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d3a8d4f8-e2c3-4c39-a066-ec870b0826d7_isrForCrosstalkSources_855417_38.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d3a8d4f8-e2c3-4c39-a066-ec870b0826d7_isrForCrosstalkSources_855417_38.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 844dfe77f40dd659307d26deed9e463d parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6f3bc0c5-5347-4c3b-8735-493f809a26e2_isrForCrosstalkSources_855424_40.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6f3bc0c5-5347-4c3b-8735-493f809a26e2_isrForCrosstalkSources_855424_40.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash e77f8bb675fa1a1003c89d0604bea3b1 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b7fef9d0-aa1c-492e-b1b7-f3d241633f3a_isrForCrosstalkSources_855433_40.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b7fef9d0-aa1c-492e-b1b7-f3d241633f3a_isrForCrosstalkSources_855433_40.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash ef9e7e6f96a5d947ed23413b5f3b586d parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/af9b66f0-fd96-47aa-937f-8a239544cb65_isrForCrosstalkSources_855394_47.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/af9b66f0-fd96-47aa-937f-8a239544cb65_isrForCrosstalkSources_855394_47.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 7eb6c150dadbd1efb281545cef132057 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/56b31609-dd26-4eb9-89ae-75b246fa1306_isrForCrosstalkSources_855440_23.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/56b31609-dd26-4eb9-89ae-75b246fa1306_isrForCrosstalkSources_855440_23.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash a6063bae3650e39fc89c8adcdcc9d5c6 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bdf00d32-9c37-4465-8eb2-947ec874e591_isrForCrosstalkSources_855438_36.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bdf00d32-9c37-4465-8eb2-947ec874e591_isrForCrosstalkSources_855438_36.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 6b8ca94a726d8d9fec60b0ee8d52bc69 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/121921d7-d0e7-4ab2-b11f-2ba15ee1b354_isrForCrosstalkSources_855386_13.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/121921d7-d0e7-4ab2-b11f-2ba15ee1b354_isrForCrosstalkSources_855386_13.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 27638ebe8ccaf02cff667900efaf1323 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/80d98ee8-6872-4446-a050-fe6c562fe69d_isrForCrosstalkSources_855428_20.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/80d98ee8-6872-4446-a050-fe6c562fe69d_isrForCrosstalkSources_855428_20.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 783bc7e7892fcb9dc9c227dbf8ca82a2 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3943638-5ea3-4aa4-8e1c-008b7474b3ab_isrForCrosstalkSources_855379_51.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3943638-5ea3-4aa4-8e1c-008b7474b3ab_isrForCrosstalkSources_855379_51.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash f9606a946a0dcec590f31a341928d7f8 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/718b5130-436b-4873-bd34-bcde617e454c_isrForCrosstalkSources_855423_8.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/718b5130-436b-4873-bd34-bcde617e454c_isrForCrosstalkSources_855423_8.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 3e9f4c4ba8541076cfb9ecb7a5d8faea parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f6171bf4-b2ac-4b1f-93d9-a2ecf9c62fd3_isrForCrosstalkSources_855430_52.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f6171bf4-b2ac-4b1f-93d9-a2ecf9c62fd3_isrForCrosstalkSources_855430_52.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 20b2befd526cab64f6ae4b397407e6e8 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e7abd982-e230-4d71-a2e9-ced546f99d4c_isrForCrosstalkSources_855394_52.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e7abd982-e230-4d71-a2e9-ced546f99d4c_isrForCrosstalkSources_855394_52.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 3987e25239bcad74de544db555181f69 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ae1e9950-8237-4a2b-8e31-ddb1a00db78d_isrForCrosstalkSources_855395_37.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ae1e9950-8237-4a2b-8e31-ddb1a00db78d_isrForCrosstalkSources_855395_37.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 368358961223256aaba4eae45f25c964 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a39000-6584-4e75-b414-bed1e5929546_isrForCrosstalkSources_855391_32.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a39000-6584-4e75-b414-bed1e5929546_isrForCrosstalkSources_855391_32.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash d4c83238742b37372125b71bef18b3fe parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b8573cee-4239-497c-a68d-5327ad98bd14_isrForCrosstalkSources_855447_61.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b8573cee-4239-497c-a68d-5327ad98bd14_isrForCrosstalkSources_855447_61.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash a8668ab71e686958ec57e76c9e33f8a8 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/af238054-654c-452d-824f-2599a615df9a_isrForCrosstalkSources_855388_3.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/af238054-654c-452d-824f-2599a615df9a_isrForCrosstalkSources_855388_3.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 5e7600a25273f04e3b3407bc1d5782be parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/307da723-04e8-48bb-9893-c148a3dd434c_isrForCrosstalkSources_855448_14.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/307da723-04e8-48bb-9893-c148a3dd434c_isrForCrosstalkSources_855448_14.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 3f2ac01d984c8f9d06c5e0accc8b5965 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/088f26e9-0d45-45d2-aa89-cf07dccc5aa8_isrForCrosstalkSources_855389_20.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/088f26e9-0d45-45d2-aa89-cf07dccc5aa8_isrForCrosstalkSources_855389_20.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 6365863928f12e09a5efc1a0f416130e parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/80e58bc6-5d86-4ef8-ae5b-c7aa7bf3915f_isrForCrosstalkSources_855451_54.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/80e58bc6-5d86-4ef8-ae5b-c7aa7bf3915f_isrForCrosstalkSources_855451_54.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash c3f2879a5eef89228302444c59092403 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/c40875d3-1bc5-4167-903d-03377f8a7dd8_isrForCrosstalkSources_855432_4.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/c40875d3-1bc5-4167-903d-03377f8a7dd8_isrForCrosstalkSources_855432_4.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 7790b466921808dc55b3b376907f20b2 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/21605bcd-c832-424f-b65f-f7758b1bf6bf_isrForCrosstalkSources_855442_22.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/21605bcd-c832-424f-b65f-f7758b1bf6bf_isrForCrosstalkSources_855442_22.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 028ead258ecd0cff95ffadd55917bbd9 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2ee48f71-9365-48e9-8be2-9121c56441f2_isrForCrosstalkSources_855456_24.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2ee48f71-9365-48e9-8be2-9121c56441f2_isrForCrosstalkSources_855456_24.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 1cb3d4e79b207fc85d5170759893dd3c parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/14ad085b-3874-454e-ad7e-5cd5217bdee7_isrForCrosstalkSources_855446_45.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/14ad085b-3874-454e-ad7e-5cd5217bdee7_isrForCrosstalkSources_855446_45.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash cd748764c0563b62fe94c3d1b009eff5 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/129c15a2-6e9f-4f6e-86c2-e53967438ded_isrForCrosstalkSources_855447_41.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/129c15a2-6e9f-4f6e-86c2-e53967438ded_isrForCrosstalkSources_855447_41.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 84480d7ac5308cf443dc906dbbb4866d parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f3420479-8c87-4e17-afb1-40894225986b_isrForCrosstalkSources_855455_26.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f3420479-8c87-4e17-afb1-40894225986b_isrForCrosstalkSources_855455_26.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 20de5cf8d14c507a693751ef544aec00 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/57f32464-29e3-4bbe-94df-2b266295ab5b_isrForCrosstalkSources_855389_22.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/57f32464-29e3-4bbe-94df-2b266295ab5b_isrForCrosstalkSources_855389_22.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 8ff89e617bf81ad574596d1fbbfe9c69 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d292ccb3-4e86-419e-b3a2-a9206638fc45_isrForCrosstalkSources_855401_23.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d292ccb3-4e86-419e-b3a2-a9206638fc45_isrForCrosstalkSources_855401_23.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 98139a98ec93715794f070687dbec35c parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e5c8368c-520c-4428-bb0c-248a62dee7f3_isrForCrosstalkSources_855412_52.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e5c8368c-520c-4428-bb0c-248a62dee7f3_isrForCrosstalkSources_855412_52.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 8cf55acabdbbd83a8cb815d6e08d728b parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/531137ee-3e7d-4b22-83ea-14add9143058_isrForCrosstalkSources_855443_49.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/531137ee-3e7d-4b22-83ea-14add9143058_isrForCrosstalkSources_855443_49.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash ba6bbc0824eea9885e6023b83ebd9551 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/be6f1ec2-182e-40b6-b098-9da8560d839e_isrForCrosstalkSources_855408_15.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/be6f1ec2-182e-40b6-b098-9da8560d839e_isrForCrosstalkSources_855408_15.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 4f8fadb304f065225a30b321f7896bc1 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a95d7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032f1fbd-4781-41d2-9ee1-b1473b1c7aee_isrForCrosstalkSources_855414_30.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032f1fbd-4781-41d2-9ee1-b1473b1c7aee_isrForCrosstalkSources_855414_30.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash ff49e44efddc67966f64d2f2d2c5cfcb parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0731c929-fff3-46b1-ad43-3f7e2e8153fb_isrForCrosstalkSources_855457_7.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0731c929-fff3-46b1-ad43-3f7e2e8153fb_isrForCrosstalkSources_855457_7.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 0e38ec961dff4b30db488e1e278a8151 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/2ed1df3e-7474-4a2d-a853-5d098f38381b_isrForCrosstalkSources_855402_4.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/2ed1df3e-7474-4a2d-a853-5d098f38381b_isrForCrosstalkSources_855402_4.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 79e3bb4982cb1ec04cddd67973e7f351 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/869c1a48-3b3c-4a5c-81ee-5f06d17bb690_isrForCrosstalkSources_855388_8.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/869c1a48-3b3c-4a5c-81ee-5f06d17bb690_isrForCrosstalkSources_855388_8.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash febe58d86de650a5dc013b570b9e3ff7 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d519d9d1-3c63-4809-a63a-bef6d32ac91c_isrForCrosstalkSources_855450_36.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d519d9d1-3c63-4809-a63a-bef6d32ac91c_isrForCrosstalkSources_855450_36.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 3672507f4b1e6abdc503e0a67637ad9a parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dd311b05-0593-4c4c-b333-ee9d94d1a709_isrForCrosstalkSources_855432_47.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dd311b05-0593-4c4c-b333-ee9d94d1a709_isrForCrosstalkSources_855432_47.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 84bd926e6e6a7c478160e69382c26f27 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/196600e6-7d29-4001-86d0-ed770af2899f_isrForCrosstalkSources_855449_7.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/196600e6-7d29-4001-86d0-ed770af2899f_isrForCrosstalkSources_855449_7.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash ffae716217ed1abfaa9b25572ac8591d parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b26adb54-6ff1-4062-b784-ef485a2c8671_isrForCrosstalkSources_855416_60.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b26adb54-6ff1-4062-b784-ef485a2c8671_isrForCrosstalkSources_855416_60.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash da8ec5cf3fd873e71730003cf34a0c33 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3be627f4-d673-4b63-a510-034c1158ae8b_isrForCrosstalkSources_855397_34.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3be627f4-d673-4b63-a510-034c1158ae8b_isrForCrosstalkSources_855397_34.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash a0006f0806362b92d546304b21db030c parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2fd31897-d9c9-4b76-b0de-3981b76fadf8_isrForCrosstalkSources_855441_19.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2fd31897-d9c9-4b76-b0de-3981b76fadf8_isrForCrosstalkSources_855441_19.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 4abd4312d6d148ba4b96760270538273 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a29152d3-845a-4cc9-b368-079f2165a343_isrForCrosstalkSources_855452_4.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a29152d3-845a-4cc9-b368-079f2165a343_isrForCrosstalkSources_855452_4.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash ff0000674fcf8c686d5c395556f50d89 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5e6776f9-1619-4fdc-b91d-246ba3a20ac0_isrForCrosstalkSources_855381_7.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5e6776f9-1619-4fdc-b91d-246ba3a20ac0_isrForCrosstalkSources_855381_7.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 994873518b00b2c7644709b2e92a0a4f parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dfd43642-dff0-48c6-bc2e-cb491a024352_isrForCrosstalkSources_855446_42.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dfd43642-dff0-48c6-bc2e-cb491a024352_isrForCrosstalkSources_855446_42.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash df514658f0853303aa0ab25ecc3b72a8 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/d46e3f0c-6e98-4198-bb24-063550de7cc9_isrForCrosstalkSources_855380_40.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/d46e3f0c-6e98-4198-bb24-063550de7cc9_isrForCrosstalkSources_855380_40.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 6951cf74c15619c538301d6e1e8882bc parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/22a56394-798f-493a-9450-f3d1700abfe9_isrForCrosstalkSources_855401_12.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/22a56394-798f-493a-9450-f3d1700abfe9_isrForCrosstalkSources_855401_12.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash cbda82a47c334a41722c65524270fde9 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/765a8962-0a9a-4d93-b0ed-d90a2c391d20_isrForCrosstalkSources_855400_23.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/765a8962-0a9a-4d93-b0ed-d90a2c391d20_isrForCrosstalkSources_855400_23.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 13adbd5e67788e21ad8ae75abf73d512 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/760f2261-c0a4-4a7e-a3b3-62666fb8b5cd_isrForCrosstalkSources_855458_1.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/760f2261-c0a4-4a7e-a3b3-62666fb8b5cd_isrForCrosstalkSources_855458_1.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash d67f06805f32a399f79978e9b98e0453 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7ef09968-9bcd-4e4c-86eb-626deb8c175d_isrForCrosstalkSources_855392_19.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7ef09968-9bcd-4e4c-86eb-626deb8c175d_isrForCrosstalkSources_855392_19.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 1d6b03509d79e17b2b5ab5a6a6c7a83d parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8bfe6c0d-e8e7-4224-84d8-07e2134378ac_isrForCrosstalkSources_855435_42.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8bfe6c0d-e8e7-4224-84d8-07e2134378ac_isrForCrosstalkSources_855435_42.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 47cbe44febcd804f54035466391da83c parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c1be2b2f-900e-428e-940e-8f70f174f550_isrForCrosstalkSources_855387_45.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c1be2b2f-900e-428e-940e-8f70f174f550_isrForCrosstalkSources_855387_45.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 0c2c4748f9e223e196565bff40d50f2a parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/8f52564c-9dac-4aa4-bf6e-696967ddb4b1_isrForCrosstalkSources_855457_10.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/8f52564c-9dac-4aa4-bf6e-696967ddb4b1_isrForCrosstalkSources_855457_10.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash d0097fb15f6c8e6fc5ef39736ad9cc42 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5a533f14-4490-47a4-a3f0-36330b79ccc0_isrForCrosstalkSources_855389_16.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5a533f14-4490-47a4-a3f0-36330b79ccc0_isrForCrosstalkSources_855389_16.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 3c61a920a5df611ac7296b2646e0cd5e parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6ca531a5-1d9f-4abd-aefc-20c0aaf833f6_isrForCrosstalkSources_855457_16.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6ca531a5-1d9f-4abd-aefc-20c0aaf833f6_isrForCrosstalkSources_855457_16.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash e2b3782971d35bb5b83c4ebdd87fdaf8 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14a697f5-913b-495f-b5ca-1eb9f6413efe_isrForCrosstalkSources_855433_58.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14a697f5-913b-495f-b5ca-1eb9f6413efe_isrForCrosstalkSources_855433_58.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash e6f68b3ef3d9103df948d4191b552eaf parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6cfd199c-b9e1-4e7a-9173-d20b2e001614_isrForCrosstalkSources_855411_34.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6cfd199c-b9e1-4e7a-9173-d20b2e001614_isrForCrosstalkSources_855411_34.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 295fd91dbec263e9a577b45316e3b576 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/8d3483b7-e813-4abd-b7b2-2399221eb3fd_isrForCrosstalkSources_855439_46.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/8d3483b7-e813-4abd-b7b2-2399221eb3fd_isrForCrosstalkSources_855439_46.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 792b22cf5bd305c7d60c0c20fbcac193 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3488ac9c-8425-4925-93db-80f8eea4c0da_isrForCrosstalkSources_855448_3.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3488ac9c-8425-4925-93db-80f8eea4c0da_isrForCrosstalkSources_855448_3.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash a82710866fd95f0d6e36982d62d4bdf3 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/95f65656-957b-4103-8055-64be67db81d7_isrForCrosstalkSources_855384_16.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/95f65656-957b-4103-8055-64be67db81d7_isrForCrosstalkSources_855384_16.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash c581c64af9cb2caf7432e30c43f3ec67 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/66c66b28-7048-4238-85c9-c0baa0cdf845_isrForCrosstalkSources_855456_4.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/66c66b28-7048-4238-85c9-c0baa0cdf845_isrForCrosstalkSources_855456_4.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 216c8ed93c8ff9f8bb3329e9f4058f05 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a9c208ea-3661-43ef-9518-26359e58ea7b_isrForCrosstalkSources_855383_6.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a9c208ea-3661-43ef-9518-26359e58ea7b_isrForCrosstalkSources_855383_6.stderr parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash e7d934e6026b3c09c421975d86435571 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b791784c-6f9e-498f-8b1b-57b0c9473d81_isrForCrosstalkSources_855447_37.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b791784c-6f9e-498f-8b1b-57b0c9473d81_isrForCrosstalkSources_855447_37.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash d3be5f2213aa876052a15b76ee8ffd99 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/f4fc59f8-92f8-40db-ba4a-5e1355b58312_isrForCrosstalkSources_855435_26.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/f4fc59f8-92f8-40db-ba4a-5e1355b58312_isrForCrosstalkSources_855435_26.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 658372388d90bf0ad5d67fce650f6df3 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4e6ac140-1dd7-47be-bc69-af348b0b4c26_isrForCrosstalkSources_855408_7.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4e6ac140-1dd7-47be-bc69-af348b0b4c26_isrForCrosstalkSources_855408_7.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 0acdfeb00193ea9e99bb5ec6de950463 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/ce6b283e-443b-4500-b8da-c45e6dd861fb_isrForCrosstalkSources_855393_13.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/ce6b283e-443b-4500-b8da-c45e6dd861fb_isrForCrosstalkSources_855393_13.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 91ded540258b6d684fef06f09caf08c2 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/db7934fb-bade-443b-8f4c-3401b8186833_isrForCrosstalkSources_855410_36.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/db7934fb-bade-443b-8f4c-3401b8186833_isrForCrosstalkSources_855410_36.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash dd5c2bb355e6e2f350ce1465d23cd6d2 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/585f568c-a3d4-400f-a559-fefa753b89c1_isrForCrosstalkSources_855433_61.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/585f568c-a3d4-400f-a559-fefa753b89c1_isrForCrosstalkSources_855433_61.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 2a56d47cf554ebf0f27388bcd44149b6 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/9eab1ec7-0a41-44ec-a161-ba6a981662df_isrForCrosstalkSources_855442_61.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/9eab1ec7-0a41-44ec-a161-ba6a981662df_isrForCrosstalkSources_855442_61.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash a1cf3b99872ca6e2f8c0331950cc3561 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60e80b68-faf3-43ff-9929-58e7a33dea5e_isrForCrosstalkSources_855433_60.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60e80b68-faf3-43ff-9929-58e7a33dea5e_isrForCrosstalkSources_855433_60.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 5938c2e5eaaf2ab2c99ffd4217e6056e parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/af73dbac-083e-4170-bdf5-5647681f7ec3_isrForCrosstalkSources_855413_16.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/af73dbac-083e-4170-bdf5-5647681f7ec3_isrForCrosstalkSources_855413_16.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 2ea1f1c10b1b74dbfe9dd506613a9d87 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1606d311-11b8-4f0b-bae1-c20bb531121b_isrForCrosstalkSources_855407_55.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1606d311-11b8-4f0b-bae1-c20bb531121b_isrForCrosstalkSources_855407_55.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 956a4a6e79bb4322bad50889ef855bee parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b9bfb513-e48e-4899-ac26-0cf55d94c58c_isrForCrosstalkSources_855380_1.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b9bfb513-e48e-4899-ac26-0cf55d94c58c_isrForCrosstalkSources_855380_1.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 0235b524b9a9f66004faea04c286a076 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c2e77f37-8ed9-4bd6-aada-67958e6c2ba3_isrForCrosstalkSources_855450_2.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c2e77f37-8ed9-4bd6-aada-67958e6c2ba3_isrForCrosstalkSources_855450_2.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 546b49dd1d515e0c88d7178d9f8752f6 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d07b7e89-3fcd-44ee-b8f7-f81eb2e9e2e8_isrForCrosstalkSources_855421_58.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d07b7e89-3fcd-44ee-b8f7-f81eb2e9e2e8_isrForCrosstalkSources_855421_58.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 9ba32d4e236ac3a6fafce71b1c621ac6 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/781b3f27-e102-4c93-a567-bed34f26a5d7_isrForCrosstalkSources_855452_29.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/781b3f27-e102-4c93-a567-bed34f26a5d7_isrForCrosstalkSources_855452_29.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 854b8e6fef11cf256f088eef4666aabf parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7604c41c-5da8-46b6-a7ce-e0f948aad4d4_isrForCrosstalkSources_855457_43.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7604c41c-5da8-46b6-a7ce-e0f948aad4d4_isrForCrosstalkSources_855457_43.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 37f1f838748fdee4bdd9a1e083aac9fc parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bb10a081-9383-442a-9fd2-a1a41c0de923_isrForCrosstalkSources_855417_54.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bb10a081-9383-442a-9fd2-a1a41c0de923_isrForCrosstalkSources_855417_54.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash e493a653b583ca224e07da51a71876c3 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01074f34-8615-48f4-8766-54814148430d_isrForCrosstalkSources_855417_59.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01074f34-8615-48f4-8766-54814148430d_isrForCrosstalkSources_855417_59.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 74addeb80a67d540f74f03a2060f1bec parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:55 proc_lsst.multi:170 [INFO] [multi] job_id 16964033 proc_lsst.multi INFO: [multi] job_id 16964033 2024-03-09 14:49:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 14:49:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964033 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964033 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16964033 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-09 14:49:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-09 14:49:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 14:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3006ad57-ef21-45a1-84bf-ba7cd821dd9c_isrForCrosstalkSources_855395_45.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3006ad57-ef21-45a1-84bf-ba7cd821dd9c_isrForCrosstalkSources_855395_45.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 88ca49365e697fff7c3e1e630eeb30de parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/950ce9ad-8c12-4bca-993f-b21ecfb72c3b_isrForCrosstalkSources_855409_37.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/950ce9ad-8c12-4bca-993f-b21ecfb72c3b_isrForCrosstalkSources_855409_37.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash def1b2ceb2792a4617235027afca62bc parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9107560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20df5d3c-f489-487b-ad08-0161a8c4c6e9_isrForCrosstalkSources_855419_60.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20df5d3c-f489-487b-ad08-0161a8c4c6e9_isrForCrosstalkSources_855419_60.stderr parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 7dc626c027106488a70fb33e93aeb4d1 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/76f91000-94af-4c85-98a1-38831260a687_isrForCrosstalkSources_855398_17.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/76f91000-94af-4c85-98a1-38831260a687_isrForCrosstalkSources_855398_17.stderr parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 65d758b31169eef7eae4b90fb2a80fc8 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8e24c85e-96c0-40ee-8fb7-48740df78339_isrForCrosstalkSources_855395_4.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8e24c85e-96c0-40ee-8fb7-48740df78339_isrForCrosstalkSources_855395_4.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 3966093abc4b5b1dd4315144f9980023 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a2481450-1285-4e8c-b221-477bbb741648_isrForCrosstalkSources_855391_7.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a2481450-1285-4e8c-b221-477bbb741648_isrForCrosstalkSources_855391_7.stderr parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 266b6cda3a99e37e3cfeb1793031bff6 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/00330167-0d1e-4625-ab71-208a92c859b1_isrForCrosstalkSources_855394_53.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/00330167-0d1e-4625-ab71-208a92c859b1_isrForCrosstalkSources_855394_53.stderr parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 223b2ade1ffd35428e832af51c0967b8 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e08dcc5a-b56c-4a1b-9be4-8f944091b9cb_isrForCrosstalkSources_855406_59.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e08dcc5a-b56c-4a1b-9be4-8f944091b9cb_isrForCrosstalkSources_855406_59.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 2f389f1bbfd3c979deb5980f1d7ab348 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a68de22c-4767-4831-b6ae-a4ed3b33492b_isrForCrosstalkSources_855404_49.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a68de22c-4767-4831-b6ae-a4ed3b33492b_isrForCrosstalkSources_855404_49.stderr parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash a41b2182fe4d40b7311870479e763f8f parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/360c2b89-917e-42c6-8ab2-846ea21cf4a5_isrForCrosstalkSources_855456_34.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/360c2b89-917e-42c6-8ab2-846ea21cf4a5_isrForCrosstalkSources_855456_34.stderr parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 8d8d149826a0c39a31edd9b0e030ba27 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9105a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/343ca097-18e4-48e6-a817-60aa420c43a8_isrForCrosstalkSources_855428_22.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/343ca097-18e4-48e6-a817-60aa420c43a8_isrForCrosstalkSources_855428_22.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 8322c920c7f158ad4ee56a3a2d4c9191 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/82657c2e-d87d-43c5-aba1-fb36e696ecac_isrForCrosstalkSources_855437_33.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/82657c2e-d87d-43c5-aba1-fb36e696ecac_isrForCrosstalkSources_855437_33.stderr parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash 026431b7fc131279654f5e6a558c9ee5 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/0e763dae-5063-42d6-b37b-2e6261e77448_isrForCrosstalkSources_855414_53.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/0e763dae-5063-42d6-b37b-2e6261e77448_isrForCrosstalkSources_855414_53.stderr parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 953848568fc55dc8087ed8072bb740a7 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8463841-f13e-41ae-accb-4a462ddefd53_isrForCrosstalkSources_855435_21.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8463841-f13e-41ae-accb-4a462ddefd53_isrForCrosstalkSources_855435_21.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash a5f6726729b3c0ce7e895606cd5513f0 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7f0b80e9-bece-4826-8859-167df417c9b0_isrForCrosstalkSources_855392_28.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7f0b80e9-bece-4826-8859-167df417c9b0_isrForCrosstalkSources_855392_28.stderr parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash a4d1a5475e31eb1446b96d38a76989a1 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5fa40abb-3ad6-41ee-a9e2-f1da7fd4fb81_isrForCrosstalkSources_855456_19.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5fa40abb-3ad6-41ee-a9e2-f1da7fd4fb81_isrForCrosstalkSources_855456_19.stderr parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash a90e201ad052b10f93506b4de12fcb67 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b020bf5b-dbd4-4374-a1f3-6a5d155b6f4c_isrForCrosstalkSources_855446_54.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b020bf5b-dbd4-4374-a1f3-6a5d155b6f4c_isrForCrosstalkSources_855446_54.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 09d699e546957b1792080b1cea70b48b parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/732268b6-6607-48da-8ec0-35740f9ac431_isrForCrosstalkSources_855429_19.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/732268b6-6607-48da-8ec0-35740f9ac431_isrForCrosstalkSources_855429_19.stderr parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 91736ac89fcea696d36adb3170f45d3a parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/63ee067b-971c-4e7e-856a-cd55b92c1716_isrForCrosstalkSources_855427_5.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/63ee067b-971c-4e7e-856a-cd55b92c1716_isrForCrosstalkSources_855427_5.stderr parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 8feefbb2810c6978c37cd0cde83699b4 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/5d5a616a-2dba-4065-980e-4af35f1707e3_isrForCrosstalkSources_855438_8.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/5d5a616a-2dba-4065-980e-4af35f1707e3_isrForCrosstalkSources_855438_8.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash e493d70816f9dfc4141fa7638781a6b6 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d7ed5e7b-264c-44c8-b1ce-2f247c4d64b5_isrForCrosstalkSources_855432_53.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d7ed5e7b-264c-44c8-b1ce-2f247c4d64b5_isrForCrosstalkSources_855432_53.stderr parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 0bb0be581be3cf2145bb4c9b95c59d03 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/af0ebe5d-b6cf-4c2b-a394-d98248e2a9ad_isrForCrosstalkSources_855405_15.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/af0ebe5d-b6cf-4c2b-a394-d98248e2a9ad_isrForCrosstalkSources_855405_15.stderr parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 7299218449a9c44b44fb854cb2cecc04 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/060da2a9-6b4c-4900-a594-9ae5ad1cc333_isrForCrosstalkSources_855396_31.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/060da2a9-6b4c-4900-a594-9ae5ad1cc333_isrForCrosstalkSources_855396_31.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash e73ccd0482e1f2e3fd719a9458f739fb parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f247ee06-a7db-4525-ba14-745959ca61d1_isrForCrosstalkSources_855397_5.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f247ee06-a7db-4525-ba14-745959ca61d1_isrForCrosstalkSources_855397_5.stderr parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash e0caf6b6623f699a2d6af43d02496fd8 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b4301f3a-edc0-4115-a71c-98ea59950265_isrForCrosstalkSources_855402_10.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b4301f3a-edc0-4115-a71c-98ea59950265_isrForCrosstalkSources_855402_10.stderr parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash ac4aede68f2ad8d5c66a2f7bbcb2f279 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/de115f10-d816-4469-a28f-34d9125ae7c8_isrForCrosstalkSources_855395_35.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/de115f10-d816-4469-a28f-34d9125ae7c8_isrForCrosstalkSources_855395_35.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 13ca1b3f67df0dccda0f6be93e22a72f parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84563fe8-f65f-4901-a6bb-171bbd45c31f_isrForCrosstalkSources_855416_61.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84563fe8-f65f-4901-a6bb-171bbd45c31f_isrForCrosstalkSources_855416_61.stderr parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash c14dc5651ed0a0de0eeed3f9b0eadde4 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/498b8665-c7ae-4abc-9df9-890e81f7355d_isrForCrosstalkSources_855453_60.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/498b8665-c7ae-4abc-9df9-890e81f7355d_isrForCrosstalkSources_855453_60.stderr parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 2723b9e70b8c3f853eecae403a0ea926 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/00fadcdd-9f60-4d34-a9d5-b747fe1fa1b3_isrForCrosstalkSources_855433_11.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/00fadcdd-9f60-4d34-a9d5-b747fe1fa1b3_isrForCrosstalkSources_855433_11.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 68aedc568b3afe7c5ef8d1dde4dfa691 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/51c707e1-0074-4755-94f9-03e4983f5099_isrForCrosstalkSources_855419_50.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/51c707e1-0074-4755-94f9-03e4983f5099_isrForCrosstalkSources_855419_50.stderr parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash ab5c2a2075a90b97f94147766e28e93e parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d9d72fd1-0439-4bcd-9d98-d27d0bc2acbf_isrForCrosstalkSources_855384_1.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d9d72fd1-0439-4bcd-9d98-d27d0bc2acbf_isrForCrosstalkSources_855384_1.stderr parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 71684b9ebc0592c4884ec1b0475fec20 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/215eeadf-0426-4d31-b5b7-2a37e42f21d3_isrForCrosstalkSources_855458_37.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/215eeadf-0426-4d31-b5b7-2a37e42f21d3_isrForCrosstalkSources_855458_37.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash f051779f6fa49301a0f6b15f14d82c5b parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4dc7553c-c7e5-4c91-a122-d38a7a14a6d6_isrForCrosstalkSources_855452_13.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4dc7553c-c7e5-4c91-a122-d38a7a14a6d6_isrForCrosstalkSources_855452_13.stderr parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash e0d55a5158c68766274d7f014f04888b parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5857eefa-91b9-4a80-b879-d734b006375e_isrForCrosstalkSources_855433_37.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5857eefa-91b9-4a80-b879-d734b006375e_isrForCrosstalkSources_855433_37.stderr parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash cad52d0ee83afd11b7b39fc868227f24 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/666cb698-5146-479c-bf0d-72fa52f04e7b_isrForCrosstalkSources_855391_23.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/666cb698-5146-479c-bf0d-72fa52f04e7b_isrForCrosstalkSources_855391_23.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash fed76a8dfe266515ce356626fd972d2a parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64ea3858-1eb4-43e6-823c-3ab49a43ec32_isrForCrosstalkSources_855424_5.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64ea3858-1eb4-43e6-823c-3ab49a43ec32_isrForCrosstalkSources_855424_5.stderr parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash dcc6f8ab676028039a9043f793cb69f5 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/06bb76db-4e06-4c5c-b0b1-cd792f1128a1_isrForCrosstalkSources_855438_61.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/06bb76db-4e06-4c5c-b0b1-cd792f1128a1_isrForCrosstalkSources_855438_61.stderr parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 6d8ff932bdee9b50bd7fa85bb934bd23 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/3a158af4-597b-4506-8c8c-b932d1cbe34a_isrForCrosstalkSources_855396_56.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/3a158af4-597b-4506-8c8c-b932d1cbe34a_isrForCrosstalkSources_855396_56.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 2ddea91c7dcabafa79349f6536c1f606 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/90dec67f-699c-4136-87ba-0bd2e1b881b6_isrForCrosstalkSources_855448_54.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/90dec67f-699c-4136-87ba-0bd2e1b881b6_isrForCrosstalkSources_855448_54.stderr parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 1708ead7e097ffca49829c656be685f6 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0a8a0e58-dd03-4a0f-bd76-895129a33b0f_isrForCrosstalkSources_855458_60.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0a8a0e58-dd03-4a0f-bd76-895129a33b0f_isrForCrosstalkSources_855458_60.stderr parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 716ee7d007cc59ddfdc8edd924b86809 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8460d0a5-9d8a-4ba8-95b7-17fd2aea2e2a_isrForCrosstalkSources_855382_27.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8460d0a5-9d8a-4ba8-95b7-17fd2aea2e2a_isrForCrosstalkSources_855382_27.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash c35a01a5b737f5164685da6bca8e917c parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9cd2348d-f15c-4ad3-88d9-242fb0e4b4d6_isrForCrosstalkSources_855422_32.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9cd2348d-f15c-4ad3-88d9-242fb0e4b4d6_isrForCrosstalkSources_855422_32.stderr parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash fadd7bb6b50febf6591732b9723c785b parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8ef3b5a6-5bec-441d-bcd4-1b9e36906a88_isrForCrosstalkSources_855446_37.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8ef3b5a6-5bec-441d-bcd4-1b9e36906a88_isrForCrosstalkSources_855446_37.stderr parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 87111de109bf0e98ea9f7b67b7eab32e parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f7682276-6594-40bd-a009-fbbec542057f_isrForCrosstalkSources_855444_52.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f7682276-6594-40bd-a009-fbbec542057f_isrForCrosstalkSources_855444_52.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 741705bf9f016230a12cfe74df565a05 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8f54ebc1-5fb2-4947-962a-2a970080c14f_isrForCrosstalkSources_855382_13.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8f54ebc1-5fb2-4947-962a-2a970080c14f_isrForCrosstalkSources_855382_13.stderr parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 412172a91c66bd796710392e651467f4 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a9208d1-10c0-4797-b466-5cf5d68fcee2_isrForCrosstalkSources_855434_1.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a9208d1-10c0-4797-b466-5cf5d68fcee2_isrForCrosstalkSources_855434_1.stderr parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash f80f28f7c6b11a9e1303544e63bbdb07 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/aff5adc8-48ae-479f-a817-c2e6652ece98_isrForCrosstalkSources_855393_24.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/aff5adc8-48ae-479f-a817-c2e6652ece98_isrForCrosstalkSources_855393_24.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 59746ea1850084325ed99a3b2d2ad8fd parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8530d9c9-953c-4f00-8bca-74afb74e13a6_isrForCrosstalkSources_855440_56.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8530d9c9-953c-4f00-8bca-74afb74e13a6_isrForCrosstalkSources_855440_56.stderr parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 7d418952514eb3cde4ee5675be65a146 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/214e181d-dfaa-4a17-9e20-bc96065ebb1e_isrForCrosstalkSources_855416_53.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/214e181d-dfaa-4a17-9e20-bc96065ebb1e_isrForCrosstalkSources_855416_53.stderr parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash e928dc1f1bb0a6fef49dc83b543700e0 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/1a2d0d85-ba4b-4452-9767-2188496d694d_isrForCrosstalkSources_855396_51.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/1a2d0d85-ba4b-4452-9767-2188496d694d_isrForCrosstalkSources_855396_51.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 13f14a627b64c978aa7fe8c4320a018b parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/28c04c8a-6c35-4bab-b959-f7685fae2e82_isrForCrosstalkSources_855406_49.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/28c04c8a-6c35-4bab-b959-f7685fae2e82_isrForCrosstalkSources_855406_49.stderr parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 7164d31499e5957892b1e1c2d08a8e03 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c4639f4b-85f7-4ff8-bbae-dd18a2f38bf9_isrForCrosstalkSources_855405_9.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c4639f4b-85f7-4ff8-bbae-dd18a2f38bf9_isrForCrosstalkSources_855405_9.stderr parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 253a21a8e019ae889783fbab47bec605 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c8b7f14b-5c25-4ebc-b9e1-4b445fe295e0_isrForCrosstalkSources_855425_22.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c8b7f14b-5c25-4ebc-b9e1-4b445fe295e0_isrForCrosstalkSources_855425_22.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 4a6d0dd8eef7d94f3e29cc64f1e0906e parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36fdc32f-2b20-4315-a9d3-f79158c1f342_isrForCrosstalkSources_855388_41.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36fdc32f-2b20-4315-a9d3-f79158c1f342_isrForCrosstalkSources_855388_41.stderr parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 05b5f80c5ab7c8e6b12bb2f740d403fb parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964034 proc_lsst.multi INFO: [multi] job_id 16964034 2024-03-09 14:49:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 14:49:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964034 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964034 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16964034 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-09 14:49:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-09 14:49:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 14:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dbf9db19-28a5-4114-9f2f-b1422a880327_isrForCrosstalkSources_855405_22.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dbf9db19-28a5-4114-9f2f-b1422a880327_isrForCrosstalkSources_855405_22.stderr parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash be87c50eafe29d0ce0227b1ed0e84528 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5e299dc3-92c1-4c50-9aa1-04d6bbcba966_isrForCrosstalkSources_855395_58.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5e299dc3-92c1-4c50-9aa1-04d6bbcba966_isrForCrosstalkSources_855395_58.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 0fa5c2a677872e3cd9102f6e6a24da6a parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/50d2edfb-9467-4c99-9bae-31907593a724_isrForCrosstalkSources_855455_51.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/50d2edfb-9467-4c99-9bae-31907593a724_isrForCrosstalkSources_855455_51.stderr parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 06d9ded3fdb18b33f145c4a0e4da371e parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3a85ab9-8185-4f36-9f68-bdadc56e1225_isrForCrosstalkSources_855395_12.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3a85ab9-8185-4f36-9f68-bdadc56e1225_isrForCrosstalkSources_855395_12.stderr parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 13772b8cf33bf2cddc4be3c288666568 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c78c7297-f931-4bbf-a333-cb6cb0852ddc_isrForCrosstalkSources_855421_54.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c78c7297-f931-4bbf-a333-cb6cb0852ddc_isrForCrosstalkSources_855421_54.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash c49c85db1d073a6ad28ef13002949cd2 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bf2b8cbc-a619-4aa1-a242-8e160c8f36d2_isrForCrosstalkSources_855405_58.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bf2b8cbc-a619-4aa1-a242-8e160c8f36d2_isrForCrosstalkSources_855405_58.stderr parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 4793af67401dbdf2efcafbff1d22af50 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9240df80-4da9-45b3-b96a-e32a71c4de51_isrForCrosstalkSources_855410_24.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9240df80-4da9-45b3-b96a-e32a71c4de51_isrForCrosstalkSources_855410_24.stderr parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash f2855794e3bcc0817f377bee6c480c61 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4bcfc7aa-7bcc-419f-8fa2-5b8ded007f05_isrForCrosstalkSources_855425_49.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4bcfc7aa-7bcc-419f-8fa2-5b8ded007f05_isrForCrosstalkSources_855425_49.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 274ceb8ca21f3a6d8ab9d058abc0ceb8 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/582bc923-4781-46a6-9ea1-92163c4cd6cf_isrForCrosstalkSources_855390_42.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/582bc923-4781-46a6-9ea1-92163c4cd6cf_isrForCrosstalkSources_855390_42.stderr parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 903d3d45e838e41345ee35822365948d parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/23ec2d2a-bf41-488c-b94d-0e1e5d7d47ac_isrForCrosstalkSources_855386_23.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/23ec2d2a-bf41-488c-b94d-0e1e5d7d47ac_isrForCrosstalkSources_855386_23.stderr parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 6885549b9a65d0c028fdbff38106a855 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d49098ac-6ddd-4425-8006-c9c029be6ab8_isrForCrosstalkSources_855426_12.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d49098ac-6ddd-4425-8006-c9c029be6ab8_isrForCrosstalkSources_855426_12.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 7f44942cabe7df105d2a9058a504e0c4 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0528d0ea-7381-4f0f-87b1-60e91c93139f_isrForCrosstalkSources_855386_11.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0528d0ea-7381-4f0f-87b1-60e91c93139f_isrForCrosstalkSources_855386_11.stderr parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 694bbec11854bd402520e056cde601d6 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/70b6fe76-eec1-4692-b438-94cfbe1f577a_isrForCrosstalkSources_855449_15.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/70b6fe76-eec1-4692-b438-94cfbe1f577a_isrForCrosstalkSources_855449_15.stderr parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 63bd53e7bec34e44fcb3012f0e50efc2 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67fe03fa-68dd-46bf-9c17-4ee4d6505615_isrForCrosstalkSources_855382_51.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67fe03fa-68dd-46bf-9c17-4ee4d6505615_isrForCrosstalkSources_855382_51.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 801838fe3d72759a6b149cd55cc6db2d parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/90e8385f-be2d-4b98-8779-add06850af47_isrForCrosstalkSources_855383_57.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/90e8385f-be2d-4b98-8779-add06850af47_isrForCrosstalkSources_855383_57.stderr parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 0d05ffd50accb7f144208cc974f74584 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/25da4179-1bc6-4d58-9adb-0cf0faac2163_isrForCrosstalkSources_855399_12.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/25da4179-1bc6-4d58-9adb-0cf0faac2163_isrForCrosstalkSources_855399_12.stderr parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash b2a86b67b9fa6eca592415e5634313d3 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/48870a82-7911-4958-8459-cd30fa45c15d_isrForCrosstalkSources_855443_22.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/48870a82-7911-4958-8459-cd30fa45c15d_isrForCrosstalkSources_855443_22.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash d3a1fb65e83ba1b20edf9b0ea6a24e23 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c01be294-e5ec-45b3-bc96-6ee9f3ac5e67_isrForCrosstalkSources_855410_17.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c01be294-e5ec-45b3-bc96-6ee9f3ac5e67_isrForCrosstalkSources_855410_17.stderr parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash a1829e36d50304f77a5af274bf402c87 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/666d9687-4adf-406a-bfd9-192d7dc988d1_isrForCrosstalkSources_855432_57.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/666d9687-4adf-406a-bfd9-192d7dc988d1_isrForCrosstalkSources_855432_57.stderr parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash eca5584ec8cd9ee8f9a036c837420f10 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/eea61c08-6922-4003-8650-a767a9558bf1_isrForCrosstalkSources_855403_35.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/eea61c08-6922-4003-8650-a767a9558bf1_isrForCrosstalkSources_855403_35.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 5e5fad7bc0632ebf2ad41fe7d9a17cd1 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e406454c-883c-4d73-bbf0-73a05ba3a07f_isrForCrosstalkSources_855431_51.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e406454c-883c-4d73-bbf0-73a05ba3a07f_isrForCrosstalkSources_855431_51.stderr parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash dc314c5316625692121eb03cc0ab2daf parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/43260daa-8b8d-4487-8eb1-f43ec10ee80d_isrForCrosstalkSources_855445_42.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/43260daa-8b8d-4487-8eb1-f43ec10ee80d_isrForCrosstalkSources_855445_42.stderr parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 6e07b810ad3345019171e2002056567e parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0327a516-3af2-41be-9340-bc30c24bfe6a_isrForCrosstalkSources_855454_14.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0327a516-3af2-41be-9340-bc30c24bfe6a_isrForCrosstalkSources_855454_14.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 4eb360727bfa0cb460e0ef9f612fe2c6 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9232741-4cae-49bb-9c2c-5b5a1d68be54_isrForCrosstalkSources_855458_21.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9232741-4cae-49bb-9c2c-5b5a1d68be54_isrForCrosstalkSources_855458_21.stderr parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 8aa3b26aef5912b52d2b613067b723eb parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/47d87d7c-9972-4cd6-9dbc-04eaa846a59e_isrForCrosstalkSources_855433_33.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/47d87d7c-9972-4cd6-9dbc-04eaa846a59e_isrForCrosstalkSources_855433_33.stderr parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 0d3c344bce0e7159cebc48e2ae145ea7 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b2b39faf-9c4b-4506-87db-050df079640e_isrForCrosstalkSources_855380_13.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b2b39faf-9c4b-4506-87db-050df079640e_isrForCrosstalkSources_855380_13.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 49bc0752ca57d0360a446bfea5076bcf parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e92ace5e-9aff-4b84-87f1-d1fc27fa3029_isrForCrosstalkSources_855379_38.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e92ace5e-9aff-4b84-87f1-d1fc27fa3029_isrForCrosstalkSources_855379_38.stderr parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 40083d46ce92ad6d8fc27c7e4c6b82f2 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8f6ee23-809d-442a-9543-5b8fc8046b06_isrForCrosstalkSources_855399_52.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8f6ee23-809d-442a-9543-5b8fc8046b06_isrForCrosstalkSources_855399_52.stderr parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 62b1f435396e6f70eaffe7803a954d3a parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964035 proc_lsst.multi INFO: [multi] job_id 16964035 2024-03-09 14:49:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 14:49:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964035 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964035 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16964035 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-09 14:49:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-09 14:49:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 14:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ec00687d-c6ce-440d-a39b-676388a36be1_isrForCrosstalkSources_855440_61.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ec00687d-c6ce-440d-a39b-676388a36be1_isrForCrosstalkSources_855440_61.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 340810548af47e36162b8eb4370c0ad1 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/769a141a-5f76-4e0f-8bf0-3b4ae4dd9c08_isrForCrosstalkSources_855403_28.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/769a141a-5f76-4e0f-8bf0-3b4ae4dd9c08_isrForCrosstalkSources_855403_28.stderr parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 9856b9b4e9efe10805110d4dba7b9203 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ddbe6799-1c3b-48e7-8e55-2e4d8c04453b_isrForCrosstalkSources_855381_13.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ddbe6799-1c3b-48e7-8e55-2e4d8c04453b_isrForCrosstalkSources_855381_13.stderr parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 71c76014dd8aaa16c78db73880b182be parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e955e7c3-7083-4611-bff4-6f28e9c55bec_isrForCrosstalkSources_855402_12.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e955e7c3-7083-4611-bff4-6f28e9c55bec_isrForCrosstalkSources_855402_12.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash ad263e856b19b28590e187b69acb65d8 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/af839781-1958-40fa-a6f0-6b2cf63504d3_isrForCrosstalkSources_855425_8.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/af839781-1958-40fa-a6f0-6b2cf63504d3_isrForCrosstalkSources_855425_8.stderr parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 8a8ab7281a00da5b585015741b57b39c parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3074b678-b8e6-4a6f-b774-e1e18324394c_isrForCrosstalkSources_855383_54.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3074b678-b8e6-4a6f-b774-e1e18324394c_isrForCrosstalkSources_855383_54.stderr parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash e9de30d163c9e84bb2a11d462951bf04 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0a767632-35ea-4790-8677-532071a7d052_isrForCrosstalkSources_855403_18.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0a767632-35ea-4790-8677-532071a7d052_isrForCrosstalkSources_855403_18.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash b7076bb34fd02334e17dfcae22f65f1f parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ea43d1f7-72ab-4d9d-82d2-7687178c9fb3_isrForCrosstalkSources_855430_36.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ea43d1f7-72ab-4d9d-82d2-7687178c9fb3_isrForCrosstalkSources_855430_36.stderr parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 506e42c1f81d525740cd807313d8b2d8 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eea189e5-fc1b-4cfa-9fc1-d9749f4824d6_isrForCrosstalkSources_855446_13.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eea189e5-fc1b-4cfa-9fc1-d9749f4824d6_isrForCrosstalkSources_855446_13.stderr parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 2e3be51ee76089fc7327595e77cf5e9c parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e10b9409-f8a8-4f51-b0d7-85e0b111923a_isrForCrosstalkSources_855411_18.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e10b9409-f8a8-4f51-b0d7-85e0b111923a_isrForCrosstalkSources_855411_18.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 9ca9f88f7ab0b54a5c92602d2edb2d23 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/22f075d6-b8e0-4d48-ae8c-b8f16d2222b1_isrForCrosstalkSources_855456_12.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/22f075d6-b8e0-4d48-ae8c-b8f16d2222b1_isrForCrosstalkSources_855456_12.stderr parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash ef6e26b132a4017671d4d0ee7cd1d63a parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c73e254d-06cf-431e-be32-f9ac212573d8_isrForCrosstalkSources_855458_3.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c73e254d-06cf-431e-be32-f9ac212573d8_isrForCrosstalkSources_855458_3.stderr parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash e93aa82cc639e51e885f61e966140a5b parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0835e213-e26a-49de-bb90-ffd1b7483e0b_isrForCrosstalkSources_855402_26.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0835e213-e26a-49de-bb90-ffd1b7483e0b_isrForCrosstalkSources_855402_26.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 13536d89379ec5d6c9717b86f0193cea parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0b294003-9f7e-44da-b9c8-449641df6628_isrForCrosstalkSources_855420_40.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0b294003-9f7e-44da-b9c8-449641df6628_isrForCrosstalkSources_855420_40.stderr parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 20777e4e79e756a06602f5deca6f61b0 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/122f7cc4-b143-4cfd-9c02-a2120bd92514_isrForCrosstalkSources_855412_33.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/122f7cc4-b143-4cfd-9c02-a2120bd92514_isrForCrosstalkSources_855412_33.stderr parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 73e2945a3c51917eda66262306e6ee3d parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964036 proc_lsst.multi INFO: [multi] job_id 16964036 2024-03-09 14:49:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 14:49:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964036 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964036 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16964036 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-09 14:49:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-09 14:49:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 14:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/52c3fce0-e7e8-4536-bbff-727a4a3ae340_isrForCrosstalkSources_855434_6.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/52c3fce0-e7e8-4536-bbff-727a4a3ae340_isrForCrosstalkSources_855434_6.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 96bcad031dac6c03ff3203ff3fa75150 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9db72332-4b46-4b3c-95f4-1807645132fe_isrForCrosstalkSources_855452_16.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9db72332-4b46-4b3c-95f4-1807645132fe_isrForCrosstalkSources_855452_16.stderr parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 2507c381a5fa403b033dca964b5a3e45 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ca13abe5-735f-4642-9168-8bb796485acb_isrForCrosstalkSources_855398_43.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ca13abe5-735f-4642-9168-8bb796485acb_isrForCrosstalkSources_855398_43.stderr parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 4e5689742f82c287954959ef0925ecb1 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/58840948-8d1a-40b5-b2c9-9c174d4c875b_isrForCrosstalkSources_855379_26.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/58840948-8d1a-40b5-b2c9-9c174d4c875b_isrForCrosstalkSources_855379_26.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 41c5e64a5ad12dd02a65f2bdca77eafa parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/2f4a6e06-f0af-4710-bd0f-554c7df5d493_isrForCrosstalkSources_855407_46.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/2f4a6e06-f0af-4710-bd0f-554c7df5d493_isrForCrosstalkSources_855407_46.stderr parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 7f4e101c604a1b70e98041bf02f272b3 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/16f630aa-fbba-488d-acab-47a6bcc9c8b4_isrForCrosstalkSources_855400_43.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/16f630aa-fbba-488d-acab-47a6bcc9c8b4_isrForCrosstalkSources_855400_43.stderr parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 066bc57b7ce8914ad76a9ac9f1f9dbd5 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7ec8fc48-cfb5-45a0-9d66-9a78777d6004_isrForCrosstalkSources_855382_37.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7ec8fc48-cfb5-45a0-9d66-9a78777d6004_isrForCrosstalkSources_855382_37.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 275f0f019c9793b4bcbc51256b505112 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7913d145-472b-440f-ad83-3b92732aec8f_isrForCrosstalkSources_855431_42.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7913d145-472b-440f-ad83-3b92732aec8f_isrForCrosstalkSources_855431_42.stderr parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash d07eaa5a954d441748ccf654c34032d2 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/20722a66-6e8e-4e62-8561-a5d65c051a10_isrForCrosstalkSources_855442_15.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/20722a66-6e8e-4e62-8561-a5d65c051a10_isrForCrosstalkSources_855442_15.stderr parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 940fd429d83d5853bdcb530194ad0fdd parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9682fd69-eded-4c2e-b3ac-aa5fcaf4ab95_isrForCrosstalkSources_855411_32.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9682fd69-eded-4c2e-b3ac-aa5fcaf4ab95_isrForCrosstalkSources_855411_32.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash a9824c38738d9408971aead72a7725d4 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a27e47ff-a4df-4cb6-a308-c9e9e623a754_isrForCrosstalkSources_855401_59.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a27e47ff-a4df-4cb6-a308-c9e9e623a754_isrForCrosstalkSources_855401_59.stderr parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 9a1692c3ea7c4ec28989298663784cd3 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/5181e166-7186-42d4-b12e-3c168a7ff679_isrForCrosstalkSources_855387_11.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/5181e166-7186-42d4-b12e-3c168a7ff679_isrForCrosstalkSources_855387_11.stderr parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 7a423d8489ee2aa575e9f2fae8cfdc51 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/43205f56-7575-4cae-9b85-9d37074aa7b5_isrForCrosstalkSources_855404_21.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/43205f56-7575-4cae-9b85-9d37074aa7b5_isrForCrosstalkSources_855404_21.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 574da1913abecd7d723c6ee757cb4644 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a62a820a-518b-462e-ae64-afaf59ffdba6_isrForCrosstalkSources_855383_13.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a62a820a-518b-462e-ae64-afaf59ffdba6_isrForCrosstalkSources_855383_13.stderr parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 135c26aa8a13a1a2ca25b36c054e4150 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/59733803-5aa2-4ea0-a20e-bc541644982b_isrForCrosstalkSources_855397_42.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/59733803-5aa2-4ea0-a20e-bc541644982b_isrForCrosstalkSources_855397_42.stderr parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 86bde349b5d68d600bde2e391301d1f2 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3d6a3b07-1864-4af8-b063-a011b0e2b577_isrForCrosstalkSources_855447_2.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3d6a3b07-1864-4af8-b063-a011b0e2b577_isrForCrosstalkSources_855447_2.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash eb5098a900fc8484422da894df0efe53 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/39e0b0ae-3d42-4b73-9189-635ceebe31aa_isrForCrosstalkSources_855383_45.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/39e0b0ae-3d42-4b73-9189-635ceebe31aa_isrForCrosstalkSources_855383_45.stderr parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 5458ad511114e483d84cddcc89825211 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ea112bf-ddb9-4618-a40e-7774b17adc6e_isrForCrosstalkSources_855396_37.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ea112bf-ddb9-4618-a40e-7774b17adc6e_isrForCrosstalkSources_855396_37.stderr parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 5f0573f225e9e914e1f193ca5a075b1d parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9bfec801-c10c-4222-b9b4-1722d74bfc3d_isrForCrosstalkSources_855432_16.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9bfec801-c10c-4222-b9b4-1722d74bfc3d_isrForCrosstalkSources_855432_16.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash bb1694785b3f841c083812e8ed4956b4 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c611efd7-9e00-4dcc-8f30-d53913701ee6_isrForCrosstalkSources_855390_9.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c611efd7-9e00-4dcc-8f30-d53913701ee6_isrForCrosstalkSources_855390_9.stderr parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash d00b3c41045a6c56adbe95cbf3c5a339 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f88dbbe9-66ea-4f8d-96d9-6d54b12290d3_isrForCrosstalkSources_855437_20.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f88dbbe9-66ea-4f8d-96d9-6d54b12290d3_isrForCrosstalkSources_855437_20.stderr parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 79de9835c19a6a947dfd159ae496845a parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5534c7df-9ceb-4a4d-98a1-a79a7bb2bb03_isrForCrosstalkSources_855451_36.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5534c7df-9ceb-4a4d-98a1-a79a7bb2bb03_isrForCrosstalkSources_855451_36.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash fa4c338e4fcaac6af1a2b2f948d75838 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/70f13dba-0553-4bbf-b2cd-9cbefbc02f8e_isrForCrosstalkSources_855418_6.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/70f13dba-0553-4bbf-b2cd-9cbefbc02f8e_isrForCrosstalkSources_855418_6.stderr parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash c771fcd147bb4e242a5e315d8d4c179b parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/b13cf44c-6ab3-4e17-b0f5-8457bd371cd0_isrForCrosstalkSources_855412_1.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/b13cf44c-6ab3-4e17-b0f5-8457bd371cd0_isrForCrosstalkSources_855412_1.stderr parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 78eae4d5bbd5694768333dbc9d5cdf94 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:49:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964037 proc_lsst.multi INFO: [multi] job_id 16964037 2024-03-09 14:49:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 14:49:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964037 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964037 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16964037 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-09 14:49:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-09 14:49:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/20a96fcb-c53a-4120-bd4f-e337da5304ae_isrForCrosstalkSources_855398_46.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/20a96fcb-c53a-4120-bd4f-e337da5304ae_isrForCrosstalkSources_855398_46.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 5e5b0985c0febf02315d2bc3d083e214 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9106e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:49:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 14:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:49:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8347059c-a5da-4897-8829-31a5006c48e8_isrForCrosstalkSources_855406_24.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8347059c-a5da-4897-8829-31a5006c48e8_isrForCrosstalkSources_855406_24.stderr parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 05b6259df19911021b351ea45fd412e9 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/463f321c-ce7a-4432-9c50-589bd9e9ec21_isrForCrosstalkSources_855444_59.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/463f321c-ce7a-4432-9c50-589bd9e9ec21_isrForCrosstalkSources_855444_59.stderr parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash a02d492fedeef5f0fc915a27e8bcc72e parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1bf0182f-cb35-4b86-82ff-2d67aa0ab18e_isrForCrosstalkSources_855381_18.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1bf0182f-cb35-4b86-82ff-2d67aa0ab18e_isrForCrosstalkSources_855381_18.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 2fe7fdb0d7be2c5246c5e3b7c1886287 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a5bbd32-66f0-411b-a141-a947a63d04dc_isrForCrosstalkSources_855449_19.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a5bbd32-66f0-411b-a141-a947a63d04dc_isrForCrosstalkSources_855449_19.stderr parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 202149f4a333021b68c2161820a2c965 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e61d012f-918a-49fb-8636-308eaf0c3dc1_isrForCrosstalkSources_855451_60.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e61d012f-918a-49fb-8636-308eaf0c3dc1_isrForCrosstalkSources_855451_60.stderr parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 0f1245875b14f94908849b5f96ecbaa1 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e41b34b0-7ebf-48c8-903b-a0ada95ac75c_isrForCrosstalkSources_855431_31.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e41b34b0-7ebf-48c8-903b-a0ada95ac75c_isrForCrosstalkSources_855431_31.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 6b2bf4ea75c82581374b2052995d04f2 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/76636563-678b-4e7b-88c0-b6f7dc6b99ee_isrForCrosstalkSources_855459_21.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/76636563-678b-4e7b-88c0-b6f7dc6b99ee_isrForCrosstalkSources_855459_21.stderr parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 7403bf44577e77f3e7826a41245c2112 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/08252cbb-7f44-4b2f-9cdf-6ae6f9bdb469_isrForCrosstalkSources_855420_60.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/08252cbb-7f44-4b2f-9cdf-6ae6f9bdb469_isrForCrosstalkSources_855420_60.stderr parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 86cd24b455832b115aa1a24028dc2209 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/df07a6b6-bca1-48d1-86b2-1e552173c72a_isrForCrosstalkSources_855434_28.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/df07a6b6-bca1-48d1-86b2-1e552173c72a_isrForCrosstalkSources_855434_28.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 2cc99ceefb9d3b715ef7ca005dc57788 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e4236383-5bf3-47fd-865d-f4937f354dcd_isrForCrosstalkSources_855451_51.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e4236383-5bf3-47fd-865d-f4937f354dcd_isrForCrosstalkSources_855451_51.stderr parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 50b0a263603aadc357a5673c13ab4afa parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea343de0-be58-42ab-8e0a-61bd9c56785c_isrForCrosstalkSources_855382_61.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea343de0-be58-42ab-8e0a-61bd9c56785c_isrForCrosstalkSources_855382_61.stderr parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 1173b033d93f412a514333a0353701a3 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/77f7a647-198d-47b1-bc77-27f4feab504c_isrForCrosstalkSources_855414_28.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/77f7a647-198d-47b1-bc77-27f4feab504c_isrForCrosstalkSources_855414_28.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash f3d4dbf0d49c58287302d2326bf580b2 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/df0125b3-f877-4bfa-8df5-e62355942dfe_isrForCrosstalkSources_855386_49.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/df0125b3-f877-4bfa-8df5-e62355942dfe_isrForCrosstalkSources_855386_49.stderr parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 2da912c5fdc256b7499ec21ae553974f parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/078c7896-e2c2-4479-8231-ebd16d6c6a40_isrForCrosstalkSources_855406_34.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/078c7896-e2c2-4479-8231-ebd16d6c6a40_isrForCrosstalkSources_855406_34.stderr parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 8eea19bbe3ace23c38560872a3ae1181 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c4c1a9e-ad38-4339-bce7-9a9c9bff7437_isrForCrosstalkSources_855400_56.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c4c1a9e-ad38-4339-bce7-9a9c9bff7437_isrForCrosstalkSources_855400_56.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 12bc0ac89b16e8e17ef40e9e0affe416 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9305cdcb-c43f-45c2-b23d-1d995b015fdb_isrForCrosstalkSources_855434_57.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9305cdcb-c43f-45c2-b23d-1d995b015fdb_isrForCrosstalkSources_855434_57.stderr parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 2b8b77faa88a4f73f87c0afe0623a0d5 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ed837a86-77e7-459a-b2f3-f16e378fe107_isrForCrosstalkSources_855445_32.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ed837a86-77e7-459a-b2f3-f16e378fe107_isrForCrosstalkSources_855445_32.stderr parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 07b2b92b0bca94ae678ce92b4d041cdc parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e9b49380-f465-49ef-82c0-0f2bb8edbe62_isrForCrosstalkSources_855391_62.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e9b49380-f465-49ef-82c0-0f2bb8edbe62_isrForCrosstalkSources_855391_62.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 8b03ed853a4752c4320d1139d8a2255d parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c12aef35-a75a-48b8-bd6a-a4f772b5483b_isrForCrosstalkSources_855409_38.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c12aef35-a75a-48b8-bd6a-a4f772b5483b_isrForCrosstalkSources_855409_38.stderr parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 84c92ce2103ec37d058f7dfcdc31e926 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/41037bec-b935-44f4-8bbb-40325e6a668a_isrForCrosstalkSources_855430_34.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/41037bec-b935-44f4-8bbb-40325e6a668a_isrForCrosstalkSources_855430_34.stderr parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 9b512f4152e943ab82d7549267066669 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6ecc9803-8814-48c3-a5be-e142edfb4809_isrForCrosstalkSources_855440_60.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6ecc9803-8814-48c3-a5be-e142edfb4809_isrForCrosstalkSources_855440_60.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash d6851995ac95ad0e0799aecb73090b85 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/db983aeb-829a-46a8-a45f-8fc2ac0458c7_isrForCrosstalkSources_855435_52.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/db983aeb-829a-46a8-a45f-8fc2ac0458c7_isrForCrosstalkSources_855435_52.stderr parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash d3c23f4cb4a96bc45bb8dcf6d2e19bb7 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2758b3f1-fbf4-4e80-b1ce-e7fd30b59a37_isrForCrosstalkSources_855417_31.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2758b3f1-fbf4-4e80-b1ce-e7fd30b59a37_isrForCrosstalkSources_855417_31.stderr parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash c417feee61a8a54b46db0ca16bcb43dd parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fca1425d-1f51-4d0c-b1ac-c194939d25a2_isrForCrosstalkSources_855423_40.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fca1425d-1f51-4d0c-b1ac-c194939d25a2_isrForCrosstalkSources_855423_40.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 9e77648d7efc196c1768f2e520147e56 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b51f22e4-cb69-4e7c-bc7c-0ee032951a35_isrForCrosstalkSources_855423_9.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b51f22e4-cb69-4e7c-bc7c-0ee032951a35_isrForCrosstalkSources_855423_9.stderr parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash ec809708698468ab97fd6b4d69bc5ce5 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0ccb1095-8e15-4b3d-933a-2a799b5ab45f_isrForCrosstalkSources_855406_42.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0ccb1095-8e15-4b3d-933a-2a799b5ab45f_isrForCrosstalkSources_855406_42.stderr parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 8fdae345ca26e6568e2e97bc8bd7268d parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f899d0a0-70ae-4684-9b54-c164a323d9ed_isrForCrosstalkSources_855436_35.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f899d0a0-70ae-4684-9b54-c164a323d9ed_isrForCrosstalkSources_855436_35.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 3ab05c42fb1dde829c2d0114c53f64b9 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/61268c9e-8632-4773-8b0c-92ed263e52a5_isrForCrosstalkSources_855417_25.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/61268c9e-8632-4773-8b0c-92ed263e52a5_isrForCrosstalkSources_855417_25.stderr parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 5ec589415e04ce54fb23e4f9f71d75c7 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f6f0be86-f5a7-4599-a682-a67e2d566152_isrForCrosstalkSources_855443_34.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f6f0be86-f5a7-4599-a682-a67e2d566152_isrForCrosstalkSources_855443_34.stderr parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash c8e4a0c2981ef175a359766114ad4260 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9805e960-a540-4d4e-8109-9e2e792823c3_isrForCrosstalkSources_855409_49.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9805e960-a540-4d4e-8109-9e2e792823c3_isrForCrosstalkSources_855409_49.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 52185b8c7274102377511472be905c97 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/69dcc0f3-abec-4812-a1f0-f290deaa4db2_isrForCrosstalkSources_855429_46.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/69dcc0f3-abec-4812-a1f0-f290deaa4db2_isrForCrosstalkSources_855429_46.stderr parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 241dc42a47c944c8a8f14580040fbc1b parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8212d46e-f162-473e-a728-11a31b6e545c_isrForCrosstalkSources_855447_59.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8212d46e-f162-473e-a728-11a31b6e545c_isrForCrosstalkSources_855447_59.stderr parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 6a886434f52a6e9485ca5a22275b6229 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6705a3b5-ade9-4ed4-97d6-9d4c714997b8_isrForCrosstalkSources_855429_58.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6705a3b5-ade9-4ed4-97d6-9d4c714997b8_isrForCrosstalkSources_855429_58.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash e721485c1bb9d42e2ea10a7b54190eeb parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3bb88089-4b46-438a-8ddf-9c2d64207218_isrForCrosstalkSources_855420_44.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3bb88089-4b46-438a-8ddf-9c2d64207218_isrForCrosstalkSources_855420_44.stderr parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 39670e679680b40ac79b94ba4be36881 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8b006288-6030-441f-93dc-b930cbb9ed62_isrForCrosstalkSources_855407_57.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8b006288-6030-441f-93dc-b930cbb9ed62_isrForCrosstalkSources_855407_57.stderr parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 0351069189a465960005e36b55c62198 parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/491d3030-b499-4f42-a2ad-42a89640945b_isrForCrosstalkSources_855383_56.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/491d3030-b499-4f42-a2ad-42a89640945b_isrForCrosstalkSources_855383_56.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 3f5e4d865855573affd860bb6d608771 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f6f15a12-7311-49b1-90f5-151a699a40c3_isrForCrosstalkSources_855428_8.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f6f15a12-7311-49b1-90f5-151a699a40c3_isrForCrosstalkSources_855428_8.stderr parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash dc6c7d57b465ce2616c21958bad2e4ef parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4f63fbc7-c2bb-4441-a592-7ba014fdec84_isrForCrosstalkSources_855391_57.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4f63fbc7-c2bb-4441-a592-7ba014fdec84_isrForCrosstalkSources_855391_57.stderr parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 67ab6adfa11fce616cefeb0cce9a90b8 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/29ed965a-81b5-485b-a601-9d91f80bcbc1_isrForCrosstalkSources_855441_3.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/29ed965a-81b5-485b-a601-9d91f80bcbc1_isrForCrosstalkSources_855441_3.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 121494858abb101b6a76118873b87b13 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964038 proc_lsst.multi INFO: [multi] job_id 16964038 2024-03-09 14:50:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 14:50:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964038 parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f7bb8cfb-d0ac-4445-82f3-c033e006ec43_isrForCrosstalkSources_855388_54.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f7bb8cfb-d0ac-4445-82f3-c033e006ec43_isrForCrosstalkSources_855388_54.stderr parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 498ae510255bcca785a9e75b96ea4027 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a8c48ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964038 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16964038 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-09 14:50:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-09 14:50:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 14:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/40254421-1fda-4b34-bc87-59b6fe7634fc_isrForCrosstalkSources_855449_20.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/40254421-1fda-4b34-bc87-59b6fe7634fc_isrForCrosstalkSources_855449_20.stderr parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 3d58749269eb9dd07e5e3600842f2960 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4e59b6c7-6bb9-4dba-8711-df4f736e20d6_isrForCrosstalkSources_855426_18.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4e59b6c7-6bb9-4dba-8711-df4f736e20d6_isrForCrosstalkSources_855426_18.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 8f6c17d05d3da1742b7f9f78cce69a73 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a239cd6f-148b-4436-bba7-afaaa3b4ece8_isrForCrosstalkSources_855398_14.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a239cd6f-148b-4436-bba7-afaaa3b4ece8_isrForCrosstalkSources_855398_14.stderr parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 26d12f51622ed42e9f220f52aacb7c21 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81ed0258-4bbb-4d29-8058-9f1bff78a566_isrForCrosstalkSources_855398_59.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81ed0258-4bbb-4d29-8058-9f1bff78a566_isrForCrosstalkSources_855398_59.stderr parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 5540f1cb644e836dff8b4174892794da parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d24acbe8-b57f-4d5a-9710-99fdbf93a3ca_isrForCrosstalkSources_855409_3.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d24acbe8-b57f-4d5a-9710-99fdbf93a3ca_isrForCrosstalkSources_855409_3.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash a6f932ee08b87c8f33f1160e8ec1562b parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8a711335-405e-46b4-8bcd-c1e36dd1d919_isrForCrosstalkSources_855381_6.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8a711335-405e-46b4-8bcd-c1e36dd1d919_isrForCrosstalkSources_855381_6.stderr parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 06c02659f50e397ddcb71a8b07e6e429 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0beb003e-9437-45ee-885f-9856f621df6a_isrForCrosstalkSources_855420_24.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0beb003e-9437-45ee-885f-9856f621df6a_isrForCrosstalkSources_855420_24.stderr parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash aad5ef06313d297f04545d553ae5c882 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/caad7119-da14-43df-8780-4636e0a3efb6_isrForCrosstalkSources_855422_3.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/caad7119-da14-43df-8780-4636e0a3efb6_isrForCrosstalkSources_855422_3.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 5beefa09b6ee3290e8c049492ad0a77d parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7299a1e4-89f6-4b87-8f94-ce2bff6bb28e_isrForCrosstalkSources_855450_33.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7299a1e4-89f6-4b87-8f94-ce2bff6bb28e_isrForCrosstalkSources_855450_33.stderr parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash d2d7fe6b9c2cdb2604edb12043629748 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c3ff0a66-bb65-4e23-bc11-7b9b09829d9b_isrForCrosstalkSources_855446_59.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c3ff0a66-bb65-4e23-bc11-7b9b09829d9b_isrForCrosstalkSources_855446_59.stderr parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 05abb6838ccf7dbcca886768f245ff1e parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/74c9a681-9062-42bf-8ef4-6491132ef8dc_isrForCrosstalkSources_855416_37.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/74c9a681-9062-42bf-8ef4-6491132ef8dc_isrForCrosstalkSources_855416_37.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash d62555587939b3e68cd2e04c085997ce parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bf4bae2a-45c1-4f36-8648-a21878221511_isrForCrosstalkSources_855427_20.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bf4bae2a-45c1-4f36-8648-a21878221511_isrForCrosstalkSources_855427_20.stderr parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 8ececd3121fd2df7896fc01240131761 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/88af174a-8897-4161-bc16-b54dcf8b03c5_isrForCrosstalkSources_855410_52.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/88af174a-8897-4161-bc16-b54dcf8b03c5_isrForCrosstalkSources_855410_52.stderr parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash cc898725e5c22b648094af1c0a707acf parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e6a290ef-cc63-4140-aa4d-354ee73204ba_isrForCrosstalkSources_855398_6.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e6a290ef-cc63-4140-aa4d-354ee73204ba_isrForCrosstalkSources_855398_6.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 2147803bec99e5dfa2f8cb181c416e7f parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9fab4831-8572-4578-8930-739e9f6f630d_isrForCrosstalkSources_855450_42.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9fab4831-8572-4578-8930-739e9f6f630d_isrForCrosstalkSources_855450_42.stderr parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash cb8cd89ca35008a5cdac71a53a8d1fb8 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/dd58d932-fee4-42ed-b185-6d5ca10b775f_isrForCrosstalkSources_855416_6.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/dd58d932-fee4-42ed-b185-6d5ca10b775f_isrForCrosstalkSources_855416_6.stderr parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 2fa8041d001863ec21d30cc6832fa082 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/06dd2db3-a8d6-4d93-8341-cc9175bca26b_isrForCrosstalkSources_855436_17.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/06dd2db3-a8d6-4d93-8341-cc9175bca26b_isrForCrosstalkSources_855436_17.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash d72b19664876c098ac6844ecb01cc3ef parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/10ec6cfb-84b6-41c1-a187-1cbda34080f8_isrForCrosstalkSources_855381_22.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/10ec6cfb-84b6-41c1-a187-1cbda34080f8_isrForCrosstalkSources_855381_22.stderr parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 9795d11263019faed06103ebbdabeb92 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c8c0d428-6360-485e-95b9-6e1037704bef_isrForCrosstalkSources_855415_35.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c8c0d428-6360-485e-95b9-6e1037704bef_isrForCrosstalkSources_855415_35.stderr parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 345ac789903185b79d5895eb3ea837d2 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/00935590-d862-4d3a-a8ec-302c01f4a54a_isrForCrosstalkSources_855424_14.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/00935590-d862-4d3a-a8ec-302c01f4a54a_isrForCrosstalkSources_855424_14.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash eba05976ed6f56a9b8bb506a7ca2eec9 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3a596a89-33af-4acf-827c-1a96ccc938a5_isrForCrosstalkSources_855416_35.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3a596a89-33af-4acf-827c-1a96ccc938a5_isrForCrosstalkSources_855416_35.stderr parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 77d618a2ec5033bffbcf9cb164b43c73 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6800ab46-434d-4992-81eb-6ffa7717c284_isrForCrosstalkSources_855453_59.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6800ab46-434d-4992-81eb-6ffa7717c284_isrForCrosstalkSources_855453_59.stderr parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 4ae4a350728ea8e90e77b31f1344ff05 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2f48c4c9-b9c5-4de8-9434-abdae44a1ff4_isrForCrosstalkSources_855436_4.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2f48c4c9-b9c5-4de8-9434-abdae44a1ff4_isrForCrosstalkSources_855436_4.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 31aaf151fc1c59c857b1cec0929427e7 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6af8d7d8-1237-45fc-82a6-df104da8df4a_isrForCrosstalkSources_855456_26.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6af8d7d8-1237-45fc-82a6-df104da8df4a_isrForCrosstalkSources_855456_26.stderr parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 7244700e99a47d6e5113c39ac7715f58 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0aa14a4b-29d5-4284-a29f-04e360207b23_isrForCrosstalkSources_855405_50.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0aa14a4b-29d5-4284-a29f-04e360207b23_isrForCrosstalkSources_855405_50.stderr parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 83df9b98a7037374c543592181532f48 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/96af1b50-7526-4a92-a5d4-23759e6d3aa4_isrForCrosstalkSources_855384_50.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/96af1b50-7526-4a92-a5d4-23759e6d3aa4_isrForCrosstalkSources_855384_50.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 9385a862575aeb324087ae5e07da63c6 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964039 proc_lsst.multi INFO: [multi] job_id 16964039 2024-03-09 14:50:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 14:50:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964039 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964039 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16964039 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-09 14:50:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-09 14:50:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 14:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b9b1c96c-07dd-4ee6-8e53-1ac487b19805_isrForCrosstalkSources_855437_11.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b9b1c96c-07dd-4ee6-8e53-1ac487b19805_isrForCrosstalkSources_855437_11.stderr parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 7070dfc34003126244fd041cc216d6b6 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/8255a95d-1d6f-447b-b3a3-48925aa46b6d_isrForCrosstalkSources_855385_1.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/8255a95d-1d6f-447b-b3a3-48925aa46b6d_isrForCrosstalkSources_855385_1.stderr parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash ddd0e83134c83b9659d2f451bca44300 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/8eee20c6-cae9-418a-a59f-6df8034582f9_isrForCrosstalkSources_855433_48.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/8eee20c6-cae9-418a-a59f-6df8034582f9_isrForCrosstalkSources_855433_48.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 23e2699acef5b79768dd25115144ba99 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/81634ecb-18d0-424d-b3de-50083315fce8_isrForCrosstalkSources_855419_45.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/81634ecb-18d0-424d-b3de-50083315fce8_isrForCrosstalkSources_855419_45.stderr parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash a88798880c386f04ea4cca15edb21444 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab4760bd-8ea2-4995-a4ee-4f9d756642b4_isrForCrosstalkSources_855437_41.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab4760bd-8ea2-4995-a4ee-4f9d756642b4_isrForCrosstalkSources_855437_41.stderr parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 2eaaf5138e6f1ab6575740eabc3fcc80 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ef3d2cb6-403a-4231-9bb6-02dd14f87dc5_isrForCrosstalkSources_855399_51.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ef3d2cb6-403a-4231-9bb6-02dd14f87dc5_isrForCrosstalkSources_855399_51.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 0a7d6109904cc8a9f51e5ce2d5871ddb parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b49dbf64-cac5-4c8b-9c4a-d5eb54e62b03_isrForCrosstalkSources_855444_9.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b49dbf64-cac5-4c8b-9c4a-d5eb54e62b03_isrForCrosstalkSources_855444_9.stderr parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash f765bbabe557cf95bc36d3b23ee9e8d4 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d1bdf053-75be-44bd-a2c3-91cf57e7e971_isrForCrosstalkSources_855392_40.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d1bdf053-75be-44bd-a2c3-91cf57e7e971_isrForCrosstalkSources_855392_40.stderr parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 27a0e33ef1d0d1239335be944fa7c67c parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18c50c4a-f444-498a-970d-dae13cd0c338_isrForCrosstalkSources_855423_41.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18c50c4a-f444-498a-970d-dae13cd0c338_isrForCrosstalkSources_855423_41.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 33f96149595490533ee42e6ed8f02fad parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cb57b7cc-1a1f-497a-bfd3-6e9b5722c39c_isrForCrosstalkSources_855454_49.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cb57b7cc-1a1f-497a-bfd3-6e9b5722c39c_isrForCrosstalkSources_855454_49.stderr parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash b8b7141479f9ee4fe9ba6a74e01029d3 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2ec74cda-6aff-4b94-9748-d6989d45b2fd_isrForCrosstalkSources_855414_51.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2ec74cda-6aff-4b94-9748-d6989d45b2fd_isrForCrosstalkSources_855414_51.stderr parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash cb34ae0f099a0286bf96c4e116099731 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6630d9fa-12d6-459b-8ca3-e498150fe2c1_isrForCrosstalkSources_855407_32.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6630d9fa-12d6-459b-8ca3-e498150fe2c1_isrForCrosstalkSources_855407_32.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash d432874bfe45cc1d7e04850b74439aed parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/33f53db0-3efb-4435-b5e5-48d921616e02_isrForCrosstalkSources_855398_19.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/33f53db0-3efb-4435-b5e5-48d921616e02_isrForCrosstalkSources_855398_19.stderr parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 97fa6d6547a67f1a95cdd9b5d055d283 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e0d8b53-337e-4cea-84f6-df967b63a85e_isrForCrosstalkSources_855451_9.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e0d8b53-337e-4cea-84f6-df967b63a85e_isrForCrosstalkSources_855451_9.stderr parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 06f2928e7811c72a873767147f210d31 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6332d152-7fd9-4f01-8004-7ab6ac8a17b1_isrForCrosstalkSources_855407_52.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6332d152-7fd9-4f01-8004-7ab6ac8a17b1_isrForCrosstalkSources_855407_52.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash d56869b1dee5a8c6352a41f3fb0d3823 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5f94427a-6a45-40b4-b695-af6b29c88a22_isrForCrosstalkSources_855403_61.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5f94427a-6a45-40b4-b695-af6b29c88a22_isrForCrosstalkSources_855403_61.stderr parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash d0e5d9e6e4030857c45b1ef27eb30ed3 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/88d189fe-4fef-4cae-b66b-f96fcd232a4e_isrForCrosstalkSources_855427_12.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/88d189fe-4fef-4cae-b66b-f96fcd232a4e_isrForCrosstalkSources_855427_12.stderr parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 181d984f847a5c233ebb5a1e499e49d0 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/2158ddf2-db5c-4a22-a911-6e4c1049bec9_isrForCrosstalkSources_855386_40.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/2158ddf2-db5c-4a22-a911-6e4c1049bec9_isrForCrosstalkSources_855386_40.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash ec5b179de778abebcf9a93b2e8837cab parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964040 proc_lsst.multi INFO: [multi] job_id 16964040 2024-03-09 14:50:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 14:50:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964040 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964040 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16964040 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-09 14:50:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-09 14:50:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 14:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/252d58f5-678c-4107-8703-9d8c0ea407a2_isrForCrosstalkSources_855393_51.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/252d58f5-678c-4107-8703-9d8c0ea407a2_isrForCrosstalkSources_855393_51.stderr parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 66828c2d10a86b576ec58def41ce547a parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/ee9f8a90-d094-408b-9774-02fb1c558b4f_isrForCrosstalkSources_855385_3.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/ee9f8a90-d094-408b-9774-02fb1c558b4f_isrForCrosstalkSources_855385_3.stderr parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash f470ce79f51443e60cdfef64808a161a parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d101d21a-d9fd-4d80-94dc-ba181c2423b4_isrForCrosstalkSources_855401_32.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d101d21a-d9fd-4d80-94dc-ba181c2423b4_isrForCrosstalkSources_855401_32.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 4facd724ae3e25302622b7bca6643e08 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3a6b6cba-6d45-4dca-837b-7b76ae85a197_isrForCrosstalkSources_855426_42.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3a6b6cba-6d45-4dca-837b-7b76ae85a197_isrForCrosstalkSources_855426_42.stderr parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 24614113ae7f8bb4ea7bfa6954d96469 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8bdd2d29-d09c-46e7-95e8-c1f99d5fd8a7_isrForCrosstalkSources_855449_44.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8bdd2d29-d09c-46e7-95e8-c1f99d5fd8a7_isrForCrosstalkSources_855449_44.stderr parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 38ac94c757dced93c56a7316c64a6194 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ef7c6578-1a31-409a-b423-45c648c68aff_isrForCrosstalkSources_855388_58.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ef7c6578-1a31-409a-b423-45c648c68aff_isrForCrosstalkSources_855388_58.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash e1b67996eb6c9a0ad4bd4b0878f1df45 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/baae7d1a-7f29-43ba-97bc-c176accd720e_isrForCrosstalkSources_855406_39.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/baae7d1a-7f29-43ba-97bc-c176accd720e_isrForCrosstalkSources_855406_39.stderr parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 676d98a8bc200276033abad9ea7e916c parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/fd4df691-9511-4504-97be-73781a5b67b2_isrForCrosstalkSources_855383_26.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/fd4df691-9511-4504-97be-73781a5b67b2_isrForCrosstalkSources_855383_26.stderr parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 717fd83b7c08da4c4768e9fd3085baa4 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f71e72ab-5d91-4689-be58-d5f1e5ef5b8e_isrForCrosstalkSources_855446_33.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f71e72ab-5d91-4689-be58-d5f1e5ef5b8e_isrForCrosstalkSources_855446_33.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 0f8b1b23ee29a9d9017cfbad82ac95a3 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d45f891f-5ba8-435e-a0ec-aba327aa74fe_isrForCrosstalkSources_855388_60.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d45f891f-5ba8-435e-a0ec-aba327aa74fe_isrForCrosstalkSources_855388_60.stderr parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash aa6621da994a4b9ca02f66bc8922c3e7 parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad78fedc-a8a4-47fe-aee4-c845e3c0d00c_isrForCrosstalkSources_855429_10.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad78fedc-a8a4-47fe-aee4-c845e3c0d00c_isrForCrosstalkSources_855429_10.stderr parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 138bd4a4f6740c67667bf6c8d66d03fa parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29a06e5d-9c95-4643-be3e-fde3ed4d5e0c_isrForCrosstalkSources_855447_50.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29a06e5d-9c95-4643-be3e-fde3ed4d5e0c_isrForCrosstalkSources_855447_50.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 025f4af5bd70ad9f69d2a7b49be22c07 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0a93ab36-d53c-44b2-9c74-28e7b0586aa4_isrForCrosstalkSources_855380_43.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0a93ab36-d53c-44b2-9c74-28e7b0586aa4_isrForCrosstalkSources_855380_43.stderr parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 0450b7c9b23c221f07c7764e95f7a530 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0b8e6bfc-8acd-4052-bc17-920a7c9d8da6_isrForCrosstalkSources_855438_47.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0b8e6bfc-8acd-4052-bc17-920a7c9d8da6_isrForCrosstalkSources_855438_47.stderr parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash ce34db73e4504b72d0b90c5bb4af85f1 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ba43a520-6ccb-4fc8-9294-95a4533011d2_isrForCrosstalkSources_855384_19.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ba43a520-6ccb-4fc8-9294-95a4533011d2_isrForCrosstalkSources_855384_19.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 96ca4696f0faeff499fb777c81c4df23 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a46259a7-6acb-497c-882b-62605aa1c439_isrForCrosstalkSources_855395_51.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a46259a7-6acb-497c-882b-62605aa1c439_isrForCrosstalkSources_855395_51.stderr parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 956d1fd16751c535c5af9c0cdfa4579c parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cb891bdb-2b67-4ac6-aa20-a80d392ab9e4_isrForCrosstalkSources_855418_34.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cb891bdb-2b67-4ac6-aa20-a80d392ab9e4_isrForCrosstalkSources_855418_34.stderr parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 00e85844c5cf35a547868294f08d7d8d parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/b54448d4-a009-4a1d-ab67-45ca4a123f42_isrForCrosstalkSources_855443_26.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/b54448d4-a009-4a1d-ab67-45ca4a123f42_isrForCrosstalkSources_855443_26.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 01459185a63e43c48e465199265ee2af parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/67fc0035-199c-4a6a-bb60-f3099491bcf0_isrForCrosstalkSources_855411_31.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/67fc0035-199c-4a6a-bb60-f3099491bcf0_isrForCrosstalkSources_855411_31.stderr parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 51af4695def8dec9b87ba12c6c90425f parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a7ec152e-1f46-491c-acf0-d272fb5bc754_isrForCrosstalkSources_855420_33.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a7ec152e-1f46-491c-acf0-d272fb5bc754_isrForCrosstalkSources_855420_33.stderr parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 2a5581a06ad8859fe673984f582ad429 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4315aed2-7a46-480e-b6a0-9e888ee04fe0_isrForCrosstalkSources_855405_3.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4315aed2-7a46-480e-b6a0-9e888ee04fe0_isrForCrosstalkSources_855405_3.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 6ee232055de828da05ddbe0bfd39381e parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0b041358-705a-4d85-bcac-eb3100c4246a_isrForCrosstalkSources_855390_62.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0b041358-705a-4d85-bcac-eb3100c4246a_isrForCrosstalkSources_855390_62.stderr parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 4bab5be049f7f2a8cb4a60caf48c9cb8 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/735bea73-d488-498e-be2a-ef0a9a54c40b_isrForCrosstalkSources_855440_53.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/735bea73-d488-498e-be2a-ef0a9a54c40b_isrForCrosstalkSources_855440_53.stderr parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash d44f1ed7fad549dcf976185969332c33 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fce2a697-3e3a-4af3-86d1-eeb84338b8c8_isrForCrosstalkSources_855425_44.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fce2a697-3e3a-4af3-86d1-eeb84338b8c8_isrForCrosstalkSources_855425_44.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash cd7c3790694aa0cd1ae25d125013b50a parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24cd8d8f-cb82-4c5a-899e-99848b6fadc9_isrForCrosstalkSources_855390_17.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24cd8d8f-cb82-4c5a-899e-99848b6fadc9_isrForCrosstalkSources_855390_17.stderr parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 7058f8837898ffe049edaed01263ee51 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7d873681-0e44-47ce-ab17-726e00ecd218_isrForCrosstalkSources_855396_44.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7d873681-0e44-47ce-ab17-726e00ecd218_isrForCrosstalkSources_855396_44.stderr parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 6ebfa022cd761c59d224e321ed2b25d7 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/80d26bfc-91a6-4052-9c8e-a23e2d130dfb_isrForCrosstalkSources_855383_2.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/80d26bfc-91a6-4052-9c8e-a23e2d130dfb_isrForCrosstalkSources_855383_2.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 521cd15a1b2a0f111536ccd2dcd4b1ce parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/846826b1-44cd-4eba-b61c-3575ba511ac7_isrForCrosstalkSources_855405_40.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/846826b1-44cd-4eba-b61c-3575ba511ac7_isrForCrosstalkSources_855405_40.stderr parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash e6c83571304134f663f458b8cbfcb27d parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/31ae4494-99d0-48b4-bed8-6c0772dd4b0b_isrForCrosstalkSources_855402_60.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/31ae4494-99d0-48b4-bed8-6c0772dd4b0b_isrForCrosstalkSources_855402_60.stderr parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash fd2a155d7cdd69b8cae6894a0eb92b0d parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/d10c3232-d338-4559-9666-5ba60bbf65c0_isrForCrosstalkSources_855440_35.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/d10c3232-d338-4559-9666-5ba60bbf65c0_isrForCrosstalkSources_855440_35.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 57242c2021bb4b07abfe9cd516377f4f parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/223baf76-bc6d-4699-a591-6d39fc225bee_isrForCrosstalkSources_855403_19.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/223baf76-bc6d-4699-a591-6d39fc225bee_isrForCrosstalkSources_855403_19.stderr parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash d5046bd768f4c85d1dd79db60f403ce4 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e9da9976-6214-4a2f-9c65-2b83e0dc3f1d_isrForCrosstalkSources_855441_29.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e9da9976-6214-4a2f-9c65-2b83e0dc3f1d_isrForCrosstalkSources_855441_29.stderr parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash ff4bbc7c2fe1fda1132699ff62160402 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/6dc97f8f-c8c8-43a0-8782-826d1d1373b6_isrForCrosstalkSources_855409_39.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/6dc97f8f-c8c8-43a0-8782-826d1d1373b6_isrForCrosstalkSources_855409_39.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash ea5ae940467bcd9d632518d96098ae54 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9cd9c5cd-776b-40c1-a120-1c8ebf960aaf_isrForCrosstalkSources_855432_8.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9cd9c5cd-776b-40c1-a120-1c8ebf960aaf_isrForCrosstalkSources_855432_8.stderr parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash a8984fa2be9ed633eca4f216fa60de0d parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/71ced3a8-ca9d-4061-827f-787840fd0dbf_isrForCrosstalkSources_855412_31.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/71ced3a8-ca9d-4061-827f-787840fd0dbf_isrForCrosstalkSources_855412_31.stderr parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash c4041f97690b44dfe9e7bedfd0a3c4e6 parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964041 proc_lsst.multi INFO: [multi] job_id 16964041 2024-03-09 14:50:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 14:50:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964041 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964041 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16964041 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-09 14:50:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-09 14:50:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 14:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3b1c8ae-b16a-48cb-a21c-885e6dd94a5d_isrForCrosstalkSources_855395_33.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3b1c8ae-b16a-48cb-a21c-885e6dd94a5d_isrForCrosstalkSources_855395_33.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 8571a2b0db74bc3d3ab0dba2856a5290 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8e49db95-240a-4685-a940-6f20eeb3dc9b_isrForCrosstalkSources_855384_28.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8e49db95-240a-4685-a940-6f20eeb3dc9b_isrForCrosstalkSources_855384_28.stderr parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 17e2b858bcb91dd6faa0981067c46595 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f2c43b46-8d8b-4868-b57b-a1ec29164d77_isrForCrosstalkSources_855417_5.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f2c43b46-8d8b-4868-b57b-a1ec29164d77_isrForCrosstalkSources_855417_5.stderr parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash a8ed5a0a196fa38c3ea24fc248a785e1 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b68b7ed-8052-4213-9169-a35f737a0055_isrForCrosstalkSources_855400_1.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b68b7ed-8052-4213-9169-a35f737a0055_isrForCrosstalkSources_855400_1.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash fd18d4d1d41cde6ae8501dd335807526 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9e2e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5478fbc9-cfa8-43e4-9007-5eb40e3271c7_isrForCrosstalkSources_855445_24.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5478fbc9-cfa8-43e4-9007-5eb40e3271c7_isrForCrosstalkSources_855445_24.stderr parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 86c9b10175f18b531a22fd3e043ccc23 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/353db4d8-4054-46e7-8426-5079152559b9_isrForCrosstalkSources_855433_7.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/353db4d8-4054-46e7-8426-5079152559b9_isrForCrosstalkSources_855433_7.stderr parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash a9c45030ad034cd77849434b6780611d parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/762e2699-d365-4709-8768-38d15842d89c_isrForCrosstalkSources_855392_23.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/762e2699-d365-4709-8768-38d15842d89c_isrForCrosstalkSources_855392_23.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash b362238dab5180d75382689b0b799821 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/235fb2d0-d60b-4986-82c8-b39f485a44cb_isrForCrosstalkSources_855447_57.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/235fb2d0-d60b-4986-82c8-b39f485a44cb_isrForCrosstalkSources_855447_57.stderr parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash f2e5aceb81806e04fde72332140e6f3d parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d12537f-fac3-4cf3-b385-56712b6a294e_isrForCrosstalkSources_855444_51.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d12537f-fac3-4cf3-b385-56712b6a294e_isrForCrosstalkSources_855444_51.stderr parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 410c84dc341deea920df501f7c2a73d9 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/7dd1aaf8-917c-45a4-99fc-7ad9a5d563d5_isrForCrosstalkSources_855427_51.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/7dd1aaf8-917c-45a4-99fc-7ad9a5d563d5_isrForCrosstalkSources_855427_51.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 613358720b0b4cc878a9075801de3d32 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b9ad56ad-b601-4580-ad48-cf5e5993e4f3_isrForCrosstalkSources_855393_18.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b9ad56ad-b601-4580-ad48-cf5e5993e4f3_isrForCrosstalkSources_855393_18.stderr parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 3b785110913c5ae3e771096ae2b69eed parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/acb75461-03b8-4ec4-9106-6c2361159adf_isrForCrosstalkSources_855437_37.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/acb75461-03b8-4ec4-9106-6c2361159adf_isrForCrosstalkSources_855437_37.stderr parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 2ff3f3a328e5717e8ac2b7059777cc84 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7b301cb5-60e9-4bd6-b7ce-f8b6e4b53265_isrForCrosstalkSources_855458_52.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7b301cb5-60e9-4bd6-b7ce-f8b6e4b53265_isrForCrosstalkSources_855458_52.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 1880ce00599873654e17d4efd67b33e1 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f2af03ba-9b3b-4b3f-a4aa-6fed0a753705_isrForCrosstalkSources_855407_53.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f2af03ba-9b3b-4b3f-a4aa-6fed0a753705_isrForCrosstalkSources_855407_53.stderr parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 991da20cff810f745703587a3489f848 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/a3333d50-3e97-4639-ba24-dfa9f628df4c_isrForCrosstalkSources_855414_14.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/a3333d50-3e97-4639-ba24-dfa9f628df4c_isrForCrosstalkSources_855414_14.stderr parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash c5d11e8f2e1625c6ae151bb28cc0f119 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eeaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/319379bd-3d64-405b-aeb5-c57f91b48e59_isrForCrosstalkSources_855403_51.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/319379bd-3d64-405b-aeb5-c57f91b48e59_isrForCrosstalkSources_855403_51.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 29327225753b152e9810ef105c0c3c5b parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0876f5d8-29e8-43c0-84c9-fcab28bbf342_isrForCrosstalkSources_855439_41.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0876f5d8-29e8-43c0-84c9-fcab28bbf342_isrForCrosstalkSources_855439_41.stderr parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 34dc8cdfaa323c93540f0e35befa7191 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cc45d36e-147a-495a-9ce4-4fb659034644_isrForCrosstalkSources_855411_15.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cc45d36e-147a-495a-9ce4-4fb659034644_isrForCrosstalkSources_855411_15.stderr parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash c7185c97adaa65a8e814ab05c04f3ce3 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b2f27ae0-928c-4387-b78b-edf249caf4e2_isrForCrosstalkSources_855451_57.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b2f27ae0-928c-4387-b78b-edf249caf4e2_isrForCrosstalkSources_855451_57.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash d8f0127a3b73351dc11a0c9f0d2c8c4a parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/75b6adbc-3de6-4682-be9a-f3058aa85d23_isrForCrosstalkSources_855448_7.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/75b6adbc-3de6-4682-be9a-f3058aa85d23_isrForCrosstalkSources_855448_7.stderr parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 8f337da864a327ee61b113636e5deacb parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/94ecfe57-bea2-445f-8a4c-290e85106e38_isrForCrosstalkSources_855399_28.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/94ecfe57-bea2-445f-8a4c-290e85106e38_isrForCrosstalkSources_855399_28.stderr parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash b70ec1270e1438b1000dc3077f421a70 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7a8187c5-a690-4a95-a267-df673fb1bc98_isrForCrosstalkSources_855421_16.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7a8187c5-a690-4a95-a267-df673fb1bc98_isrForCrosstalkSources_855421_16.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 2719f610407cf71a3cb0d12305f09d3b parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac040bcd-0d8c-48f8-9c7f-e59856acc977_isrForCrosstalkSources_855401_36.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac040bcd-0d8c-48f8-9c7f-e59856acc977_isrForCrosstalkSources_855401_36.stderr parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash d2ffc6b4b78466e1bc28eeaf66d4a871 parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e7d54203-58fb-4964-a48e-109cdc21460f_isrForCrosstalkSources_855398_5.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e7d54203-58fb-4964-a48e-109cdc21460f_isrForCrosstalkSources_855398_5.stderr parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 2507ba8f8b85bbebc1a6f08d2869237c parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/649397c9-0264-4c7d-b228-20dd8af92d74_isrForCrosstalkSources_855398_31.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/649397c9-0264-4c7d-b228-20dd8af92d74_isrForCrosstalkSources_855398_31.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 0ae39f20fa584598a74d84e6210508d5 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36b41f05-361a-4893-a221-d94e30d8cce9_isrForCrosstalkSources_855390_56.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36b41f05-361a-4893-a221-d94e30d8cce9_isrForCrosstalkSources_855390_56.stderr parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash e68a3cf6321f072643eab7c01eb0472a parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0befe143-a041-45fa-b7ad-19e0ad7fe1bc_isrForCrosstalkSources_855451_15.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0befe143-a041-45fa-b7ad-19e0ad7fe1bc_isrForCrosstalkSources_855451_15.stderr parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash ecec74d700fda83fa46159b954704a93 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf3bc02-836c-4ef4-9303-c7f6360f99d5_isrForCrosstalkSources_855388_53.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf3bc02-836c-4ef4-9303-c7f6360f99d5_isrForCrosstalkSources_855388_53.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 45db70f9f26782e708ed4c0f0d546cfc parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e226c445-9702-41d3-8d69-9e911b1982d5_isrForCrosstalkSources_855411_37.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e226c445-9702-41d3-8d69-9e911b1982d5_isrForCrosstalkSources_855411_37.stderr parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash ed692d4b597d96666ba894e8a638a475 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46698106-192f-4b40-b775-325c6c1eb79d_isrForCrosstalkSources_855448_50.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46698106-192f-4b40-b775-325c6c1eb79d_isrForCrosstalkSources_855448_50.stderr parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 9ec1901ed9f35cd6eb561a3cb8342854 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/18ae984d-cd7b-4be6-935d-7031906d327e_isrForCrosstalkSources_855383_18.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/18ae984d-cd7b-4be6-935d-7031906d327e_isrForCrosstalkSources_855383_18.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash ae8f65b5e367f68b403a2f5b537102dd parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/91897d18-ce8c-41d6-b97a-d2d706c202e5_isrForCrosstalkSources_855448_56.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/91897d18-ce8c-41d6-b97a-d2d706c202e5_isrForCrosstalkSources_855448_56.stderr parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash cab36b455e3e24095167ad61627893cb parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3e5f98d4-8b12-4531-8248-4dda92aae82c_isrForCrosstalkSources_855388_16.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3e5f98d4-8b12-4531-8248-4dda92aae82c_isrForCrosstalkSources_855388_16.stderr parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash e3438e3ae166726fa41b76f517f8119c parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b3613d6a-b21d-4f70-97bd-ec5e8a19961e_isrForCrosstalkSources_855408_46.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b3613d6a-b21d-4f70-97bd-ec5e8a19961e_isrForCrosstalkSources_855408_46.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 0871cfa635ac4ab9386e4e1c3c8f794c parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/664ac984-c781-400f-8d6a-246865ff1149_isrForCrosstalkSources_855394_33.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/664ac984-c781-400f-8d6a-246865ff1149_isrForCrosstalkSources_855394_33.stderr parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 2df8951954b410c9b5dc6b9098553bf7 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3a64e963-9a60-4a76-a180-ba28c195be1c_isrForCrosstalkSources_855402_14.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3a64e963-9a60-4a76-a180-ba28c195be1c_isrForCrosstalkSources_855402_14.stderr parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash fc2f171520afd15906eb0c55c9d214a4 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/82f53011-9ee1-404d-9da1-0beac0ec0f65_isrForCrosstalkSources_855380_41.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/82f53011-9ee1-404d-9da1-0beac0ec0f65_isrForCrosstalkSources_855380_41.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash c58631efc7cb73f85c9e436f9221172f parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b2ca3646-64f1-4925-9a8b-529de070c8a1_isrForCrosstalkSources_855422_4.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b2ca3646-64f1-4925-9a8b-529de070c8a1_isrForCrosstalkSources_855422_4.stderr parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 8cec1fc323fffdc1c296c60d8cc8919d parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f0463a44-1e7e-4711-ab5c-3eae6e4cd93a_isrForCrosstalkSources_855459_31.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f0463a44-1e7e-4711-ab5c-3eae6e4cd93a_isrForCrosstalkSources_855459_31.stderr parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 335126924f01fc8ea428812b71b6ab23 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/4ffd7dd9-61f0-4863-851f-7ec9bf5d4ed9_isrForCrosstalkSources_855432_51.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/4ffd7dd9-61f0-4863-851f-7ec9bf5d4ed9_isrForCrosstalkSources_855432_51.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 71bf1f6cecc18742a8566f49bc814c26 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1fc73d4b-e997-4cf8-8353-75275602aa4e_isrForCrosstalkSources_855388_35.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1fc73d4b-e997-4cf8-8353-75275602aa4e_isrForCrosstalkSources_855388_35.stderr parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash b73ac018643e8d75ef10b30fe9a70782 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/65cce904-c77c-4673-8810-1f9e160b5e65_isrForCrosstalkSources_855449_4.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/65cce904-c77c-4673-8810-1f9e160b5e65_isrForCrosstalkSources_855449_4.stderr parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 00f2b928fd9577cef90692f4a518818b parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6c373c01-7427-4810-a60a-d88a20b1d17e_isrForCrosstalkSources_855394_21.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6c373c01-7427-4810-a60a-d88a20b1d17e_isrForCrosstalkSources_855394_21.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 46ed477bc0a41f48b9410fc25834bc55 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/782f1aed-7754-44e3-9396-987b3e9feae9_isrForCrosstalkSources_855408_49.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/782f1aed-7754-44e3-9396-987b3e9feae9_isrForCrosstalkSources_855408_49.stderr parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 8c47e2fbfb9224b4b07e7f3ebc3c392d parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964042 proc_lsst.multi INFO: [multi] job_id 16964042 2024-03-09 14:50:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 14:50:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964042 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964042 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16964042 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-09 14:50:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-09 14:50:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 14:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/202ea002-fa94-4728-af39-d8d8ff3b58e6_isrForCrosstalkSources_855415_5.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/202ea002-fa94-4728-af39-d8d8ff3b58e6_isrForCrosstalkSources_855415_5.stderr parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 1cfff598cde030f5624265dec5b0522b parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/6ece48b3-cab5-4f6b-81f3-bf16257a96c8_isrForCrosstalkSources_855433_1.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/6ece48b3-cab5-4f6b-81f3-bf16257a96c8_isrForCrosstalkSources_855433_1.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash d7bf03b082448e3864d46e866b48b15f parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cba4755c-d239-48b3-9d23-6907616fcc74_isrForCrosstalkSources_855446_26.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cba4755c-d239-48b3-9d23-6907616fcc74_isrForCrosstalkSources_855446_26.stderr parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 9e26ea0b61729a358279c838bbdf2f9d parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9eea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/39ff4820-2423-47cd-b0aa-3307d99e53c8_isrForCrosstalkSources_855423_37.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/39ff4820-2423-47cd-b0aa-3307d99e53c8_isrForCrosstalkSources_855423_37.stderr parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 1542eec74f72813579e1023f8cc79099 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3cc3cbda-b54f-44a9-b84c-ff5146aa6cef_isrForCrosstalkSources_855383_1.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3cc3cbda-b54f-44a9-b84c-ff5146aa6cef_isrForCrosstalkSources_855383_1.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 3554b7f05ab507c0239a8e5ae2978b16 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cfcae953-b652-4954-b40c-94a737cb477b_isrForCrosstalkSources_855445_61.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cfcae953-b652-4954-b40c-94a737cb477b_isrForCrosstalkSources_855445_61.stderr parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash c6ac34237c341a6524d977fc1b3e86be parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e39d796c-d475-47b8-97e8-4ea2bbfa5459_isrForCrosstalkSources_855413_36.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e39d796c-d475-47b8-97e8-4ea2bbfa5459_isrForCrosstalkSources_855413_36.stderr parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 8d89595c5b8702bdd6da07ec78c7e3db parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a24c86d2-38bf-45dc-a73e-c3299bebe355_isrForCrosstalkSources_855394_26.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a24c86d2-38bf-45dc-a73e-c3299bebe355_isrForCrosstalkSources_855394_26.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 57490dd68fc9244186d85495ac97b2a1 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/2da167f2-8cf4-494a-81d4-2d4b1e5e480b_isrForCrosstalkSources_855394_40.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/2da167f2-8cf4-494a-81d4-2d4b1e5e480b_isrForCrosstalkSources_855394_40.stderr parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash a1d670f55b9dde4200279d8e9d2c4cbf parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/93bbe6da-3f4a-4daf-9dd2-1685a9fea4d1_isrForCrosstalkSources_855430_27.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/93bbe6da-3f4a-4daf-9dd2-1685a9fea4d1_isrForCrosstalkSources_855430_27.stderr parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 036ac81cd9dc251ab0077ced1be921ac parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7effbb11-823e-4f0d-b3f2-09fa6e7d19f4_isrForCrosstalkSources_855436_2.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7effbb11-823e-4f0d-b3f2-09fa6e7d19f4_isrForCrosstalkSources_855436_2.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 14d2771948097072edd4582fc062b337 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f13bcc2f-d973-4e56-88aa-b8fdde591f95_isrForCrosstalkSources_855426_58.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f13bcc2f-d973-4e56-88aa-b8fdde591f95_isrForCrosstalkSources_855426_58.stderr parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash f27a57280d1c9ae0733a428f8d0872c7 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/adf05dae-16a4-4258-8731-68d7dffa7f9b_isrForCrosstalkSources_855426_14.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/adf05dae-16a4-4258-8731-68d7dffa7f9b_isrForCrosstalkSources_855426_14.stderr parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 9fb72d71a51a0c1cd9b37beae8991852 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964043 proc_lsst.multi INFO: [multi] job_id 16964043 2024-03-09 14:50:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 14:50:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964043 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964043 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16964043 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-09 14:50:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-09 14:50:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 14:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccf3d496-b326-4382-8a5f-b9f1189ff645_isrForCrosstalkSources_855403_39.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccf3d496-b326-4382-8a5f-b9f1189ff645_isrForCrosstalkSources_855403_39.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 43c7cf3abcce388bb4b0a415b0dcb7b1 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cb0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e94ced9-3551-4f6e-bb6f-a0c0a8139714_isrForCrosstalkSources_855423_18.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e94ced9-3551-4f6e-bb6f-a0c0a8139714_isrForCrosstalkSources_855423_18.stderr parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash bb2e4f713bf804d7e0bcbc81a9db36d9 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b76677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/abe38278-5b9d-428a-b046-10e9ff7cfe1c_isrForCrosstalkSources_855431_13.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/abe38278-5b9d-428a-b046-10e9ff7cfe1c_isrForCrosstalkSources_855431_13.stderr parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 551d99d8a9cf806891c1a0af544322d4 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/45d63b97-0953-465e-a4aa-0e0f9b84df9f_isrForCrosstalkSources_855432_15.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/45d63b97-0953-465e-a4aa-0e0f9b84df9f_isrForCrosstalkSources_855432_15.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash e2dbd552593b825b5d1df890799be66f parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b4e4f868-3b79-4aa9-968e-e7329ef13d2f_isrForCrosstalkSources_855447_18.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b4e4f868-3b79-4aa9-968e-e7329ef13d2f_isrForCrosstalkSources_855447_18.stderr parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash a8f831d41e1578ee331d05eac76ed93d parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0a34c0bb-3a02-40be-ae12-d37bab388cb2_isrForCrosstalkSources_855389_61.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0a34c0bb-3a02-40be-ae12-d37bab388cb2_isrForCrosstalkSources_855389_61.stderr parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 084064fe3dfbaa988d72d2f6625a40b2 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/fd1038fa-9a73-4199-8741-a92191e685a1_isrForCrosstalkSources_855433_54.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/fd1038fa-9a73-4199-8741-a92191e685a1_isrForCrosstalkSources_855433_54.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 4a9715c1206916dd43dff3455664c272 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3c92488-ec2f-4b32-ab63-bd4acd9538e1_isrForCrosstalkSources_855414_56.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3c92488-ec2f-4b32-ab63-bd4acd9538e1_isrForCrosstalkSources_855414_56.stderr parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 81c7980a8e8a5e60acd148f28dbc912e parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/14d3c164-2130-49b1-9a75-9acd41e1451e_isrForCrosstalkSources_855428_52.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/14d3c164-2130-49b1-9a75-9acd41e1451e_isrForCrosstalkSources_855428_52.stderr parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 6dec22420f1920e9ecf55a9ca1493fcc parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/84bad68c-fc94-4562-ae85-865a48b3677c_isrForCrosstalkSources_855431_1.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/84bad68c-fc94-4562-ae85-865a48b3677c_isrForCrosstalkSources_855431_1.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash fd123ba388b9de74d5cf773ffea028db parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f57e8385-6449-49b7-9745-5330bbd2c4bb_isrForCrosstalkSources_855383_31.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f57e8385-6449-49b7-9745-5330bbd2c4bb_isrForCrosstalkSources_855383_31.stderr parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 1fa3c7c4ab6a1809b4c439081a0ee429 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/521deab0-5edf-47e5-9aa6-b3bf66255377_isrForCrosstalkSources_855418_4.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/521deab0-5edf-47e5-9aa6-b3bf66255377_isrForCrosstalkSources_855418_4.stderr parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 9dc307c2facf84d0e7d65962d25fe585 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/bbe51069-a6bd-4e23-9f72-b0bb6f08a173_isrForCrosstalkSources_855402_33.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/bbe51069-a6bd-4e23-9f72-b0bb6f08a173_isrForCrosstalkSources_855402_33.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash edfe31528eb6272cb6b97fb19e551518 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c980437d-e7fb-4f10-b38b-6d61373f9a16_isrForCrosstalkSources_855420_10.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c980437d-e7fb-4f10-b38b-6d61373f9a16_isrForCrosstalkSources_855420_10.stderr parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 05b294867334d051582ea43076e49de3 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d0a4a43e-ba3d-4db7-8a1c-8b05efe824dd_isrForCrosstalkSources_855430_31.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d0a4a43e-ba3d-4db7-8a1c-8b05efe824dd_isrForCrosstalkSources_855430_31.stderr parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash fef3cae623976360c4e8a47e660c9a91 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1ce5ecd6-8095-4746-8f99-2a80c88fd89e_isrForCrosstalkSources_855437_18.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1ce5ecd6-8095-4746-8f99-2a80c88fd89e_isrForCrosstalkSources_855437_18.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash f8b4f01d9c120696c960afc0361f50d2 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/23792329-2a57-459e-a538-3851434165e3_isrForCrosstalkSources_855418_23.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/23792329-2a57-459e-a538-3851434165e3_isrForCrosstalkSources_855418_23.stderr parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 574ece74f4ba0a61d57e78138e7ff990 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ecf76e-202f-4374-879c-539f6aa87723_isrForCrosstalkSources_855437_5.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ecf76e-202f-4374-879c-539f6aa87723_isrForCrosstalkSources_855437_5.stderr parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 066e523aa154e2aa5affbb4f039ce72f parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bae434fa-bbcf-4126-b535-e9ecf2e48984_isrForCrosstalkSources_855430_22.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bae434fa-bbcf-4126-b535-e9ecf2e48984_isrForCrosstalkSources_855430_22.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash fa4db97e84469f50a36e81424e1e8225 parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/54e5c7d0-ee1e-49bc-95c5-154d2a10c4f7_isrForCrosstalkSources_855428_43.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/54e5c7d0-ee1e-49bc-95c5-154d2a10c4f7_isrForCrosstalkSources_855428_43.stderr parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash b92f3396fb71ab2e7c0a070f28582b8c parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ef8f681c-dc55-430b-8a52-4bca83acfa5e_isrForCrosstalkSources_855453_21.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ef8f681c-dc55-430b-8a52-4bca83acfa5e_isrForCrosstalkSources_855453_21.stderr parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 0aad0418e1837f4ff1cf3d4b8c1aa174 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8b944c93-4702-4153-a036-f3bd14db8838_isrForCrosstalkSources_855430_26.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8b944c93-4702-4153-a036-f3bd14db8838_isrForCrosstalkSources_855430_26.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash bfac1c92571fbdca5f62d17127903e83 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0dcb8e5c-c3f3-427d-be4e-89eb386f7f08_isrForCrosstalkSources_855438_29.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0dcb8e5c-c3f3-427d-be4e-89eb386f7f08_isrForCrosstalkSources_855438_29.stderr parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 6270e987da110feada2fbb31dfdb05e9 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/734f7c68-4483-4c19-9879-c3a37cb1321a_isrForCrosstalkSources_855381_30.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/734f7c68-4483-4c19-9879-c3a37cb1321a_isrForCrosstalkSources_855381_30.stderr parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 75cc1205d3fbd79cce83581df4131abc parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8d1c4f5e-6008-44ef-8c38-d07aa00f6b65_isrForCrosstalkSources_855431_50.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8d1c4f5e-6008-44ef-8c38-d07aa00f6b65_isrForCrosstalkSources_855431_50.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 6a2e0bc0b7235ed30c2cb32842dc7c60 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c2930f49-0bfc-4e67-a3c5-6fcc14eb9fa1_isrForCrosstalkSources_855402_27.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c2930f49-0bfc-4e67-a3c5-6fcc14eb9fa1_isrForCrosstalkSources_855402_27.stderr parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 3dfa2b31ba22c650cc81d8315b7f61d4 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3e6618a-db6d-4deb-8db3-49b25607a382_isrForCrosstalkSources_855431_35.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3e6618a-db6d-4deb-8db3-49b25607a382_isrForCrosstalkSources_855431_35.stderr parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 79d010714ebd8988ca1453e2503196a8 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b6a923a3-7ee6-48b5-8ae2-3eb6259b4d4a_isrForCrosstalkSources_855380_30.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b6a923a3-7ee6-48b5-8ae2-3eb6259b4d4a_isrForCrosstalkSources_855380_30.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash ad3f850d65714701e3404e2d651202a7 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/38bd0c46-f376-43f2-ba18-a464615af381_isrForCrosstalkSources_855394_27.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/38bd0c46-f376-43f2-ba18-a464615af381_isrForCrosstalkSources_855394_27.stderr parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 7dfd00c4d4872e50767625250a87d240 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/28d6aa5b-ad88-4f7e-b4fa-c2c34819972a_isrForCrosstalkSources_855424_4.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/28d6aa5b-ad88-4f7e-b4fa-c2c34819972a_isrForCrosstalkSources_855424_4.stderr parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash e66789a9bf6b6b744580b60a753fb617 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31b98e7a-4e0e-484d-837d-086ab79dd0b9_isrForCrosstalkSources_855401_60.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31b98e7a-4e0e-484d-837d-086ab79dd0b9_isrForCrosstalkSources_855401_60.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 5909c988984f93cafdcbe8b8e00c823d parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8a64f2c-30aa-4a24-994c-c233779a41e2_isrForCrosstalkSources_855412_54.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8a64f2c-30aa-4a24-994c-c233779a41e2_isrForCrosstalkSources_855412_54.stderr parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash a6e9d6e7f0b009aad7b406d6f3354c2b parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c120748b-f606-40c9-8d5c-a8bfa1130cc9_isrForCrosstalkSources_855383_38.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c120748b-f606-40c9-8d5c-a8bfa1130cc9_isrForCrosstalkSources_855383_38.stderr parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 3bbec0a6d68f864902235df40f9cc059 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fd0e242-a6bf-43a7-a3ef-9a26a518f5f0_isrForCrosstalkSources_855387_47.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fd0e242-a6bf-43a7-a3ef-9a26a518f5f0_isrForCrosstalkSources_855387_47.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash b5f981fc77b92b7d83e9a576b608f417 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a8d6b9c-fea8-4942-a09d-e670727a493a_isrForCrosstalkSources_855410_18.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a8d6b9c-fea8-4942-a09d-e670727a493a_isrForCrosstalkSources_855410_18.stderr parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 37f04303242f8baea72dc8319506ae7a parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7ff971a8-4032-4cac-8fac-eda54c6c2de5_isrForCrosstalkSources_855450_11.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7ff971a8-4032-4cac-8fac-eda54c6c2de5_isrForCrosstalkSources_855450_11.stderr parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 8ab92ee6b150d12512bc1df595bf8768 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/af954b42-e94b-4e79-940c-4b89478c1f74_isrForCrosstalkSources_855456_38.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/af954b42-e94b-4e79-940c-4b89478c1f74_isrForCrosstalkSources_855456_38.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash e8dbdc7a5b3434397451c68f1df181f7 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964044 proc_lsst.multi INFO: [multi] job_id 16964044 2024-03-09 14:50:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 14:50:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964044 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964044 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16964044 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-09 14:50:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-09 14:50:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 14:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f6c121fb-9999-4892-b2c9-1a9eb2294e9e_isrForCrosstalkSources_855421_32.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f6c121fb-9999-4892-b2c9-1a9eb2294e9e_isrForCrosstalkSources_855421_32.stderr parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash dd149253911e77206f8bf4f99771326d parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c3e7f722-2877-4b03-b780-d8b3ff56bb23_isrForCrosstalkSources_855423_29.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c3e7f722-2877-4b03-b780-d8b3ff56bb23_isrForCrosstalkSources_855423_29.stderr parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash a994c33ef3ae34036a588196caeb972b parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6a791e39-583e-4ee1-bf01-21dfbeec9c4f_isrForCrosstalkSources_855415_12.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6a791e39-583e-4ee1-bf01-21dfbeec9c4f_isrForCrosstalkSources_855415_12.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 34c70c85923476c34351f7440c4d3587 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/68e3bea7-ed25-4f25-b043-e855fc921a37_isrForCrosstalkSources_855447_30.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/68e3bea7-ed25-4f25-b043-e855fc921a37_isrForCrosstalkSources_855447_30.stderr parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 31ad507c4cb18974a9595d42cbde4052 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/538b97cd-52ab-42c0-8956-7ff0944dfd6c_isrForCrosstalkSources_855381_51.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/538b97cd-52ab-42c0-8956-7ff0944dfd6c_isrForCrosstalkSources_855381_51.stderr parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash b84d7e4dfba3368e8df0ce8da3c7e3b1 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5485db8c-1a4e-434f-8dbd-9711f16bf8f9_isrForCrosstalkSources_855402_54.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5485db8c-1a4e-434f-8dbd-9711f16bf8f9_isrForCrosstalkSources_855402_54.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 44c735e76ed99b0438d279539ef7a341 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/55991eaa-8e87-4583-98ac-1c38cb33675f_isrForCrosstalkSources_855394_43.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/55991eaa-8e87-4583-98ac-1c38cb33675f_isrForCrosstalkSources_855394_43.stderr parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash d8bfac9268e5191f3f61bb2fd40c4f6a parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/86be32e1-3057-4ef3-a2b2-1a5fe7d59be3_isrForCrosstalkSources_855404_25.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/86be32e1-3057-4ef3-a2b2-1a5fe7d59be3_isrForCrosstalkSources_855404_25.stderr parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 8e3de34d5aaa91c108534005e8e14736 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/921ea495-ced7-412a-9077-9bd594360a30_isrForCrosstalkSources_855442_10.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/921ea495-ced7-412a-9077-9bd594360a30_isrForCrosstalkSources_855442_10.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 6f4e7ff8985d29e020fe2de1187003b8 parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d39055aa-a547-44ab-a735-302ca7f86e24_isrForCrosstalkSources_855419_55.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d39055aa-a547-44ab-a735-302ca7f86e24_isrForCrosstalkSources_855419_55.stderr parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash b2b3cb7e980f75dbf38951dbc8083322 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/93cecb44-fb90-46c5-99d1-374323a2dd69_isrForCrosstalkSources_855436_26.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/93cecb44-fb90-46c5-99d1-374323a2dd69_isrForCrosstalkSources_855436_26.stderr parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 2e38b216ddfc70c58ac595094d0d9ba4 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4b869a4f-f288-49ef-bd7d-255b1a5fc436_isrForCrosstalkSources_855439_17.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4b869a4f-f288-49ef-bd7d-255b1a5fc436_isrForCrosstalkSources_855439_17.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 8e546cde3b351086b9bb9d4f19783dbb parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/6c8163dd-68df-43c1-9d83-2abd6f6af59f_isrForCrosstalkSources_855423_23.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/6c8163dd-68df-43c1-9d83-2abd6f6af59f_isrForCrosstalkSources_855423_23.stderr parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash f7f05605c67e15ee45337303b553cb77 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/65926a1c-8ccd-42dc-87d7-8f8e990bfcc9_isrForCrosstalkSources_855403_8.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/65926a1c-8ccd-42dc-87d7-8f8e990bfcc9_isrForCrosstalkSources_855403_8.stderr parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 673f9acedec57660448dd6f2da2fdbe2 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b4d9fa0e-b7df-40bb-85e1-7a24aa680dc6_isrForCrosstalkSources_855428_54.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b4d9fa0e-b7df-40bb-85e1-7a24aa680dc6_isrForCrosstalkSources_855428_54.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash c2e23cc21d3ceeee4256839f85286c3c parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4eca5b12-fa24-4509-97d5-1b9efe3b6b74_isrForCrosstalkSources_855440_26.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4eca5b12-fa24-4509-97d5-1b9efe3b6b74_isrForCrosstalkSources_855440_26.stderr parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash c08b72751812fed82e67ec50c4968257 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/026d8372-33ca-4a94-a57d-8eafae82bf37_isrForCrosstalkSources_855417_58.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/026d8372-33ca-4a94-a57d-8eafae82bf37_isrForCrosstalkSources_855417_58.stderr parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 3aa846290a2de4d1502df4db231d35c5 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c453eb69-1b9c-40f6-b221-a271e9960bfc_isrForCrosstalkSources_855404_47.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c453eb69-1b9c-40f6-b221-a271e9960bfc_isrForCrosstalkSources_855404_47.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 5108f8f50cced2c11089047c18baf550 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0f7cb022-e69c-456f-96b8-3fe44eb5e868_isrForCrosstalkSources_855406_30.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0f7cb022-e69c-456f-96b8-3fe44eb5e868_isrForCrosstalkSources_855406_30.stderr parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 152a1755383de392c0fadd980cae8e74 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/37c03dbb-48d3-4cf4-b402-7ce6e6062c38_isrForCrosstalkSources_855454_16.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/37c03dbb-48d3-4cf4-b402-7ce6e6062c38_isrForCrosstalkSources_855454_16.stderr parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash d7638607d37f5e7e38003663108d6a74 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f99e1f06-dcbf-40c2-9483-6df1b5a98da3_isrForCrosstalkSources_855455_32.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f99e1f06-dcbf-40c2-9483-6df1b5a98da3_isrForCrosstalkSources_855455_32.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 3a85d16f73b13d673b0754aca3d22e13 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d63f9f8a-0697-4dcd-a23e-4bd3ecf9a555_isrForCrosstalkSources_855397_43.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d63f9f8a-0697-4dcd-a23e-4bd3ecf9a555_isrForCrosstalkSources_855397_43.stderr parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 3f40cb051793a88fa966f6c45df5f09e parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b1bef419-9ec6-4363-833b-95c66e6478b1_isrForCrosstalkSources_855440_28.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b1bef419-9ec6-4363-833b-95c66e6478b1_isrForCrosstalkSources_855440_28.stderr parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash a1c817d95a52e5a7388d595280facd8c parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80431459-e8c9-4acc-aee6-828824343d85_isrForCrosstalkSources_855426_54.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80431459-e8c9-4acc-aee6-828824343d85_isrForCrosstalkSources_855426_54.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 79a29ed967c527a2953590d5bbd0c7d2 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/d8bf6efa-c5db-4875-97a9-710d5b31d03e_isrForCrosstalkSources_855398_37.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/d8bf6efa-c5db-4875-97a9-710d5b31d03e_isrForCrosstalkSources_855398_37.stderr parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 1e7a458c5c77278ec905ecaa6e2adf22 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stderr parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 8c3e06d0ae95866fbe420644ed949c6e parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash eee190888085a014e2496ec72d0baad1 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stderr parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 05794fc12ed60142732abdaed4d25bec parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stderr parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 5b206c1e89c81ea81fde7bb005171730 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash daf3d1daf138b382a49930693605b896 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2402c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stderr parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 71ea44cf6efbebf6de2ba24206180614 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stderr parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash b2d020d98c9f6fb6df30831818517ee0 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 415f643c16d33fb8b9172458dab79d6b parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stderr parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash b3f23a67a0d40f776cdd5221d20f985a parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stderr parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 9952d59c8ebfd5b0565529b1da2bcc9c parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2405e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 4d6aad113247ed5ac8eb070948cdc11f parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stderr parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 972c49fedc3efbbf9f277a960b822c55 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stderr parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 347d4182371e0f83516a93341e40f981 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash ec11fa2fffad88bc471b77b669a7a928 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stderr parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 63334f9bd3766f10de65a7c9d6cd1930 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stderr parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 456489b176c6b9e6ba5c51b7a16706df parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0f551a5a-0782-48a0-b411-eaf4aa75cf9b_isrForCrosstalkSources_855459_34.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0f551a5a-0782-48a0-b411-eaf4aa75cf9b_isrForCrosstalkSources_855459_34.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 8a884e924730dd0715d5142e0111b9bb parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7d28587b-cc8a-447c-a7bd-d371526a70a8_isrForCrosstalkSources_855439_31.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7d28587b-cc8a-447c-a7bd-d371526a70a8_isrForCrosstalkSources_855439_31.stderr parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash b6376c9981d082954edcd80eec3265b3 parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ef9df52d-7b94-4606-9f91-e41014b2e458_isrForCrosstalkSources_855455_11.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ef9df52d-7b94-4606-9f91-e41014b2e458_isrForCrosstalkSources_855455_11.stderr parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash fd912447b2a2780bb752578623df423a parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5eda16c6-d840-4ea4-8f87-37ce09c11f90_isrForCrosstalkSources_855419_25.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5eda16c6-d840-4ea4-8f87-37ce09c11f90_isrForCrosstalkSources_855419_25.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash c6f6187be6a1855315e997e3af9328a3 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6b3b6acd-242d-48e9-a3a9-0a912761cd68_isrForCrosstalkSources_855402_8.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6b3b6acd-242d-48e9-a3a9-0a912761cd68_isrForCrosstalkSources_855402_8.stderr parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 1a2a8fcae35182cb716afb85a9a7b641 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3e79a0f8-e824-44b2-9201-38bd2ed6ef31_isrForCrosstalkSources_855387_60.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3e79a0f8-e824-44b2-9201-38bd2ed6ef31_isrForCrosstalkSources_855387_60.stderr parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash f2b81a540a46e6a556ad3f76b1656202 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964045 proc_lsst.multi INFO: [multi] job_id 16964045 2024-03-09 14:50:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964045 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964045 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16964045 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-09 14:50:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-09 14:50:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e8e14a1a-bae1-4a68-90d5-8efde4cd4158_isrForCrosstalkSources_855405_25.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e8e14a1a-bae1-4a68-90d5-8efde4cd4158_isrForCrosstalkSources_855405_25.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash cf6fcf0aafde3e4ddc76afc2697321ed parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7b76203c-6b18-4799-8a4d-8827bb4ad7ee_isrForCrosstalkSources_855405_10.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7b76203c-6b18-4799-8a4d-8827bb4ad7ee_isrForCrosstalkSources_855405_10.stderr parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash bbf035899cec50cf0b485decda337f7d parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f94ad932-9544-4442-a653-2e11b312f288_isrForCrosstalkSources_855459_19.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f94ad932-9544-4442-a653-2e11b312f288_isrForCrosstalkSources_855459_19.stderr parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 15ac2abed5a8f8da6f9cf844961200e3 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8d51b0e1-9341-48d7-8e1d-b875846fce03_isrForCrosstalkSources_855384_9.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8d51b0e1-9341-48d7-8e1d-b875846fce03_isrForCrosstalkSources_855384_9.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash e9a4fddfd0bd21acadf8a3754e76ca7b parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3b2c964b-cd63-414e-a8b5-8c72e6e08a63_isrForCrosstalkSources_855432_23.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3b2c964b-cd63-414e-a8b5-8c72e6e08a63_isrForCrosstalkSources_855432_23.stderr parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash ed24c34e6582a8846681d6efc859972f parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8a783725-7adf-4362-98ea-c9ea91dd79ba_isrForCrosstalkSources_855384_44.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8a783725-7adf-4362-98ea-c9ea91dd79ba_isrForCrosstalkSources_855384_44.stderr parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 5771aaa339b5d242d86a4bae37feaf08 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/897087bf-9ffa-4950-9c34-3113d3204a4c_isrForCrosstalkSources_855381_40.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/897087bf-9ffa-4950-9c34-3113d3204a4c_isrForCrosstalkSources_855381_40.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash a5834dbc9b78b3eda4ec2f2a7815efdd parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/26af3ad5-f2b9-4441-a660-f9746383f2e1_isrForCrosstalkSources_855431_48.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/26af3ad5-f2b9-4441-a660-f9746383f2e1_isrForCrosstalkSources_855431_48.stderr parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 84b7579879bbee6e5a1d643fdd8afaff parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2411c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6554d286-eb4b-4e3c-8961-62f63576a2fd_isrForCrosstalkSources_855452_33.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6554d286-eb4b-4e3c-8961-62f63576a2fd_isrForCrosstalkSources_855452_33.stderr parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash e2fe8c88d66564a12e61f0993a3caea9 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c05f929b-0925-44bf-a835-62eaff6e6ed0_isrForCrosstalkSources_855417_12.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c05f929b-0925-44bf-a835-62eaff6e6ed0_isrForCrosstalkSources_855417_12.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 9d1a10abd21bd14ec7de1f7c6c6285c0 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2413a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/28c8a0c6-45f5-4353-8210-8c0261927a5d_isrForCrosstalkSources_855452_44.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/28c8a0c6-45f5-4353-8210-8c0261927a5d_isrForCrosstalkSources_855452_44.stderr parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 187f660dcf7f4aebfd912abf9ad89ae7 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/cc1b6ddd-7074-4752-b17b-dd79649c5a41_isrForCrosstalkSources_855423_30.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/cc1b6ddd-7074-4752-b17b-dd79649c5a41_isrForCrosstalkSources_855423_30.stderr parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 575ea955f8aaa94d692a15bf8b36c57e parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dcb50532-0c8a-4824-8bd9-75124288d3b4_isrForCrosstalkSources_855387_44.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dcb50532-0c8a-4824-8bd9-75124288d3b4_isrForCrosstalkSources_855387_44.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 11b5db86479e9b6bb8e6430982120898 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8c1eed94-7ee6-47d1-9660-946d82be5665_isrForCrosstalkSources_855379_60.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8c1eed94-7ee6-47d1-9660-946d82be5665_isrForCrosstalkSources_855379_60.stderr parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 6c9ad25a32ef8ad6b8935d054274367e parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/75323d62-087f-4d7c-b288-416f9bfba915_isrForCrosstalkSources_855421_17.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/75323d62-087f-4d7c-b288-416f9bfba915_isrForCrosstalkSources_855421_17.stderr parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash ac0a28b18a9c1cf734962f28a5762549 parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1d679a65-4529-480c-ae1a-bb05a4a3d4ac_isrForCrosstalkSources_855417_42.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1d679a65-4529-480c-ae1a-bb05a4a3d4ac_isrForCrosstalkSources_855417_42.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash a8ad221a5ea3fe4a1fb4bc181f2c4922 parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/427d8e8c-bf40-474a-8030-e8941803dabd_isrForCrosstalkSources_855452_62.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/427d8e8c-bf40-474a-8030-e8941803dabd_isrForCrosstalkSources_855452_62.stderr parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 3db0abfa4a7fd40e50409342675e5d82 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/121ebf3b-3849-40aa-9302-587cebd88fc9_isrForCrosstalkSources_855399_5.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/121ebf3b-3849-40aa-9302-587cebd88fc9_isrForCrosstalkSources_855399_5.stderr parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash 7ba28ee95b04b9885047c8b78c3c6c37 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fcfc4187-7123-4c67-96d8-559e429cb1af_isrForCrosstalkSources_855379_52.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fcfc4187-7123-4c67-96d8-559e429cb1af_isrForCrosstalkSources_855379_52.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 46c33b159d04989558b83cf5022510f4 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2416c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a879df5c-27c5-4512-ac54-b52bffadec70_isrForCrosstalkSources_855449_30.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a879df5c-27c5-4512-ac54-b52bffadec70_isrForCrosstalkSources_855449_30.stderr parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash d489c72138927b25a9385a845c7fe155 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/aff462e7-f22c-476f-aad3-59503a30d98a_isrForCrosstalkSources_855421_28.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/aff462e7-f22c-476f-aad3-59503a30d98a_isrForCrosstalkSources_855421_28.stderr parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash fd03808e885ba3500b50648841ba45ef parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2418a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/685d9ab0-5608-4e25-a0fe-3c1d5cfca3ed_isrForCrosstalkSources_855391_6.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/685d9ab0-5608-4e25-a0fe-3c1d5cfca3ed_isrForCrosstalkSources_855391_6.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash bcea4e19a5b842d2db8706f644567d2a parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2419e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/2c97c79f-bf31-4122-b70d-10616e738c19_isrForCrosstalkSources_855390_27.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/2c97c79f-bf31-4122-b70d-10616e738c19_isrForCrosstalkSources_855390_27.stderr parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 9e255883d04e7266d1a613373a940c6b parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a849958-2c45-45bc-a6ab-d78ae65b674c_isrForCrosstalkSources_855452_24.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a849958-2c45-45bc-a6ab-d78ae65b674c_isrForCrosstalkSources_855452_24.stderr parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 83bcaa644f71a7f4660f6776725ca8ab parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964046 proc_lsst.multi INFO: [multi] job_id 16964046 2024-03-09 14:50:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 14:50:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964046 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964046 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16964046 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-09 14:50:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-09 14:50:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 14:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/573b230b-f219-4590-a324-f7a519aee8a6_isrForCrosstalkSources_855454_52.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/573b230b-f219-4590-a324-f7a519aee8a6_isrForCrosstalkSources_855454_52.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 14a4fb52c8471cca0f9a1327605df080 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0d1523c6-49f7-405f-8d86-fc91de1c924b_isrForCrosstalkSources_855387_41.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0d1523c6-49f7-405f-8d86-fc91de1c924b_isrForCrosstalkSources_855387_41.stderr parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash f7ca4d55b52ee116a43c17985cc51bae parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/f27a13d6-19b9-42d9-8a96-17112f717e44_isrForCrosstalkSources_855394_5.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/f27a13d6-19b9-42d9-8a96-17112f717e44_isrForCrosstalkSources_855394_5.stderr parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 1f01a5111397547aa3baa9692c0bf76b parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8e25f42f-892a-4dcf-8fdf-0bca894c26c8_isrForCrosstalkSources_855431_33.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8e25f42f-892a-4dcf-8fdf-0bca894c26c8_isrForCrosstalkSources_855431_33.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash b2d1738dac571496975b6e0dff78c3e2 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ec47a10d-5d61-45a8-a5d8-c597c22a67f2_isrForCrosstalkSources_855444_26.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ec47a10d-5d61-45a8-a5d8-c597c22a67f2_isrForCrosstalkSources_855444_26.stderr parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 594c3b2f16758d3084a587bb9367cddb parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c37ee41c-721e-44c3-9815-0907752c6f28_isrForCrosstalkSources_855430_18.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c37ee41c-721e-44c3-9815-0907752c6f28_isrForCrosstalkSources_855430_18.stderr parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 63551dd6407035802babc7c5077daa92 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/06bf3718-9e6b-45fc-a321-56369f6965a3_isrForCrosstalkSources_855418_35.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/06bf3718-9e6b-45fc-a321-56369f6965a3_isrForCrosstalkSources_855418_35.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 01a2de9c60c6585bf7c4c40e5d0aa642 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31999c57-6a28-46a9-b34c-30866d7de038_isrForCrosstalkSources_855401_34.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31999c57-6a28-46a9-b34c-30866d7de038_isrForCrosstalkSources_855401_34.stderr parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash aab7c0bc34a6cc3aa252c89242a466d5 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ebb7d6ee-a17d-4270-8beb-e421191801d6_isrForCrosstalkSources_855432_39.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ebb7d6ee-a17d-4270-8beb-e421191801d6_isrForCrosstalkSources_855432_39.stderr parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 303c72fa28075959dccc52198516a5c1 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2c7611f7-2596-443c-afaf-5bbdae65f3ac_isrForCrosstalkSources_855429_43.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2c7611f7-2596-443c-afaf-5bbdae65f3ac_isrForCrosstalkSources_855429_43.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 8c86ca933d2e87b70308c8e9e2c13cf3 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c7a6ecef-f620-48c3-b745-629fc138cf4f_isrForCrosstalkSources_855406_54.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c7a6ecef-f620-48c3-b745-629fc138cf4f_isrForCrosstalkSources_855406_54.stderr parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 9500c044e741a99d4771ed1ebff15b7c parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/90e60d38-566e-430f-b2ab-d2f74ea27d00_isrForCrosstalkSources_855446_38.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/90e60d38-566e-430f-b2ab-d2f74ea27d00_isrForCrosstalkSources_855446_38.stderr parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 5be0c1eafcfb3db729593ee63b869da7 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/567b334c-0e93-4778-9419-61fa562fd9d4_isrForCrosstalkSources_855454_11.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/567b334c-0e93-4778-9419-61fa562fd9d4_isrForCrosstalkSources_855454_11.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 6f3039975af14ca3e3016e5b5f0f1087 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/6a86f019-0589-40e1-bc43-082831cb0e9c_isrForCrosstalkSources_855455_9.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/6a86f019-0589-40e1-bc43-082831cb0e9c_isrForCrosstalkSources_855455_9.stderr parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash e88b4c6f5bb9178aea5cf6bc294ea9ea parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d5d5af3a-b6de-4fdd-af6f-3c21caf316a3_isrForCrosstalkSources_855428_50.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d5d5af3a-b6de-4fdd-af6f-3c21caf316a3_isrForCrosstalkSources_855428_50.stderr parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash ab72f2a978f637aed72439ba09664c9f parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4cb0f006-78a1-474e-8882-eb3638afdb3f_isrForCrosstalkSources_855419_56.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4cb0f006-78a1-474e-8882-eb3638afdb3f_isrForCrosstalkSources_855419_56.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 68c3d25b4356f86b195213a000ad99f1 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/204de22e-257e-41bb-9e30-e8de5a3d87e9_isrForCrosstalkSources_855403_24.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/204de22e-257e-41bb-9e30-e8de5a3d87e9_isrForCrosstalkSources_855403_24.stderr parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 00bbe1213e1268a466c4bb2b35d43129 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/87275bf1-d210-4b6f-8a0d-d2b2af963a39_isrForCrosstalkSources_855444_56.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/87275bf1-d210-4b6f-8a0d-d2b2af963a39_isrForCrosstalkSources_855444_56.stderr parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 3cc17ef991710e80a0eda45dc7405283 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2b420621-fd7e-4b7d-ac82-45773bf9b9e1_isrForCrosstalkSources_855379_44.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2b420621-fd7e-4b7d-ac82-45773bf9b9e1_isrForCrosstalkSources_855379_44.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 1cac51e42ee7659764c34c01c6cd043d parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/136de2ca-4868-4af3-bf02-a48ca8633e3c_isrForCrosstalkSources_855436_36.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/136de2ca-4868-4af3-bf02-a48ca8633e3c_isrForCrosstalkSources_855436_36.stderr parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 3732adb2301a0b7adfc32ff6b89e5ade parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e0ef2bb4-a91c-4924-9e3d-95af47d8cc83_isrForCrosstalkSources_855421_38.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e0ef2bb4-a91c-4924-9e3d-95af47d8cc83_isrForCrosstalkSources_855421_38.stderr parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 8ae53897be06e10f8354a56c1c287639 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5d45219b-9d98-49bf-b41d-9af66e9bc7d2_isrForCrosstalkSources_855427_36.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5d45219b-9d98-49bf-b41d-9af66e9bc7d2_isrForCrosstalkSources_855427_36.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 873bbb5d243d196b873fa6f314917056 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/506fb34d-8206-4211-b383-db79b5ba854d_isrForCrosstalkSources_855385_54.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/506fb34d-8206-4211-b383-db79b5ba854d_isrForCrosstalkSources_855385_54.stderr parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 2365e5ac0df7c59db2dbf4901f89a2f9 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2427a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/45858fd8-515f-47c1-972c-9b620b84d7d1_isrForCrosstalkSources_855450_1.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/45858fd8-515f-47c1-972c-9b620b84d7d1_isrForCrosstalkSources_855450_1.stderr parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 47f1fd9b76b9c29012de8cecebdcab59 parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/82f9f1c2-f793-421a-8f9e-8b3b56f2850e_isrForCrosstalkSources_855381_44.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/82f9f1c2-f793-421a-8f9e-8b3b56f2850e_isrForCrosstalkSources_855381_44.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 13b05f548df41b4c12ad20333c8a5eca parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f3a5bbd7-8fd3-4a81-8dab-ffa16d5441ba_isrForCrosstalkSources_855445_12.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f3a5bbd7-8fd3-4a81-8dab-ffa16d5441ba_isrForCrosstalkSources_855445_12.stderr parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 0b24429508f0cbc47187692b7cd29311 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8cfa0af3-1626-4039-9939-d12b7c7a1416_isrForCrosstalkSources_855426_17.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8cfa0af3-1626-4039-9939-d12b7c7a1416_isrForCrosstalkSources_855426_17.stderr parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 2b543843a842cc48a6c309ec44bbaeac parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/63fbdde6-5b23-4f2a-ae41-e35cdca51cac_isrForCrosstalkSources_855434_34.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/63fbdde6-5b23-4f2a-ae41-e35cdca51cac_isrForCrosstalkSources_855434_34.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash d772d65c82ca2590a4c4266eb320f0c8 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/80dc4fb6-865d-4049-83e5-e5fb9d8393b6_isrForCrosstalkSources_855379_27.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/80dc4fb6-865d-4049-83e5-e5fb9d8393b6_isrForCrosstalkSources_855379_27.stderr parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash b61cad0310bad243fb803db772ee5a6a parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964047 proc_lsst.multi INFO: [multi] job_id 16964047 2024-03-09 14:50:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 14:50:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964047 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964047 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16964047 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-09 14:50:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-09 14:50:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 14:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58182ce0-5f4e-4275-84a1-c45beeab8266_isrForCrosstalkSources_855407_25.stdout parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58182ce0-5f4e-4275-84a1-c45beeab8266_isrForCrosstalkSources_855407_25.stderr parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 32b83ca19d7ac6c514e3e3ab66357323 parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/95f5abde-c49d-4810-97d1-3a11d464d57f_isrForCrosstalkSources_855382_46.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/95f5abde-c49d-4810-97d1-3a11d464d57f_isrForCrosstalkSources_855382_46.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 6398c4f8f9964a0d044885e361818c07 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2420c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/27f88db1-561c-4448-8afc-ee4ba79073da_isrForCrosstalkSources_855397_7.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/27f88db1-561c-4448-8afc-ee4ba79073da_isrForCrosstalkSources_855397_7.stderr parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash eb7ff246fb080d37ab4ddfab11ec7e00 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ea1da352-bc89-48d1-b795-ebce2a1e4f67_isrForCrosstalkSources_855401_2.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ea1da352-bc89-48d1-b795-ebce2a1e4f67_isrForCrosstalkSources_855401_2.stderr parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 7f41f19b90bb86a27920a3f2ebea1d18 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/36537fe4-6c78-49b4-8c1b-9293e0484886_isrForCrosstalkSources_855445_22.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/36537fe4-6c78-49b4-8c1b-9293e0484886_isrForCrosstalkSources_855445_22.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash b8f98d718ea50290fb0926435d139fa8 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/38f0a6f7-7022-4fef-9bfe-318eb4d916b0_isrForCrosstalkSources_855440_48.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/38f0a6f7-7022-4fef-9bfe-318eb4d916b0_isrForCrosstalkSources_855440_48.stderr parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 550c2342ac0273ce736deca147192289 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8a01e716-1a06-4128-9403-d28a66e28cd4_isrForCrosstalkSources_855431_57.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8a01e716-1a06-4128-9403-d28a66e28cd4_isrForCrosstalkSources_855431_57.stderr parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 72d231f71f65d5f8cee714e620fdd622 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9c356007-d17c-49bb-a2d3-e16a3ecd483e_isrForCrosstalkSources_855396_10.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9c356007-d17c-49bb-a2d3-e16a3ecd483e_isrForCrosstalkSources_855396_10.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash d73a67fb59ecb5fa990a283c3c75b4d5 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e9552b12-6573-4a8e-b29c-029338684195_isrForCrosstalkSources_855443_16.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e9552b12-6573-4a8e-b29c-029338684195_isrForCrosstalkSources_855443_16.stderr parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 53cb7a490e9e9a4dd32be41e77111bf4 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/449aab7e-5c39-4f8e-aa52-175ea5b599c0_isrForCrosstalkSources_855385_60.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/449aab7e-5c39-4f8e-aa52-175ea5b599c0_isrForCrosstalkSources_855385_60.stderr parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash f719a1de98535bb9c3e26cccb44701a6 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/40751d0e-c512-42cb-b54f-f25b11fd7fde_isrForCrosstalkSources_855402_1.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/40751d0e-c512-42cb-b54f-f25b11fd7fde_isrForCrosstalkSources_855402_1.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 24ed0fdebe233f58ac373d8dc1ff0a33 parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ae041e43-af2a-4f42-a904-d5512ffc1c10_isrForCrosstalkSources_855388_9.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ae041e43-af2a-4f42-a904-d5512ffc1c10_isrForCrosstalkSources_855388_9.stderr parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 4bbc625620899d92fd03ad5327ec5923 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d21b97e6-c5d7-4aca-94ea-b54aa9debfcb_isrForCrosstalkSources_855439_21.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d21b97e6-c5d7-4aca-94ea-b54aa9debfcb_isrForCrosstalkSources_855439_21.stderr parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 0bdeb857e67af0934778b0af73ecd95b parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2437e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/3c2c280f-b215-4004-96d4-10f3081c1630_isrForCrosstalkSources_855455_35.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/3c2c280f-b215-4004-96d4-10f3081c1630_isrForCrosstalkSources_855455_35.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash a6a11218381ee787a92fd4e038db4e87 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c7e73acf-a325-4ea6-b95e-899129ad20fd_isrForCrosstalkSources_855400_24.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c7e73acf-a325-4ea6-b95e-899129ad20fd_isrForCrosstalkSources_855400_24.stderr parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 5292c9db49eb0473e33377351e741ca7 parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4bcb196-143d-487d-b4f2-7528c6ab524f_isrForCrosstalkSources_855434_51.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4bcb196-143d-487d-b4f2-7528c6ab524f_isrForCrosstalkSources_855434_51.stderr parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash a245962cff82485bb73626f62bbbead0 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3ff40679-4ec4-46a9-aa11-f12c7f116106_isrForCrosstalkSources_855448_53.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3ff40679-4ec4-46a9-aa11-f12c7f116106_isrForCrosstalkSources_855448_53.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 2e535ca99cd08fc7444c30a8a4e28769 parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e69ac845-2e19-4ec0-a56f-9e233a79202e_isrForCrosstalkSources_855382_6.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e69ac845-2e19-4ec0-a56f-9e233a79202e_isrForCrosstalkSources_855382_6.stderr parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash f5ecdf0d070600325a3167a18f1afac7 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2434c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9fd08a27-0612-46a9-9685-0eafb5cde0fb_isrForCrosstalkSources_855437_48.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9fd08a27-0612-46a9-9685-0eafb5cde0fb_isrForCrosstalkSources_855437_48.stderr parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 4e1917d2eb4fc993f7321fc2642fe195 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9a7ebf9a-fff8-4d0f-8245-b56304172dcd_isrForCrosstalkSources_855383_42.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9a7ebf9a-fff8-4d0f-8245-b56304172dcd_isrForCrosstalkSources_855383_42.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 761c5d10f22ad8878b5796d0df2e79e3 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c532a6e8-f216-4913-8873-9332ef81cda8_isrForCrosstalkSources_855389_52.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c532a6e8-f216-4913-8873-9332ef81cda8_isrForCrosstalkSources_855389_52.stderr parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash fac9973091e3a47ba74611d606b627e8 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/15e8b1f3-1652-4878-9f6e-3707a892d2ad_isrForCrosstalkSources_855438_46.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/15e8b1f3-1652-4878-9f6e-3707a892d2ad_isrForCrosstalkSources_855438_46.stderr parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash d213174072af27e9763ab5cdd9278b56 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6270cd0a-309c-4c45-94b6-94e80ed23445_isrForCrosstalkSources_855446_19.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6270cd0a-309c-4c45-94b6-94e80ed23445_isrForCrosstalkSources_855446_19.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 2d05cac9ee482bbb7faf68c797b86986 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ec1d25b1-af6b-4f8f-9aac-68b930f219d5_isrForCrosstalkSources_855410_30.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ec1d25b1-af6b-4f8f-9aac-68b930f219d5_isrForCrosstalkSources_855410_30.stderr parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 23408013c1def771d6fc102b41c2fe28 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dd141d94-1bb1-4c9a-8945-9a3ff788b183_isrForCrosstalkSources_855450_3.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dd141d94-1bb1-4c9a-8945-9a3ff788b183_isrForCrosstalkSources_855450_3.stderr parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 976a0e4346d4dea6d15f6b1581c86d06 parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/0c0ac941-263e-4408-94a8-cee234a07647_isrForCrosstalkSources_855442_1.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/0c0ac941-263e-4408-94a8-cee234a07647_isrForCrosstalkSources_855442_1.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash de69c4075a61ec935e656cda56b43ee9 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6f993781-2545-4125-9b63-e2d111ce0d47_isrForCrosstalkSources_855446_15.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6f993781-2545-4125-9b63-e2d111ce0d47_isrForCrosstalkSources_855446_15.stderr parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 3a1ee052bdae081430893485de36f332 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964048 proc_lsst.multi INFO: [multi] job_id 16964048 2024-03-09 14:50:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 14:50:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964048 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964048 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16964048 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-09 14:50:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-09 14:50:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 14:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4ccadd59-95c0-433e-96f1-6cdae7dcfc8a_isrForCrosstalkSources_855430_59.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4ccadd59-95c0-433e-96f1-6cdae7dcfc8a_isrForCrosstalkSources_855430_59.stderr parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash cb4f48c8e4e814fbd7c7b6d3d931d9a4 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa242fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/123b546c-1daf-4a33-8a36-689709da44f5_isrForCrosstalkSources_855409_6.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/123b546c-1daf-4a33-8a36-689709da44f5_isrForCrosstalkSources_855409_6.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 2dbfb01f65b77a210aa971e64321549a parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/68a4292f-c83a-4733-a20f-12aeb7d0dce0_isrForCrosstalkSources_855402_62.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/68a4292f-c83a-4733-a20f-12aeb7d0dce0_isrForCrosstalkSources_855402_62.stderr parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash dde838332b672591b90e05e75a770637 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c696583d-8797-4f2b-bd46-6f970f041d16_isrForCrosstalkSources_855441_51.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c696583d-8797-4f2b-bd46-6f970f041d16_isrForCrosstalkSources_855441_51.stderr parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 850dac28314b91ec1539d46327d2f479 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f10da8e1-a104-4cdb-893d-736408bc2f99_isrForCrosstalkSources_855399_59.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f10da8e1-a104-4cdb-893d-736408bc2f99_isrForCrosstalkSources_855399_59.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 17280e6104c66414cacac6813e372677 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ed983a27-7c32-480a-b4bc-f8145e3ed628_isrForCrosstalkSources_855391_46.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ed983a27-7c32-480a-b4bc-f8145e3ed628_isrForCrosstalkSources_855391_46.stderr parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 1149bd7fd496ec5e469268df085a05d4 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8061285-0e0c-4815-adfe-8321397955e3_isrForCrosstalkSources_855399_60.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8061285-0e0c-4815-adfe-8321397955e3_isrForCrosstalkSources_855399_60.stderr parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 59bb82c60df328bd696df902ef7c3e9e parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6644a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4039e89b-6188-4511-a95e-5aec5a53ee23_isrForCrosstalkSources_855452_9.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4039e89b-6188-4511-a95e-5aec5a53ee23_isrForCrosstalkSources_855452_9.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash d3ffc93f12b66ca8a17bbb1df2a55cdc parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/234aa698-449c-492f-897b-ce98ff2dcd3c_isrForCrosstalkSources_855416_5.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/234aa698-449c-492f-897b-ce98ff2dcd3c_isrForCrosstalkSources_855416_5.stderr parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 440128e003734869e9eb152b32586063 parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf646472-5e99-4942-be84-8aa7ea484c7b_isrForCrosstalkSources_855402_17.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf646472-5e99-4942-be84-8aa7ea484c7b_isrForCrosstalkSources_855402_17.stderr parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 5bdb3bdbe886ebd495ed11de365ae547 parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6e14014f-d7c8-475a-bace-5fe3c4fc5bd5_isrForCrosstalkSources_855413_32.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6e14014f-d7c8-475a-bace-5fe3c4fc5bd5_isrForCrosstalkSources_855413_32.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash ec9f7b35f97a2e9fe08fc4e309046da5 parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cb429303-f6d3-44e3-82a7-f7fe08d5ef85_isrForCrosstalkSources_855458_44.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cb429303-f6d3-44e3-82a7-f7fe08d5ef85_isrForCrosstalkSources_855458_44.stderr parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash c79025349af5622437f87bae7f84cbad parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/8f2543fe-65bd-44f5-a853-c9b9bf29ef04_isrForCrosstalkSources_855389_54.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/8f2543fe-65bd-44f5-a853-c9b9bf29ef04_isrForCrosstalkSources_855389_54.stderr parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 0475255aad292936338c18f711ead941 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/432f5ace-673f-4f2e-ab4b-9b7eb9829d51_isrForCrosstalkSources_855457_27.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/432f5ace-673f-4f2e-ab4b-9b7eb9829d51_isrForCrosstalkSources_855457_27.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 55bc27256b714013c916a75ad946deea parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3489c778-eb49-46ba-823d-1f992b5be100_isrForCrosstalkSources_855382_22.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3489c778-eb49-46ba-823d-1f992b5be100_isrForCrosstalkSources_855382_22.stderr parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 17b2e9bb86432c47cc60e95c234b91a5 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d09803d3-7ccf-42c9-bff0-5c4fd4580062_isrForCrosstalkSources_855450_38.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d09803d3-7ccf-42c9-bff0-5c4fd4580062_isrForCrosstalkSources_855450_38.stderr parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash bdd13d2943704112890cfa17c3d4a5e4 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/05d6a862-74fe-4add-ae70-2de782d0bff7_isrForCrosstalkSources_855423_17.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/05d6a862-74fe-4add-ae70-2de782d0bff7_isrForCrosstalkSources_855423_17.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash bfa2e579aed7782e27b7ad561385f136 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/52f7bfea-86d2-439a-b12c-3f5913cb7a04_isrForCrosstalkSources_855379_31.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/52f7bfea-86d2-439a-b12c-3f5913cb7a04_isrForCrosstalkSources_855379_31.stderr parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash f855fefb4d368595779ab510ed21acd9 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9d3e89c6-802c-415b-b992-5369f5970c01_isrForCrosstalkSources_855431_30.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9d3e89c6-802c-415b-b992-5369f5970c01_isrForCrosstalkSources_855431_30.stderr parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 3121c58f8f7a90a6473d9d14f44ecac6 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/50fdfd62-f6bf-4928-b5f2-b6a9bc734db9_isrForCrosstalkSources_855452_53.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/50fdfd62-f6bf-4928-b5f2-b6a9bc734db9_isrForCrosstalkSources_855452_53.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash bf21f9676e89fcb94d230f2c8f6ff58a parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9931004-9fbb-4740-a5a0-c30a217adce9_isrForCrosstalkSources_855449_37.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9931004-9fbb-4740-a5a0-c30a217adce9_isrForCrosstalkSources_855449_37.stderr parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 6b211dae0b5869e189b3c430da9d68aa parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0b081061-3ea9-4979-a642-55a132124b65_isrForCrosstalkSources_855396_48.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0b081061-3ea9-4979-a642-55a132124b65_isrForCrosstalkSources_855396_48.stderr parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 6a9791b96a5d345ef4601877d9bfe036 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/99ed4abe-63cc-4b28-8654-01fe10cfbdcb_isrForCrosstalkSources_855456_18.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/99ed4abe-63cc-4b28-8654-01fe10cfbdcb_isrForCrosstalkSources_855456_18.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash f34c93ef20200f24ee1860016fe11814 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/25358229-3199-48f3-924a-7aa01718cf94_isrForCrosstalkSources_855447_27.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/25358229-3199-48f3-924a-7aa01718cf94_isrForCrosstalkSources_855447_27.stderr parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 8ea649d8858e6b1265db09d0b78901b1 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964049 proc_lsst.multi INFO: [multi] job_id 16964049 2024-03-09 14:50:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 14:50:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964049 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964049 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16964049 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-09 14:50:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-09 14:50:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 14:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2a30f6b8-f97e-40e4-9c90-5e69fa4d1019_isrForCrosstalkSources_855436_33.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2a30f6b8-f97e-40e4-9c90-5e69fa4d1019_isrForCrosstalkSources_855436_33.stderr parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 23f96d850d97811b962fbc482bb1503f parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa243d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d3e87a54-f143-4785-927c-ad530d565ae8_isrForCrosstalkSources_855419_36.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d3e87a54-f143-4785-927c-ad530d565ae8_isrForCrosstalkSources_855419_36.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 0c62e0f003406000842581055953f9c6 parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0076bd41-c4a1-4ce5-bbd3-e112beed301d_isrForCrosstalkSources_855430_62.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0076bd41-c4a1-4ce5-bbd3-e112beed301d_isrForCrosstalkSources_855430_62.stderr parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash d52065f1471569d4ae7578bf817db38c parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa241da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/099404e0-dc83-45af-900b-ad76573a69d1_isrForCrosstalkSources_855387_40.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/099404e0-dc83-45af-900b-ad76573a69d1_isrForCrosstalkSources_855387_40.stderr parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash db41771de6bad8f6a9bb80744d262923 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/80520f54-29e7-47f0-ba20-7bcc1e7ad7bb_isrForCrosstalkSources_855405_16.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/80520f54-29e7-47f0-ba20-7bcc1e7ad7bb_isrForCrosstalkSources_855405_16.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 88f92033f8f0c47086836a84341c4d5f parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a644a031-1e6a-4a3d-ac48-5ba92060c54e_isrForCrosstalkSources_855452_3.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a644a031-1e6a-4a3d-ac48-5ba92060c54e_isrForCrosstalkSources_855452_3.stderr parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash d8d964249c8bb19df887840872ae77a1 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66bafbea-660f-4546-8b0c-a04694c0ed1d_isrForCrosstalkSources_855390_46.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66bafbea-660f-4546-8b0c-a04694c0ed1d_isrForCrosstalkSources_855390_46.stderr parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 64029facfd338593c3021c9e405713ab parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ebeaa0b2-6c91-4538-a3b8-491f5468ed7e_isrForCrosstalkSources_855435_62.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ebeaa0b2-6c91-4538-a3b8-491f5468ed7e_isrForCrosstalkSources_855435_62.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash bd481b97568287aececf209b1c691ce6 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c748a012-c0c0-47e0-b6a6-18a64ccfd849_isrForCrosstalkSources_855433_13.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c748a012-c0c0-47e0-b6a6-18a64ccfd849_isrForCrosstalkSources_855433_13.stderr parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 4af181788a856a78ee820e8a14249a39 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c9b8c76d-7dca-4277-a237-af2178858396_isrForCrosstalkSources_855430_21.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c9b8c76d-7dca-4277-a237-af2178858396_isrForCrosstalkSources_855430_21.stderr parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash c5df8f3dbcecd723e532310fe59d9879 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6654e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/da9c17d9-d595-4f8d-bd37-391f76eb1ae1_isrForCrosstalkSources_855384_26.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/da9c17d9-d595-4f8d-bd37-391f76eb1ae1_isrForCrosstalkSources_855384_26.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 63dad412335e8dfe8efc6cde31caf518 parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/f220d316-5987-41fd-8746-67c301bf0760_isrForCrosstalkSources_855457_32.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/f220d316-5987-41fd-8746-67c301bf0760_isrForCrosstalkSources_855457_32.stderr parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash caaf44c6a10544b4217a980ad2957953 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a8dc8b6d-c42c-432a-b4c5-a08952597e0c_isrForCrosstalkSources_855431_23.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a8dc8b6d-c42c-432a-b4c5-a08952597e0c_isrForCrosstalkSources_855431_23.stderr parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash cfdebae938b1f3b57590797a0e3f220e parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c966fa5f-56f0-45a3-83d6-7c1f3d08ea40_isrForCrosstalkSources_855411_59.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c966fa5f-56f0-45a3-83d6-7c1f3d08ea40_isrForCrosstalkSources_855411_59.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash fca16453e32e4b12aad103bb784e4003 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fc7ea35c-cde6-4c16-aa9c-48352ec6369b_isrForCrosstalkSources_855401_3.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fc7ea35c-cde6-4c16-aa9c-48352ec6369b_isrForCrosstalkSources_855401_3.stderr parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 632dae238f47efeca70956d7a3fc74c6 parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/daf1bf58-422c-42ec-a4cf-2fe4ef2c801e_isrForCrosstalkSources_855421_23.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/daf1bf58-422c-42ec-a4cf-2fe4ef2c801e_isrForCrosstalkSources_855421_23.stderr parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 00deadec1a376f96d46821d54f46c336 parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ecc25b4-1bb4-4f4d-94f5-7bc1dd491d26_isrForCrosstalkSources_855437_22.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ecc25b4-1bb4-4f4d-94f5-7bc1dd491d26_isrForCrosstalkSources_855437_22.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash e33c3cde7ae2752383b207d548d4fa96 parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/d9090721-b401-4b4c-b721-d40189abbd20_isrForCrosstalkSources_855406_45.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/d9090721-b401-4b4c-b721-d40189abbd20_isrForCrosstalkSources_855406_45.stderr parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 6b57efd6c547e07200ba6424cee3fa7c parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d4962b96-ca04-4176-b73a-05c2a5fd8bdc_isrForCrosstalkSources_855423_24.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d4962b96-ca04-4176-b73a-05c2a5fd8bdc_isrForCrosstalkSources_855423_24.stderr parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash c29fca88d1290f0a52dcde151b53aba8 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4794b0c9-372f-49ed-870c-80a8cc27c852_isrForCrosstalkSources_855457_54.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4794b0c9-372f-49ed-870c-80a8cc27c852_isrForCrosstalkSources_855457_54.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash e89b90f4a23cc272b98441c380843b66 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/416a4004-1f68-47d0-bffa-4573a051d940_isrForCrosstalkSources_855438_21.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/416a4004-1f68-47d0-bffa-4573a051d940_isrForCrosstalkSources_855438_21.stderr parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash c1ba9b0472c514587ef1ab63a992e000 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/aa1554c4-1e58-48e7-b845-fe9e2ca05072_isrForCrosstalkSources_855431_10.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/aa1554c4-1e58-48e7-b845-fe9e2ca05072_isrForCrosstalkSources_855431_10.stderr parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 8a49e62be23ee6154d7035a0b406f2a4 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a66a15c-df8f-4ef3-987e-dba61a81a02b_isrForCrosstalkSources_855454_13.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a66a15c-df8f-4ef3-987e-dba61a81a02b_isrForCrosstalkSources_855454_13.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 5398d515ec40857dd70e305c5186fb20 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/45e6356e-001f-4ad6-9609-e4f702d84115_isrForCrosstalkSources_855411_12.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/45e6356e-001f-4ad6-9609-e4f702d84115_isrForCrosstalkSources_855411_12.stderr parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 131428d38a5ef82e866dd43988f10cce parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/439120d1-1180-461f-92bc-0f0f88a866d1_isrForCrosstalkSources_855421_7.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/439120d1-1180-461f-92bc-0f0f88a866d1_isrForCrosstalkSources_855421_7.stderr parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 6397e00c590c909938430d8b402ea6e6 parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964050 proc_lsst.multi INFO: [multi] job_id 16964050 2024-03-09 14:50:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 14:50:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964050 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964050 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16964050 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-09 14:50:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-09 14:50:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 14:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fc95917d-8a9a-417e-8f32-125706e88bb9_isrForCrosstalkSources_855458_41.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fc95917d-8a9a-417e-8f32-125706e88bb9_isrForCrosstalkSources_855458_41.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 223e6d9a961744462c399d95126b3c35 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1923de1e-ddcc-4cd5-9e22-164859d9313a_isrForCrosstalkSources_855431_40.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1923de1e-ddcc-4cd5-9e22-164859d9313a_isrForCrosstalkSources_855431_40.stderr parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 5db2aecbe688c9f1093beb693f0bde8d parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d782645f-0769-4552-8131-7e17c7a76d25_isrForCrosstalkSources_855386_21.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d782645f-0769-4552-8131-7e17c7a76d25_isrForCrosstalkSources_855386_21.stderr parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash d0373f1703fe458d4bec0a30b3445b91 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e64f316b-077e-421b-a90d-060a1ee486e5_isrForCrosstalkSources_855444_16.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e64f316b-077e-421b-a90d-060a1ee486e5_isrForCrosstalkSources_855444_16.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 5d4698bc7cd76b8949fe91922969b1ae parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d0df42b0-3d85-4691-813e-ec1df406e33d_isrForCrosstalkSources_855443_29.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d0df42b0-3d85-4691-813e-ec1df406e33d_isrForCrosstalkSources_855443_29.stderr parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash 9e69de09236c51d36f99fcdc9f49b219 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30c19cae-c536-416c-85a1-3dcb96f58cad_isrForCrosstalkSources_855436_34.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30c19cae-c536-416c-85a1-3dcb96f58cad_isrForCrosstalkSources_855436_34.stderr parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 375980ba3c8475c73056e4fb21a784a6 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/96e0eeba-27d6-4b5c-b8ba-8dac803ebd13_isrForCrosstalkSources_855449_47.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/96e0eeba-27d6-4b5c-b8ba-8dac803ebd13_isrForCrosstalkSources_855449_47.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash dc28b61344f892f5614064bdf4b4767c parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/826c5689-64fc-4e0f-99cb-78205e96a783_isrForCrosstalkSources_855456_10.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/826c5689-64fc-4e0f-99cb-78205e96a783_isrForCrosstalkSources_855456_10.stderr parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 4859063c785f47f09804a50d0295398a parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4f2402e0-b368-4c36-94e8-7b415c603659_isrForCrosstalkSources_855457_3.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4f2402e0-b368-4c36-94e8-7b415c603659_isrForCrosstalkSources_855457_3.stderr parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash e6e94a632edeee8f9d9a3222d619d444 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6012028d-48ad-46ed-b480-81716905e09b_isrForCrosstalkSources_855408_6.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6012028d-48ad-46ed-b480-81716905e09b_isrForCrosstalkSources_855408_6.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 61df5346a20a83bedaabc9f94047e07e parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a683caaf-ec1b-4fe5-a7e1-a55349a998d8_isrForCrosstalkSources_855438_50.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a683caaf-ec1b-4fe5-a7e1-a55349a998d8_isrForCrosstalkSources_855438_50.stderr parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 59c789fbb81d6b29a6ef78c2e6371394 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8654cdb2-a957-431e-a966-f9b4ada5ff20_isrForCrosstalkSources_855425_54.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8654cdb2-a957-431e-a966-f9b4ada5ff20_isrForCrosstalkSources_855425_54.stderr parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 15dbb5b7899383520f4b437cdd269cf1 parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/725c5331-06b7-4eb1-a930-b2a6eabd228f_isrForCrosstalkSources_855386_15.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/725c5331-06b7-4eb1-a930-b2a6eabd228f_isrForCrosstalkSources_855386_15.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 91ad3f8d398ec8222bcf795131aa915e parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5dfb448d-7469-4280-b8e9-20d1248e8421_isrForCrosstalkSources_855398_61.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5dfb448d-7469-4280-b8e9-20d1248e8421_isrForCrosstalkSources_855398_61.stderr parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 9e40ddea7616439b9ea9f41442128b02 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/92cb41f8-0fe1-4dcf-b106-de1a7eb2299d_isrForCrosstalkSources_855455_25.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/92cb41f8-0fe1-4dcf-b106-de1a7eb2299d_isrForCrosstalkSources_855455_25.stderr parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 5f5cb186afdd7ffedcb4b3a72798e455 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8c4dbe61-d857-4484-9565-5ba30eab7400_isrForCrosstalkSources_855384_17.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8c4dbe61-d857-4484-9565-5ba30eab7400_isrForCrosstalkSources_855384_17.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 0ac42ee07de37ae6b922fd45699ca35a parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68eaf8fb-a4c6-42be-aec0-2cdefd608241_isrForCrosstalkSources_855455_54.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68eaf8fb-a4c6-42be-aec0-2cdefd608241_isrForCrosstalkSources_855455_54.stderr parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 093cc13a473245f9490bb7f65c21b983 parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/d5272255-b495-42ad-aace-0aea80fb82b9_isrForCrosstalkSources_855437_50.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/d5272255-b495-42ad-aace-0aea80fb82b9_isrForCrosstalkSources_855437_50.stderr parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash b30d90b4ac90b53f62dd332c8e0b046b parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/419017cb-a421-44c5-9ae0-8bd48d37f221_isrForCrosstalkSources_855455_13.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/419017cb-a421-44c5-9ae0-8bd48d37f221_isrForCrosstalkSources_855455_13.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 74ac0f6233db10adf4a39050cbbf1e4f parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a196b5d0-7d31-484e-8101-e35719fa55c6_isrForCrosstalkSources_855388_62.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a196b5d0-7d31-484e-8101-e35719fa55c6_isrForCrosstalkSources_855388_62.stderr parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 4c6298c1ef807a420d4be96ba712bf81 parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d485c358-f358-48ff-b62d-2a8525495034_isrForCrosstalkSources_855426_46.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d485c358-f358-48ff-b62d-2a8525495034_isrForCrosstalkSources_855426_46.stderr parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 37dd0a296b7c9b8e9e5d14251c66c26b parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d782ef03-c72f-40cc-9b9e-daa9265511ce_isrForCrosstalkSources_855449_33.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d782ef03-c72f-40cc-9b9e-daa9265511ce_isrForCrosstalkSources_855449_33.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 2f8c415bd3974942226446bb95904e37 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/244aef86-a315-425f-9910-c7c7a9768563_isrForCrosstalkSources_855454_34.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/244aef86-a315-425f-9910-c7c7a9768563_isrForCrosstalkSources_855454_34.stderr parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 78016b4195e1fa5e3cb8f671fae08fb7 parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f66c4a18-9360-491a-a18c-76fe6985cf58_isrForCrosstalkSources_855384_48.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f66c4a18-9360-491a-a18c-76fe6985cf58_isrForCrosstalkSources_855384_48.stderr parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 5f62af15161db67124efcbf1440d9c36 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a99c0eec-ff8a-4151-abad-1d9ee1f90db6_isrForCrosstalkSources_855439_33.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a99c0eec-ff8a-4151-abad-1d9ee1f90db6_isrForCrosstalkSources_855439_33.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 17d3bd7cc36b31435c893bef02e1a839 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964051 proc_lsst.multi INFO: [multi] job_id 16964051 2024-03-09 14:50:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 14:50:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964051 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964051 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16964051 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-09 14:50:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-09 14:50:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 14:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/1e584d68-3d4a-4b60-b6ac-68ff856c3d62_isrForCrosstalkSources_855448_17.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/1e584d68-3d4a-4b60-b6ac-68ff856c3d62_isrForCrosstalkSources_855448_17.stderr parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 18b06279aaaa17fedf8fd2e0448cf41b parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/dec4f71c-6b8c-4127-a647-7a7384510efc_isrForCrosstalkSources_855452_60.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/dec4f71c-6b8c-4127-a647-7a7384510efc_isrForCrosstalkSources_855452_60.stderr parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 979e2cf3a8e2e1c9395e158c620b2abb parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/889c171b-b589-4e55-9421-d44ac91c25a6_isrForCrosstalkSources_855395_5.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/889c171b-b589-4e55-9421-d44ac91c25a6_isrForCrosstalkSources_855395_5.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash f456015a40cc6aca8e4fda0d536d0cfa parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8b878db-defd-4f5d-aafe-3640fd649d4e_isrForCrosstalkSources_855449_58.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8b878db-defd-4f5d-aafe-3640fd649d4e_isrForCrosstalkSources_855449_58.stderr parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 0f191b52ea5d07c18b473e9c33a9d1e8 parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/41ad133e-c83b-4d5d-af88-02a089c1ad7b_isrForCrosstalkSources_855437_36.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/41ad133e-c83b-4d5d-af88-02a089c1ad7b_isrForCrosstalkSources_855437_36.stderr parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 13d08055da46e7d17d25388ba90c7940 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/78d9b8aa-8f55-4ae0-b31c-ffe39cbcc159_isrForCrosstalkSources_855452_41.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/78d9b8aa-8f55-4ae0-b31c-ffe39cbcc159_isrForCrosstalkSources_855452_41.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash e663776a5f332a5cad8ac5ae9c7de93f parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ab5c8e-c125-437f-8932-7e7b6ab87c68_isrForCrosstalkSources_855437_6.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ab5c8e-c125-437f-8932-7e7b6ab87c68_isrForCrosstalkSources_855437_6.stderr parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 358adf50f34560504eef102eab6c14f8 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fe259fb4-16ad-4963-9da4-95ebc7decd26_isrForCrosstalkSources_855454_62.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fe259fb4-16ad-4963-9da4-95ebc7decd26_isrForCrosstalkSources_855454_62.stderr parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash e598218b9033ffa0cd374bc84611a0a6 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/931a46cc-df00-4b2e-9ac7-05ac181d3eeb_isrForCrosstalkSources_855424_42.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/931a46cc-df00-4b2e-9ac7-05ac181d3eeb_isrForCrosstalkSources_855424_42.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash b7f37381f4111394b0156223be5c8137 parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e40042b2-aa82-47f7-af0a-0a375f9a589b_isrForCrosstalkSources_855431_16.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e40042b2-aa82-47f7-af0a-0a375f9a589b_isrForCrosstalkSources_855431_16.stderr parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 14119ac812742d42cbaad03a8a15b22c parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/3ac2d5e2-90dd-4c31-99f3-99de29ac36f7_isrForCrosstalkSources_855393_49.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/3ac2d5e2-90dd-4c31-99f3-99de29ac36f7_isrForCrosstalkSources_855393_49.stderr parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 74cef28cfe556967e45d01c307024e2a parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/20720e6d-15fc-4413-9ccd-9d73dcde77ee_isrForCrosstalkSources_855447_42.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/20720e6d-15fc-4413-9ccd-9d73dcde77ee_isrForCrosstalkSources_855447_42.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 6261e80d357160d4214f2541fd574c6d parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c26e2a8c-a401-495f-8150-b6b54eabcd6b_isrForCrosstalkSources_855422_59.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c26e2a8c-a401-495f-8150-b6b54eabcd6b_isrForCrosstalkSources_855422_59.stderr parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 0b53e5ff6010bc7a2a74aa2c800810cf parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1ce523cb-af51-475e-bae7-19fb735ddd9b_isrForCrosstalkSources_855431_34.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1ce523cb-af51-475e-bae7-19fb735ddd9b_isrForCrosstalkSources_855431_34.stderr parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 4ed9d1d6940e81426de02ef3c7af6089 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6679c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1d6b84fa-65c6-455d-a79d-6b860bb16daa_isrForCrosstalkSources_855419_47.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1d6b84fa-65c6-455d-a79d-6b860bb16daa_isrForCrosstalkSources_855419_47.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 9dcb46d80d5dfef25da929f391eb04e2 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4beca878-5e8a-4800-b49e-78f85050588b_isrForCrosstalkSources_855382_26.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4beca878-5e8a-4800-b49e-78f85050588b_isrForCrosstalkSources_855382_26.stderr parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash ac631e10e650e44791ecf1491b74b0ea parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/768a61ee-32fa-4b2b-8d0c-2042c2313f2a_isrForCrosstalkSources_855416_20.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/768a61ee-32fa-4b2b-8d0c-2042c2313f2a_isrForCrosstalkSources_855416_20.stderr parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash a2c56fff2e8131ec74d21b5e2efb379b parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ee35429a-31e9-4eaa-aac9-cd67ae452315_isrForCrosstalkSources_855454_50.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ee35429a-31e9-4eaa-aac9-cd67ae452315_isrForCrosstalkSources_855454_50.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 820e3697fc045c929cbd7843c5e128b3 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5a43cd8d-e5c5-45bc-a3e0-e34d9c706aa9_isrForCrosstalkSources_855398_49.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5a43cd8d-e5c5-45bc-a3e0-e34d9c706aa9_isrForCrosstalkSources_855398_49.stderr parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 289b743d8bacfaa5639ec60955394830 parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a2d4c2c7-232d-43be-b975-5d29aaaafa88_isrForCrosstalkSources_855390_53.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a2d4c2c7-232d-43be-b975-5d29aaaafa88_isrForCrosstalkSources_855390_53.stderr parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash a1f34552c6bab50e5f2e7b82b9713899 parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3731a6c2-5435-4413-80a8-a5d49a58f801_isrForCrosstalkSources_855443_15.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3731a6c2-5435-4413-80a8-a5d49a58f801_isrForCrosstalkSources_855443_15.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 1f1c81fe7b641a121524170832e002c7 parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6660c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ad28aabc-5aeb-436b-8614-1144cc697d29_isrForCrosstalkSources_855432_27.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ad28aabc-5aeb-436b-8614-1144cc697d29_isrForCrosstalkSources_855432_27.stderr parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash c6b1523cd1cb63f58daca3a4608f9626 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6676a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/580efdd1-ed1a-4fa5-8d2c-23f6b3a61b71_isrForCrosstalkSources_855424_19.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/580efdd1-ed1a-4fa5-8d2c-23f6b3a61b71_isrForCrosstalkSources_855424_19.stderr parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash fdc25665f14f59928e35697f2e3218ba parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa667740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e4241ec4-7596-4d3a-85e9-5e414df92eba_isrForCrosstalkSources_855434_20.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e4241ec4-7596-4d3a-85e9-5e414df92eba_isrForCrosstalkSources_855434_20.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 261299525bdb3afef2606d14da6fde0d parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6662a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/94d05175-94cf-4396-9da7-4ac451507bf5_isrForCrosstalkSources_855459_7.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/94d05175-94cf-4396-9da7-4ac451507bf5_isrForCrosstalkSources_855459_7.stderr parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 18e413882facab642847e83a3ef90313 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6663e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/825328c4-c3bc-4522-b511-5042d9f7efa8_isrForCrosstalkSources_855380_26.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/825328c4-c3bc-4522-b511-5042d9f7efa8_isrForCrosstalkSources_855380_26.stderr parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 3a91b5afb8e7ba6760bc9009364ca2f6 parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/9724a414-697c-4ed0-949a-d20d730a2238_isrForCrosstalkSources_855413_59.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/9724a414-697c-4ed0-949a-d20d730a2238_isrForCrosstalkSources_855413_59.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 5fd5c0ca22d011658ab8ed53527e7a50 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/3da81791-fe1d-491d-bace-60599496cfeb_isrForCrosstalkSources_855431_7.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/3da81791-fe1d-491d-bace-60599496cfeb_isrForCrosstalkSources_855431_7.stderr parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash b0a4e79e7db0da5371c96555103d406e parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5d134dcd-a396-4ec0-b023-32bdb11ca2cd_isrForCrosstalkSources_855401_54.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5d134dcd-a396-4ec0-b023-32bdb11ca2cd_isrForCrosstalkSources_855401_54.stderr parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash ddc279ce8315853bc94c77c69980004e parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa664ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/87483b3b-3c29-499b-ab0d-772bf0654bea_isrForCrosstalkSources_855386_41.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/87483b3b-3c29-499b-ab0d-772bf0654bea_isrForCrosstalkSources_855386_41.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash c9405f3e20028dc9b96e881bb30c3ad5 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa666520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964052 proc_lsst.multi INFO: [multi] job_id 16964052 2024-03-09 14:50:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 14:50:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964052 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964052 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16964052 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22375328864144 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 28, active_tasks = 998 parsl.dataflow.strategy DEBUG: Executor multi has 998 active tasks, 1/27 running/pending blocks, and 19 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 28 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 222 more blocks parsl.executors.status_handling INFO: Scaling out by 222 blocks parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-09 14:50:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-09 14:50:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 14:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8ec05c50-fadc-475b-a082-63d4b962ddf9_isrForCrosstalkSources_855423_14.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8ec05c50-fadc-475b-a082-63d4b962ddf9_isrForCrosstalkSources_855423_14.stderr parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 7e1324b9657efad86709358dedd50008 parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa665f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1c9beefd-e628-49bc-ab8d-056a85a5cb0f_isrForCrosstalkSources_855439_23.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1c9beefd-e628-49bc-ab8d-056a85a5cb0f_isrForCrosstalkSources_855439_23.stderr parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 83fd81c56123c9c0ace25bfc449470cc parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ef626d3c-0dd4-440c-ae27-d562b1bbf85f_isrForCrosstalkSources_855439_32.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ef626d3c-0dd4-440c-ae27-d562b1bbf85f_isrForCrosstalkSources_855439_32.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash f3adeb7db68ff3d05ebf35a1e1eceebc parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6640e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9728a97f-ddc2-40ea-8a4f-97e2318f118e_isrForCrosstalkSources_855431_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9728a97f-ddc2-40ea-8a4f-97e2318f118e_isrForCrosstalkSources_855431_62.stderr parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 208a3b063aab8700d6e8a0dc439b8eed parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bdea1492-f43e-4567-9edc-467aa3cdeda8_isrForCrosstalkSources_855381_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bdea1492-f43e-4567-9edc-467aa3cdeda8_isrForCrosstalkSources_855381_36.stderr parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 1eb01c37272ad9274caa4fe56d75b7c3 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/dedbc0af-3918-4f73-a8e6-d2f921691d43_isrForCrosstalkSources_855412_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/dedbc0af-3918-4f73-a8e6-d2f921691d43_isrForCrosstalkSources_855412_62.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 62483969f63e2bf7d1bd54eb3ecb661e parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/22e489cb-55fa-41e2-a6f1-53e141ed89fd_isrForCrosstalkSources_855448_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/22e489cb-55fa-41e2-a6f1-53e141ed89fd_isrForCrosstalkSources_855448_28.stderr parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 9f536c47f580ac935872157c39ada5ad parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f3d8f57b-858e-40a2-adff-b6516a6613c9_isrForCrosstalkSources_855453_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f3d8f57b-858e-40a2-adff-b6516a6613c9_isrForCrosstalkSources_855453_40.stderr parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 2c5c4db5660d3adb3753811565053324 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a14c869a-676a-4ecb-b5f5-97ba36502296_isrForCrosstalkSources_855412_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a14c869a-676a-4ecb-b5f5-97ba36502296_isrForCrosstalkSources_855412_55.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 006182ed6abee01d7da2dbaf78e0d041 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/246d087b-6a35-4eb9-ade6-a6dd5750a346_isrForCrosstalkSources_855394_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/246d087b-6a35-4eb9-ade6-a6dd5750a346_isrForCrosstalkSources_855394_44.stderr parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash b70d0bf2df625b68e9d2e26b236d7e0e parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/69f6faee-98c6-4493-ad56-73d50022f468_isrForCrosstalkSources_855438_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/69f6faee-98c6-4493-ad56-73d50022f468_isrForCrosstalkSources_855438_38.stderr parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 8d45cab23771a9edfe483208ac714746 parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7463069e-43d1-45d4-a6e7-32eb92f46a61_isrForCrosstalkSources_855429_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7463069e-43d1-45d4-a6e7-32eb92f46a61_isrForCrosstalkSources_855429_16.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 1ed0f8515559cafafc469259c2d94d34 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/54140e4a-50bb-48a3-b2af-269029d5d3fc_isrForCrosstalkSources_855397_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/54140e4a-50bb-48a3-b2af-269029d5d3fc_isrForCrosstalkSources_855397_36.stderr parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 5b9c64b8c0a3395466ab65f283210e56 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0afe2387-7330-4e7b-bdad-4c6f900d2555_isrForCrosstalkSources_855382_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0afe2387-7330-4e7b-bdad-4c6f900d2555_isrForCrosstalkSources_855382_52.stderr parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 9b0edad1a22e23e3d77c851b28838186 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/4073da7f-6f45-4068-96cc-33edfc9595e3_isrForCrosstalkSources_855423_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/4073da7f-6f45-4068-96cc-33edfc9595e3_isrForCrosstalkSources_855423_15.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 5d223b82d62bb92fcf663d1dc35da037 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa185e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/84266b3b-b3d0-44af-9fc7-94a4b371f39a_isrForCrosstalkSources_855403_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/84266b3b-b3d0-44af-9fc7-94a4b371f39a_isrForCrosstalkSources_855403_12.stderr parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 06a3fbc1c8346903a5d4013b5ca59309 parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa184a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/64b5ec51-6da0-4263-940b-865fab090604_isrForCrosstalkSources_855430_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/64b5ec51-6da0-4263-940b-865fab090604_isrForCrosstalkSources_855430_45.stderr parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash ee0a77bd9f43c5d64b72a99b206f0fb7 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/986369e1-0403-4bf2-b3a9-82d3b5c23986_isrForCrosstalkSources_855391_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/986369e1-0403-4bf2-b3a9-82d3b5c23986_isrForCrosstalkSources_855391_31.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 085ad517a882e9ddbcccaa6b4c40c8ae parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b0ca8d8f-2cb9-43e5-9044-aa8d5d7a79eb_isrForCrosstalkSources_855397_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b0ca8d8f-2cb9-43e5-9044-aa8d5d7a79eb_isrForCrosstalkSources_855397_33.stderr parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 713e1c9215c272781eb100c686f69c02 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa187c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/9d909381-dd66-4523-b306-7590cfab7025_isrForCrosstalkSources_855459_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/9d909381-dd66-4523-b306-7590cfab7025_isrForCrosstalkSources_855459_39.stderr parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash a505e3c5c20280465076be8f3505cbdd parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/3c347239-bea0-47e8-9509-aa9e0b43da1f_isrForCrosstalkSources_855435_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/3c347239-bea0-47e8-9509-aa9e0b43da1f_isrForCrosstalkSources_855435_2.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash b0e1af3babc5f842085440e15395417f parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/75b7fbf7-c25f-409f-8d19-b6020a6f9460_isrForCrosstalkSources_855456_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/75b7fbf7-c25f-409f-8d19-b6020a6f9460_isrForCrosstalkSources_855456_1.stderr parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash e444aaae71fa6afc0a29bdb265906cad parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa189a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/07436a77-6852-43ec-bc79-32ea8d34fa1a_isrForCrosstalkSources_855420_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/07436a77-6852-43ec-bc79-32ea8d34fa1a_isrForCrosstalkSources_855420_3.stderr parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 6d3bf3e932060de2c6d9aea034eb97b1 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a8a3181-9a55-4e88-9639-d15d33bca52d_isrForCrosstalkSources_855454_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a8a3181-9a55-4e88-9639-d15d33bca52d_isrForCrosstalkSources_855454_9.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash d2dd16a080e08b3a9cf7615886968bd5 parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e92ec7d3-4a70-43f7-b930-d5e54ca3f9a0_isrForCrosstalkSources_855401_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e92ec7d3-4a70-43f7-b930-d5e54ca3f9a0_isrForCrosstalkSources_855401_19.stderr parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 9ad4c8d58bbc7678ccb9334faffb473e parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/be68e3a4-28ad-4eb3-b850-d91e98f223dc_isrForCrosstalkSources_855386_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/be68e3a4-28ad-4eb3-b850-d91e98f223dc_isrForCrosstalkSources_855386_39.stderr parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 9534301c2268e1e9ee9d41458e91d42f parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/400914f0-0555-499a-b6dd-49acda4cccd6_isrForCrosstalkSources_855452_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/400914f0-0555-499a-b6dd-49acda4cccd6_isrForCrosstalkSources_855452_30.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 4ef8eda19d6623cccbd2c9b3c53e4c53 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9aefc00f-938a-4689-a081-7a441e06654d_isrForCrosstalkSources_855398_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9aefc00f-938a-4689-a081-7a441e06654d_isrForCrosstalkSources_855398_7.stderr parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash f0a413487dace40068931c8a0e7a8596 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0cab1a3-1b54-4ec2-ad12-db799382a378_isrForCrosstalkSources_855384_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0cab1a3-1b54-4ec2-ad12-db799382a378_isrForCrosstalkSources_855384_49.stderr parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 21462bf78e9e59d035138eb1533f73d7 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e07aae58-85b2-4b4b-afbe-60d9e5f73f34_isrForCrosstalkSources_855386_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e07aae58-85b2-4b4b-afbe-60d9e5f73f34_isrForCrosstalkSources_855386_54.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash 1e1febcbf01ee5a7cc97db09b647a3f2 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b4c5a669-e2c3-4cbd-98bd-9b467bc633bb_isrForCrosstalkSources_855384_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b4c5a669-e2c3-4cbd-98bd-9b467bc633bb_isrForCrosstalkSources_855384_45.stderr parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 1711ad15ead78a0a222e21974d12c57b parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/985a9c48-5cb5-4e4c-ae4a-b8f6536b2755_isrForCrosstalkSources_855457_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/985a9c48-5cb5-4e4c-ae4a-b8f6536b2755_isrForCrosstalkSources_855457_15.stderr parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 6bce0dc07fee54f2990dd9ede525a283 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5b335c02-955e-46d7-9364-a7023074e310_isrForCrosstalkSources_855383_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5b335c02-955e-46d7-9364-a7023074e310_isrForCrosstalkSources_855383_35.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash f9b4dce7349c67a57ed38538c8b8431c parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c9816fe7-c0a5-43f9-8ec8-7b3b278d79f4_isrForCrosstalkSources_855414_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c9816fe7-c0a5-43f9-8ec8-7b3b278d79f4_isrForCrosstalkSources_855414_31.stderr parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 38df398c15bf11259777beef29c66fd4 parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/5f467234-694d-4e9e-b8fa-c3816862a4ee_isrForCrosstalkSources_855448_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/5f467234-694d-4e9e-b8fa-c3816862a4ee_isrForCrosstalkSources_855448_8.stderr parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 0bff5ab1dde24f6c93d2fe6ed82cc662 parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d9ec2eac-bbaf-440d-bcc5-69854579f95b_isrForCrosstalkSources_855417_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d9ec2eac-bbaf-440d-bcc5-69854579f95b_isrForCrosstalkSources_855417_13.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash e9ecd3f34fc6eb10fae9c7a4556bfea6 parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/80270615-cc1a-45e1-83ec-b87c3bef6ae1_isrForCrosstalkSources_855402_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/80270615-cc1a-45e1-83ec-b87c3bef6ae1_isrForCrosstalkSources_855402_25.stderr parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 4b5db8523290458d0174081b30ff36a6 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9fe28-a082-4f25-b7b8-bb58712aa0a8_isrForCrosstalkSources_855429_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9fe28-a082-4f25-b7b8-bb58712aa0a8_isrForCrosstalkSources_855429_25.stderr parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash b234645b9ab2c085420979166fd2c2c2 parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7dc87fc2-503f-4879-a323-4cff797fd630_isrForCrosstalkSources_855455_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7dc87fc2-503f-4879-a323-4cff797fd630_isrForCrosstalkSources_855455_52.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 4e5f04dd92b6cd546d337cc69bc27de6 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/1c0bb058-7206-49b9-a342-ca59ed41cc4a_isrForCrosstalkSources_855458_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/1c0bb058-7206-49b9-a342-ca59ed41cc4a_isrForCrosstalkSources_855458_15.stderr parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 78156545d27f16e70c5faec17459cf8a parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10552c0e-c99e-41e9-b92d-c3ec8dd00adb_isrForCrosstalkSources_855412_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10552c0e-c99e-41e9-b92d-c3ec8dd00adb_isrForCrosstalkSources_855412_26.stderr parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 7284506ffd7809b8fc8745fd6989682b parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4a513950-6a7e-498d-8ea7-395b639b4db3_isrForCrosstalkSources_855409_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4a513950-6a7e-498d-8ea7-395b639b4db3_isrForCrosstalkSources_855409_24.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 03b837fb516532b379608b09944ec222 parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/78c1ac3b-60b7-48e1-ac58-59177eca5308_isrForCrosstalkSources_855422_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/78c1ac3b-60b7-48e1-ac58-59177eca5308_isrForCrosstalkSources_855422_42.stderr parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash e630123b05c2305d57dc88330cb43829 parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5154d4ec-3f27-4e01-aa8a-af0be50caa1c_isrForCrosstalkSources_855384_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5154d4ec-3f27-4e01-aa8a-af0be50caa1c_isrForCrosstalkSources_855384_11.stderr parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash ce2e297fcdd899197e520a94e7753a75 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964053 proc_lsst.multi INFO: [multi] job_id 16964053 2024-03-09 14:50:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 14:50:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964053 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964053 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16964053 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-09 14:50:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-09 14:50:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 14:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/eccb9ce4-0cc8-4767-abbb-e34770f95ec5_isrForCrosstalkSources_855402_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/eccb9ce4-0cc8-4767-abbb-e34770f95ec5_isrForCrosstalkSources_855402_34.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 4cd8ade43539aa8433c34610045a897d parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa199e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/1796020a-831c-47a9-9c47-de1dfee55cee_isrForCrosstalkSources_855432_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/1796020a-831c-47a9-9c47-de1dfee55cee_isrForCrosstalkSources_855432_18.stderr parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash a4821f93ea90d6ee4cedd3bc74e1026c parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/b47de29f-2e28-47eb-a96f-50e3c0555ab8_isrForCrosstalkSources_855386_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/b47de29f-2e28-47eb-a96f-50e3c0555ab8_isrForCrosstalkSources_855386_25.stderr parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash dada14a5a51a1dba1ac9747d36fa0b50 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b521c2cd-a5a3-4d57-bd29-057cfce2af62_isrForCrosstalkSources_855394_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b521c2cd-a5a3-4d57-bd29-057cfce2af62_isrForCrosstalkSources_855394_62.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash bc22f951b6970984a454c3aeeb77cede parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa182c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/23729405-38e8-46ea-b874-39f22bd962f7_isrForCrosstalkSources_855415_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/23729405-38e8-46ea-b874-39f22bd962f7_isrForCrosstalkSources_855415_2.stderr parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 98fea163669ba737bbb74a44abe845b6 parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5c3f2e66-4595-44bf-9e28-0cf3dd7e7445_isrForCrosstalkSources_855440_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5c3f2e66-4595-44bf-9e28-0cf3dd7e7445_isrForCrosstalkSources_855440_42.stderr parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash cced6e0071fd0a51f6c6a093c4dafad8 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa198a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f38dc340-3ae4-4e3f-9096-a409ab34a825_isrForCrosstalkSources_855387_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f38dc340-3ae4-4e3f-9096-a409ab34a825_isrForCrosstalkSources_855387_29.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash c187dd2dfb7cfea5d8c50910e17ddb2f parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa180e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e8b4269f-70df-4894-95c9-52f4239bd589_isrForCrosstalkSources_855444_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e8b4269f-70df-4894-95c9-52f4239bd589_isrForCrosstalkSources_855444_55.stderr parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 085537abfbba19366ec0a419a1a57985 parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa193a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f3b48296-226f-41cc-8f34-06b7113c0bdd_isrForCrosstalkSources_855452_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f3b48296-226f-41cc-8f34-06b7113c0bdd_isrForCrosstalkSources_855452_39.stderr parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash fca05c50414044e69591aac256d35506 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa191c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f737ce33-fff4-425e-ad91-94f7c080874e_isrForCrosstalkSources_855453_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f737ce33-fff4-425e-ad91-94f7c080874e_isrForCrosstalkSources_855453_51.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash df65ca9769c5c17e1fd4239952bbe9c2 parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d108866b-88be-4877-b1f2-7897ef955691_isrForCrosstalkSources_855429_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d108866b-88be-4877-b1f2-7897ef955691_isrForCrosstalkSources_855429_52.stderr parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 9b8d288f39b7e33995aff0bedf954c03 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7445c029-87a4-4c1b-b18f-7d014e6db990_isrForCrosstalkSources_855392_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7445c029-87a4-4c1b-b18f-7d014e6db990_isrForCrosstalkSources_855392_14.stderr parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 680b58fcac61110b65cda2575d5a7751 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f5bcb64-f4b7-40ed-b248-e79645de131a_isrForCrosstalkSources_855456_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f5bcb64-f4b7-40ed-b248-e79645de131a_isrForCrosstalkSources_855456_61.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 9ca255f4e9b9e7bd77f9d4c6a450ecbf parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f00d04f-ffcf-440e-bd19-a7eca80dfa09_isrForCrosstalkSources_855393_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f00d04f-ffcf-440e-bd19-a7eca80dfa09_isrForCrosstalkSources_855393_11.stderr parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 90b6f1adf97e330d490dd5b7b3ebc539 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e2773-04a2-49fb-ae30-a8ad4aa74a96_isrForCrosstalkSources_855448_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e2773-04a2-49fb-ae30-a8ad4aa74a96_isrForCrosstalkSources_855448_34.stderr parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 8e96cbdbb75fe80b05b624e7805eb59a parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/23afdc9a-b056-4326-8da5-9deec398016d_isrForCrosstalkSources_855439_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/23afdc9a-b056-4326-8da5-9deec398016d_isrForCrosstalkSources_855439_59.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash bcda995593e9f2b919cdb46748e91645 parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/627b19c7-727a-4d5f-aa1b-211ef026afa7_isrForCrosstalkSources_855428_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/627b19c7-727a-4d5f-aa1b-211ef026afa7_isrForCrosstalkSources_855428_24.stderr parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 2eed7b4ccf4386b9911ad242084a8fa0 parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9b2007a0-8ff6-4528-9585-115b8534f156_isrForCrosstalkSources_855408_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9b2007a0-8ff6-4528-9585-115b8534f156_isrForCrosstalkSources_855408_33.stderr parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 0a1ea5b5267b6505c60a905763099713 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d98b1bcc-5cf4-421b-945b-773d2a1d09b0_isrForCrosstalkSources_855444_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d98b1bcc-5cf4-421b-945b-773d2a1d09b0_isrForCrosstalkSources_855444_19.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 94ecbdd9319eddd44c48868ae5e7f115 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3696d5a4-b5fa-456a-a527-8335c5e36f6e_isrForCrosstalkSources_855382_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3696d5a4-b5fa-456a-a527-8335c5e36f6e_isrForCrosstalkSources_855382_5.stderr parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 4f08242278aa4c4e1dd7887dc80dc0b4 parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6f5c9e97-5f3f-45b0-9521-1bd51892ed31_isrForCrosstalkSources_855385_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6f5c9e97-5f3f-45b0-9521-1bd51892ed31_isrForCrosstalkSources_855385_39.stderr parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 8f386f2fa2d50273adef8ef09e2ee7d2 parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/538f88e9-83df-4374-ba70-0f4d62de7f19_isrForCrosstalkSources_855455_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/538f88e9-83df-4374-ba70-0f4d62de7f19_isrForCrosstalkSources_855455_47.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 13d41b70ddfff4680d5d037febe3ada9 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:09 proc_lsst.multi:170 [INFO] [multi] job_id 16964054 proc_lsst.multi INFO: [multi] job_id 16964054 2024-03-09 14:50:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 14:50:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964054 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964054 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16964054 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-09 14:50:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-09 14:50:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 14:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fe56cd1f-9530-45a1-ac8a-a576e7576d88_isrForCrosstalkSources_855415_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fe56cd1f-9530-45a1-ac8a-a576e7576d88_isrForCrosstalkSources_855415_14.stderr parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 5410676942865e68c85a2eccc6e9000c parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a209234d-0e16-4976-9aea-9c0e04c44f16_isrForCrosstalkSources_855455_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a209234d-0e16-4976-9aea-9c0e04c44f16_isrForCrosstalkSources_855455_48.stderr parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 2f722382aca022b4e5d2a30d4eb55a1c parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/29ca02bf-d462-4c29-8de2-ff8b1759b99e_isrForCrosstalkSources_855454_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/29ca02bf-d462-4c29-8de2-ff8b1759b99e_isrForCrosstalkSources_855454_40.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 9abf96fdf329e316ea6d35dc66119517 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/23a3d736-2fe4-40ef-a0d3-8384d2461371_isrForCrosstalkSources_855411_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/23a3d736-2fe4-40ef-a0d3-8384d2461371_isrForCrosstalkSources_855411_23.stderr parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 36a8f4f6c4a9afd9dfb36105b753c267 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/04cb54b0-7d9b-4dcd-84d1-5820eb3d2c8a_isrForCrosstalkSources_855393_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/04cb54b0-7d9b-4dcd-84d1-5820eb3d2c8a_isrForCrosstalkSources_855393_28.stderr parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 30de47ad580322d6ea96d9a7615f8198 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8b7a5750-72b2-4ae6-912e-824ec01a43af_isrForCrosstalkSources_855427_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8b7a5750-72b2-4ae6-912e-824ec01a43af_isrForCrosstalkSources_855427_1.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash dc90de48b5e04d38b3444b5e34c25912 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/e3837c24-ba35-46ab-879f-6c7b77616571_isrForCrosstalkSources_855393_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/e3837c24-ba35-46ab-879f-6c7b77616571_isrForCrosstalkSources_855393_12.stderr parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 661ab47d693973672978d8e1ced2790c parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6bb6851b-3b77-46a7-a31e-0e45de69e5b1_isrForCrosstalkSources_855392_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6bb6851b-3b77-46a7-a31e-0e45de69e5b1_isrForCrosstalkSources_855392_24.stderr parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash decdf5b21b9f51f66e884be92651c6b6 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ba848d1b-a638-4b2e-a409-d7fc68914177_isrForCrosstalkSources_855415_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ba848d1b-a638-4b2e-a409-d7fc68914177_isrForCrosstalkSources_855415_34.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 3b03c803a73a63a15adde7a2ac5dd9d8 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/5aa94319-f443-40f8-9e41-7b88ce631348_isrForCrosstalkSources_855431_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/5aa94319-f443-40f8-9e41-7b88ce631348_isrForCrosstalkSources_855431_52.stderr parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 59c5aacabfa2f9f812b41c738bc413d7 parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ef69e9c3-9e9c-49c2-9ba7-d4565513ccda_isrForCrosstalkSources_855441_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ef69e9c3-9e9c-49c2-9ba7-d4565513ccda_isrForCrosstalkSources_855441_27.stderr parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 39555b7c204ed8d96e312b981fbaafd3 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c1c08088-92bf-46c5-b0d3-cff1528364cf_isrForCrosstalkSources_855459_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c1c08088-92bf-46c5-b0d3-cff1528364cf_isrForCrosstalkSources_855459_57.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash 750e2cf7eef826c46facb659a6964610 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/49735925-507c-46a6-a6c8-17451bc265f1_isrForCrosstalkSources_855389_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/49735925-507c-46a6-a6c8-17451bc265f1_isrForCrosstalkSources_855389_30.stderr parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 90f6fda9b10296aa2d6c67cf16f6c5f6 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ee1f1e81-6565-4617-acd9-f0f8a5276a41_isrForCrosstalkSources_855386_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ee1f1e81-6565-4617-acd9-f0f8a5276a41_isrForCrosstalkSources_855386_2.stderr parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 77f3c25e470681bf7620ee71f0f3ffce parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a91459ec-226c-43bf-8d58-7491252dfd14_isrForCrosstalkSources_855401_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a91459ec-226c-43bf-8d58-7491252dfd14_isrForCrosstalkSources_855401_4.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash 49be6d4ba7cd48ba20a5c8846e1d668d parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b50d0cea-61dc-4300-90aa-476bdf48030b_isrForCrosstalkSources_855456_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b50d0cea-61dc-4300-90aa-476bdf48030b_isrForCrosstalkSources_855456_6.stderr parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 5869d4bab664cd9370ace862d6a04595 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a5b750eb-c06f-46a4-b45e-48777544e5f4_isrForCrosstalkSources_855445_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a5b750eb-c06f-46a4-b45e-48777544e5f4_isrForCrosstalkSources_855445_40.stderr parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 492aa3a5cef40485ed7753dced20277a parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c6989876-67e9-4ae8-a6c4-bdc714d4c3d1_isrForCrosstalkSources_855394_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c6989876-67e9-4ae8-a6c4-bdc714d4c3d1_isrForCrosstalkSources_855394_14.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash 449ffc2c91902600bcf0958ba21378b5 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f828aa24-09ed-4c7c-b071-ab4195ffb726_isrForCrosstalkSources_855405_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f828aa24-09ed-4c7c-b071-ab4195ffb726_isrForCrosstalkSources_855405_44.stderr parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 0d4f41450bf137ac8b4e7bafc8bac08c parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ac104252-390f-489c-b9af-2fbd99e5d647_isrForCrosstalkSources_855416_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ac104252-390f-489c-b9af-2fbd99e5d647_isrForCrosstalkSources_855416_54.stderr parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 84b41371cae0e4733cb928361c1f2c5f parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/815b423f-284b-4880-90d2-7f336a815734_isrForCrosstalkSources_855410_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/815b423f-284b-4880-90d2-7f336a815734_isrForCrosstalkSources_855410_32.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 3c7cf26c35b8fd84fd5951861f89e4d1 parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc0d9a60-65f7-4085-a1b7-016c594e16fc_isrForCrosstalkSources_855421_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc0d9a60-65f7-4085-a1b7-016c594e16fc_isrForCrosstalkSources_855421_43.stderr parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash fba2fe7edff44b06d1f361c838a70e4a parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/e329ce36-305b-42bb-b401-dc21e2c58560_isrForCrosstalkSources_855397_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/e329ce36-305b-42bb-b401-dc21e2c58560_isrForCrosstalkSources_855397_16.stderr parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 198f4c72bdbd3101c6975590b8004e36 parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/eaa24ccb-c435-4b8b-be58-1e9d46e5bee2_isrForCrosstalkSources_855459_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/eaa24ccb-c435-4b8b-be58-1e9d46e5bee2_isrForCrosstalkSources_855459_12.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 91826ff4640e4e9b420b47ec7ffd8e38 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9fdfd991-5c23-4736-9bda-5e3cc0ece19f_isrForCrosstalkSources_855408_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9fdfd991-5c23-4736-9bda-5e3cc0ece19f_isrForCrosstalkSources_855408_57.stderr parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash 17829e454b4ff80560728486828a6256 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa18ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d0012d71-bc90-48bf-baf0-46c42a585549_isrForCrosstalkSources_855454_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d0012d71-bc90-48bf-baf0-46c42a585549_isrForCrosstalkSources_855454_1.stderr parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash cc40ef06ca8160f8a9c5cd9510ca1662 parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ec524951-1239-4d76-b901-c757e6e42f8f_isrForCrosstalkSources_855438_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ec524951-1239-4d76-b901-c757e6e42f8f_isrForCrosstalkSources_855438_43.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash b28e226d0ab96c13c4b8d5e5c9d5c7e4 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/472abddd-d06a-4cc8-83f4-ff81179a352b_isrForCrosstalkSources_855395_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/472abddd-d06a-4cc8-83f4-ff81179a352b_isrForCrosstalkSources_855395_24.stderr parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 4d496ec53b763f1d7f747613e8baff56 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/aca1156e-b77f-424f-9238-ed4f20d988dc_isrForCrosstalkSources_855395_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/aca1156e-b77f-424f-9238-ed4f20d988dc_isrForCrosstalkSources_855395_49.stderr parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 986289a25d81da4e37f0138ea0fc3b95 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2eac81f-7405-4854-bfe0-a2a9a8c458a1_isrForCrosstalkSources_855402_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2eac81f-7405-4854-bfe0-a2a9a8c458a1_isrForCrosstalkSources_855402_46.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash d76973ad477dce9c036d6de5ae68dc28 parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/450f629e-7f7e-44fa-8e0f-0832e94473a8_isrForCrosstalkSources_855398_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/450f629e-7f7e-44fa-8e0f-0832e94473a8_isrForCrosstalkSources_855398_58.stderr parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 0d51655d1cd8c3ff4fdcdadd8a8c3671 parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afdba5bf-6b4d-4652-9643-a83d5a5e68d1_isrForCrosstalkSources_855452_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afdba5bf-6b4d-4652-9643-a83d5a5e68d1_isrForCrosstalkSources_855452_43.stderr parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash f8558893146b8ddacf576a833d72a68a parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b151aa41-3705-4240-9e96-37f796e80961_isrForCrosstalkSources_855426_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b151aa41-3705-4240-9e96-37f796e80961_isrForCrosstalkSources_855426_24.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash c16a046db7f4212251fa93343790e696 parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0fe32d31-55e9-4084-ad9f-a4eae2be976e_isrForCrosstalkSources_855417_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0fe32d31-55e9-4084-ad9f-a4eae2be976e_isrForCrosstalkSources_855417_32.stderr parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash a7bdad810ecb1ec83f2878a302da3543 parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa196c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5f6cd64c-b440-4f8a-bb08-3d5dd08d9f2a_isrForCrosstalkSources_855440_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5f6cd64c-b440-4f8a-bb08-3d5dd08d9f2a_isrForCrosstalkSources_855440_19.stderr parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 5da03ab37e225d09b51245ccec600f2f parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e8b8871d-1c02-451c-80fe-f62eec14727f_isrForCrosstalkSources_855418_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e8b8871d-1c02-451c-80fe-f62eec14727f_isrForCrosstalkSources_855418_45.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash 5481eb2534b79e02a13c1120c9f1dc22 parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa6665c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e8517d88-6e50-4b8a-8fa5-2da09f27bf10_isrForCrosstalkSources_855455_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e8517d88-6e50-4b8a-8fa5-2da09f27bf10_isrForCrosstalkSources_855455_10.stderr parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash d72ffe90fe976cbee7363ce260d4f19f parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3303f96c-5cc7-4288-8a7a-1275d6fa0751_isrForCrosstalkSources_855416_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3303f96c-5cc7-4288-8a7a-1275d6fa0751_isrForCrosstalkSources_855416_4.stderr parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash ed086ea6990acb4a6ec8c026cbd18267 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3d663ab8-634c-494b-8624-28d990f75140_isrForCrosstalkSources_855379_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3d663ab8-634c-494b-8624-28d990f75140_isrForCrosstalkSources_855379_11.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash b604b8a39b73d4c219e38bb593934a23 parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6de70b17-a552-4f25-ae54-9c37d944226b_isrForCrosstalkSources_855432_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6de70b17-a552-4f25-ae54-9c37d944226b_isrForCrosstalkSources_855432_11.stderr parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash b9747deb8631c5fafd11ec6ae9b79449 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5e09a3c2-7266-4b51-85e7-0ad5f7cb4112_isrForCrosstalkSources_855422_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5e09a3c2-7266-4b51-85e7-0ad5f7cb4112_isrForCrosstalkSources_855422_26.stderr parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash df2336a4962cb895c037539deff0aae4 parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c2b77439-8bce-47f5-ae57-0cfb957edbd8_isrForCrosstalkSources_855418_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c2b77439-8bce-47f5-ae57-0cfb957edbd8_isrForCrosstalkSources_855418_51.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 669be3fdfddda4d4197ab0beab89cf9d parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/85810983-268b-4b1d-bd17-db567dd8c2ab_isrForCrosstalkSources_855432_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/85810983-268b-4b1d-bd17-db567dd8c2ab_isrForCrosstalkSources_855432_29.stderr parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash 227472478fe7dbc61f3b32882678696b parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:10 proc_lsst.multi:170 [INFO] [multi] job_id 16964055 proc_lsst.multi INFO: [multi] job_id 16964055 2024-03-09 14:50:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16964055 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16964055 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16964055 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-09 14:50:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-09 14:50:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 14:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7a28f6b6-bf7a-4872-8ed4-d17ceecb581c_isrForCrosstalkSources_855448_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7a28f6b6-bf7a-4872-8ed4-d17ceecb581c_isrForCrosstalkSources_855448_26.stderr parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash 68988617701e6367cf790b04cc106610 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa194e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ee867dab-ef11-4046-ba98-47638f079966_isrForCrosstalkSources_855451_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ee867dab-ef11-4046-ba98-47638f079966_isrForCrosstalkSources_855451_13.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash cc8222849a3db2df1f981cd1a47a8334 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cf622e73-c8ae-4b83-aed0-8b9da427bf93_isrForCrosstalkSources_855422_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cf622e73-c8ae-4b83-aed0-8b9da427bf93_isrForCrosstalkSources_855422_58.stderr parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 3256e6fe01adf554845f6716a6daa407 parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa19760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/7fe35ea8-d553-42db-9d97-ff5c215f84ee_isrForCrosstalkSources_855389_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/7fe35ea8-d553-42db-9d97-ff5c215f84ee_isrForCrosstalkSources_855389_8.stderr parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 24e82faddd17aa2ab3c2ace78843e010 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3cf31f25-dd00-4dcc-b0ba-9d2aa849ad7e_isrForCrosstalkSources_855413_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3cf31f25-dd00-4dcc-b0ba-9d2aa849ad7e_isrForCrosstalkSources_855413_50.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 0f94a745dd5b42442d0dffbfac2a9b22 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/64f9faa2-4ee7-4e57-b677-449176b7292c_isrForCrosstalkSources_855427_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/64f9faa2-4ee7-4e57-b677-449176b7292c_isrForCrosstalkSources_855427_14.stderr parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash d9ad3d93e1a2b3f48a1ef04d1dc0baf4 parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/83d7af3b-6755-4d4a-b8d3-0dfa0223b197_isrForCrosstalkSources_855428_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/83d7af3b-6755-4d4a-b8d3-0dfa0223b197_isrForCrosstalkSources_855428_33.stderr parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash f8c316c261b4538a1ce83df0b5180432 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/22cae9f3-1644-4cf5-b0ac-dd4e0672b45b_isrForCrosstalkSources_855449_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/22cae9f3-1644-4cf5-b0ac-dd4e0672b45b_isrForCrosstalkSources_855449_25.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 6522aa8f5180b96e6a856f2e589d3325 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/05af7bd4-a040-41fe-ad2a-23b8c3190c20_isrForCrosstalkSources_855399_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/05af7bd4-a040-41fe-ad2a-23b8c3190c20_isrForCrosstalkSources_855399_19.stderr parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 41d2f8673ec8e3f230a288bae00c3372 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/27d364a7-0461-4238-bec5-aa4f53230a48_isrForCrosstalkSources_855383_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/27d364a7-0461-4238-bec5-aa4f53230a48_isrForCrosstalkSources_855383_24.stderr parsl.dataflow.dflow DEBUG: Task 1116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1116 has memoization hash df96a937f4e51e48b1995bd250516106 parsl.dataflow.memoization INFO: Task 1116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1116 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/759eb945-049d-43a1-b972-1732c605feed_isrForCrosstalkSources_855398_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/759eb945-049d-43a1-b972-1732c605feed_isrForCrosstalkSources_855398_24.stderr parsl.dataflow.dflow DEBUG: Task 1117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1117 has memoization hash a888bdb2224ec0a7a104a5c697829fcf parsl.dataflow.memoization INFO: Task 1117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1117 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/87cf0a73-be29-4263-af15-95055e9588e6_isrForCrosstalkSources_855423_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/87cf0a73-be29-4263-af15-95055e9588e6_isrForCrosstalkSources_855423_10.stderr parsl.dataflow.dflow DEBUG: Task 1118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1118 has memoization hash 5a5a1a883ba54fa954625d7ff5b8f291 parsl.dataflow.memoization INFO: Task 1118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1118 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/80f88bf2-b828-4777-a467-03e892576052_isrForCrosstalkSources_855411_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/80f88bf2-b828-4777-a467-03e892576052_isrForCrosstalkSources_855411_58.stderr parsl.dataflow.dflow DEBUG: Task 1119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1119 has memoization hash 75033a4745517e732529f4f35b620685 parsl.dataflow.memoization INFO: Task 1119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1119 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/18e7f7ca-190e-4988-bc18-d02e25bbcfe8_isrForCrosstalkSources_855440_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/18e7f7ca-190e-4988-bc18-d02e25bbcfe8_isrForCrosstalkSources_855440_4.stderr parsl.dataflow.dflow DEBUG: Task 1120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1120 has memoization hash c2a2965b877a7fabe8dae212e70d2bfe parsl.dataflow.memoization INFO: Task 1120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1120 try 0 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c69be069-5463-4635-af75-8f171a176a5a_isrForCrosstalkSources_855423_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c69be069-5463-4635-af75-8f171a176a5a_isrForCrosstalkSources_855423_52.stderr parsl.dataflow.dflow DEBUG: Task 1121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1121 has memoization hash 724dc40fdc6dcf535b5e6122272324cb parsl.dataflow.memoization INFO: Task 1121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1121 try 0 launched on executor multi with executor id 1122 parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0b16b48f-883e-4827-bc17-81ea736cdafc_isrForCrosstalkSources_855383_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0b16b48f-883e-4827-bc17-81ea736cdafc_isrForCrosstalkSources_855383_17.stderr parsl.dataflow.dflow DEBUG: Task 1122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1122 has memoization hash aa9955058f234eb769192e69f2c04a08 parsl.dataflow.memoization INFO: Task 1122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1122 try 0 launched on executor multi with executor id 1123 parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f883ab6-2b11-47eb-ad23-7f369abb5cd7_isrForCrosstalkSources_855415_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f883ab6-2b11-47eb-ad23-7f369abb5cd7_isrForCrosstalkSources_855415_13.stderr parsl.dataflow.dflow DEBUG: Task 1123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1123 has memoization hash 7605325d5ddb721cb804d9f845553541 parsl.dataflow.memoization INFO: Task 1123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1123 try 0 launched on executor multi with executor id 1124 parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c1316e3d-f936-4222-8a40-d482553e0b02_isrForCrosstalkSources_855405_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c1316e3d-f936-4222-8a40-d482553e0b02_isrForCrosstalkSources_855405_27.stderr parsl.dataflow.dflow DEBUG: Task 1124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1124 has memoization hash 2449637782c2cd97d5d9c935559585d3 parsl.dataflow.memoization INFO: Task 1124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1124 try 0 launched on executor multi with executor id 1125 parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0b54c141-8372-4726-a29f-94fd20a5631e_isrForCrosstalkSources_855427_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0b54c141-8372-4726-a29f-94fd20a5631e_isrForCrosstalkSources_855427_47.stderr parsl.dataflow.dflow DEBUG: Task 1125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1125 has memoization hash 7c5f99aa25ee607e4b84a9dff6a0d8e8 parsl.dataflow.memoization INFO: Task 1125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1125 try 0 launched on executor multi with executor id 1126 parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf661f1f-b3e6-4550-bc94-004aaa75719c_isrForCrosstalkSources_855401_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf661f1f-b3e6-4550-bc94-004aaa75719c_isrForCrosstalkSources_855401_50.stderr parsl.dataflow.dflow DEBUG: Task 1126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1126 has memoization hash 065df532442a6d0b1c42e35c0b7707a0 parsl.dataflow.memoization INFO: Task 1126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1126 try 0 launched on executor multi with executor id 1127 parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/0c38afec-96c6-4dd1-bc5c-d9a0ec2e2dd1_isrForCrosstalkSources_855455_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/0c38afec-96c6-4dd1-bc5c-d9a0ec2e2dd1_isrForCrosstalkSources_855455_56.stderr parsl.dataflow.dflow DEBUG: Task 1127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1127 has memoization hash a54dafa123c35803c75816849ce338ff parsl.dataflow.memoization INFO: Task 1127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1127 try 0 launched on executor multi with executor id 1128 parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d17a0f09-f881-4f86-b4b4-6052f8dd1ff7_isrForCrosstalkSources_855385_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d17a0f09-f881-4f86-b4b4-6052f8dd1ff7_isrForCrosstalkSources_855385_37.stderr parsl.dataflow.dflow DEBUG: Task 1128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1128 has memoization hash da36ce8169262ccae04ad21b9136c059 parsl.dataflow.memoization INFO: Task 1128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1128 try 0 launched on executor multi with executor id 1129 parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/66fad533-283d-4790-b9af-c2370326dfcf_isrForCrosstalkSources_855398_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/66fad533-283d-4790-b9af-c2370326dfcf_isrForCrosstalkSources_855398_54.stderr parsl.dataflow.dflow DEBUG: Task 1129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1129 has memoization hash 69e9aca02c05248a3fd87046996afa6e parsl.dataflow.memoization INFO: Task 1129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1129 try 0 launched on executor multi with executor id 1130 parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4dd62509-55cd-4279-8f13-b28f2ebf8894_isrForCrosstalkSources_855390_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4dd62509-55cd-4279-8f13-b28f2ebf8894_isrForCrosstalkSources_855390_44.stderr parsl.dataflow.dflow DEBUG: Task 1130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1130 has memoization hash 93f693501f6c98b1d55c4916268a4f59 parsl.dataflow.memoization INFO: Task 1130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f92a5490ba85f93f00f0233eed719f9c with result from task 0 parsl.dataflow.dflow INFO: Parsl task 1130 try 0 launched on executor multi with executor id 1131 parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1fa79e63-2459-4b3d-a903-3cb1cfae4283_isrForCrosstalkSources_855454_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1fa79e63-2459-4b3d-a903-3cb1cfae4283_isrForCrosstalkSources_855454_27.stderr parsl.dataflow.dflow DEBUG: Task 1131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1131 has memoization hash 3f75e589ee7d1698cb707bafe14b0299 parsl.dataflow.memoization INFO: Task 1131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1131 try 0 launched on executor multi with executor id 1132 parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8324c2-c965-453c-8f78-c9e91511f296_isrForCrosstalkSources_855404_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8324c2-c965-453c-8f78-c9e91511f296_isrForCrosstalkSources_855404_20.stderr parsl.dataflow.dflow DEBUG: Task 1132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1132 has memoization hash 0d9d088b27c025bbac496b56e3bdce6d parsl.dataflow.memoization INFO: Task 1132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8418eec2-61a0-4f28-944b-279b9a4686a5_isrForCrosstalkSources_855449_46.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8418eec2-61a0-4f28-944b-279b9a4686a5_isrForCrosstalkSources_855449_46.stderr 2024-03-09 14:50:10 proc_lsst.multi:170 [INFO] [multi] job_id 16964056 proc_lsst.multi INFO: [multi] job_id 16964056 2024-03-09 14:50:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16964056 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16964056 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16964056 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-09 14:50:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-09 14:50:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1132 try 0 launched on executor multi with executor id 1133 parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1838ce99-25bf-47f6-b4ca-29fa3087dcc4_isrForCrosstalkSources_855408_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1838ce99-25bf-47f6-b4ca-29fa3087dcc4_isrForCrosstalkSources_855408_39.stderr parsl.dataflow.dflow DEBUG: Task 1133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1133 has memoization hash be8861fcc95c1e5729c1f5b608ed6932 parsl.dataflow.memoization INFO: Task 1133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1133 try 0 launched on executor multi with executor id 1134 parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9c953f81-bda5-4498-9b21-18d0a2387798_isrForCrosstalkSources_855399_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9c953f81-bda5-4498-9b21-18d0a2387798_isrForCrosstalkSources_855399_20.stderr parsl.dataflow.dflow DEBUG: Task 1134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1134 has memoization hash b24a95af06982687dff66f117e433266 parsl.dataflow.memoization INFO: Task 1134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1134 try 0 launched on executor multi with executor id 1135 parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a8d749d8-9e29-4cc2-a4a7-64a320d4b652_isrForCrosstalkSources_855416_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a8d749d8-9e29-4cc2-a4a7-64a320d4b652_isrForCrosstalkSources_855416_48.stderr parsl.dataflow.dflow DEBUG: Task 1135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1135 has memoization hash 68c0531972ff9e9c1eff138cd72d9587 parsl.dataflow.memoization INFO: Task 1135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1135 try 0 launched on executor multi with executor id 1136 parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/aea48172-435c-416f-a598-3de0b1f510ed_isrForCrosstalkSources_855430_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/aea48172-435c-416f-a598-3de0b1f510ed_isrForCrosstalkSources_855430_20.stderr parsl.dataflow.dflow DEBUG: Task 1136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1136 has memoization hash 2203bac86d95e6341afb2dc196178cbc parsl.dataflow.memoization INFO: Task 1136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1136 try 0 launched on executor multi with executor id 1137 parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5ee66bce-24c1-4988-8730-919f9e7ad03e_isrForCrosstalkSources_855424_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5ee66bce-24c1-4988-8730-919f9e7ad03e_isrForCrosstalkSources_855424_9.stderr parsl.dataflow.dflow DEBUG: Task 1137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1137 has memoization hash b9dc3df4bde4b436b015fdc96d21a557 parsl.dataflow.memoization INFO: Task 1137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1137 try 0 launched on executor multi with executor id 1138 parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/041752d6-a314-4bb4-8d67-c19bfbf13efe_isrForCrosstalkSources_855414_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/041752d6-a314-4bb4-8d67-c19bfbf13efe_isrForCrosstalkSources_855414_57.stderr parsl.dataflow.dflow DEBUG: Task 1138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1138 has memoization hash 7c0158cdf885fdcc31673b6d9bb0ccab parsl.dataflow.memoization INFO: Task 1138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1138 try 0 launched on executor multi with executor id 1139 parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/deaaacd0-0fd2-4ea8-b21d-c712c559a72f_isrForCrosstalkSources_855389_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/deaaacd0-0fd2-4ea8-b21d-c712c559a72f_isrForCrosstalkSources_855389_21.stderr parsl.dataflow.dflow DEBUG: Task 1139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1139 has memoization hash f1aae9cfab05d2f3836da634beb30bd1 parsl.dataflow.memoization INFO: Task 1139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1139 try 0 launched on executor multi with executor id 1140 parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2c6991cb-711f-4dba-b934-8399f9d7aedb_isrForCrosstalkSources_855456_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2c6991cb-711f-4dba-b934-8399f9d7aedb_isrForCrosstalkSources_855456_52.stderr parsl.dataflow.dflow DEBUG: Task 1140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1140 has memoization hash f14c2d1b14051e0444270d12facd526a parsl.dataflow.memoization INFO: Task 1140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1140 try 0 launched on executor multi with executor id 1141 parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aec4830f-02d3-4866-9860-af4adb13d960_isrForCrosstalkSources_855449_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aec4830f-02d3-4866-9860-af4adb13d960_isrForCrosstalkSources_855449_56.stderr parsl.dataflow.dflow DEBUG: Task 1141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1141 has memoization hash 6b36d05d220529c8982e47cf8fdbc4a5 parsl.dataflow.memoization INFO: Task 1141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1141 try 0 launched on executor multi with executor id 1142 parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/706dbfe2-44bf-4875-8e65-c44dda7dd8fd_isrForCrosstalkSources_855405_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/706dbfe2-44bf-4875-8e65-c44dda7dd8fd_isrForCrosstalkSources_855405_4.stderr parsl.dataflow.dflow DEBUG: Task 1142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1142 has memoization hash 4844e2fc9a9ca7aa667469e2f73560e7 parsl.dataflow.memoization INFO: Task 1142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1142 try 0 launched on executor multi with executor id 1143 parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8534491-d0eb-4138-a490-5e99f18c12c9_isrForCrosstalkSources_855435_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8534491-d0eb-4138-a490-5e99f18c12c9_isrForCrosstalkSources_855435_54.stderr parsl.dataflow.dflow DEBUG: Task 1143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1143 has memoization hash 5b437a9c9b85d8899f142c3e3b8446ba parsl.dataflow.memoization INFO: Task 1143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1143 try 0 launched on executor multi with executor id 1144 parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/cc1f4ad7-bebc-41e3-a8e7-91b0d0fc0d78_isrForCrosstalkSources_855433_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/cc1f4ad7-bebc-41e3-a8e7-91b0d0fc0d78_isrForCrosstalkSources_855433_53.stderr parsl.dataflow.dflow DEBUG: Task 1144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1144 has memoization hash 335758e25517741cc73291d98f32505a parsl.dataflow.memoization INFO: Task 1144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1144 try 0 launched on executor multi with executor id 1145 parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1323489a-957c-4bf0-9312-dbfbaaf5bd53_isrForCrosstalkSources_855425_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1323489a-957c-4bf0-9312-dbfbaaf5bd53_isrForCrosstalkSources_855425_24.stderr parsl.dataflow.dflow DEBUG: Task 1145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1145 has memoization hash 64d4493f2e6c256235f409f3985c7447 parsl.dataflow.memoization INFO: Task 1145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20680f1d85fa696d0fabc4bf60cee66f with result from task 1 parsl.dataflow.dflow INFO: Parsl task 1145 try 0 launched on executor multi with executor id 1146 parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c7b2b421-bf8f-4774-a193-b8385a1c6108_isrForCrosstalkSources_855426_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c7b2b421-bf8f-4774-a193-b8385a1c6108_isrForCrosstalkSources_855426_49.stderr parsl.dataflow.dflow DEBUG: Task 1146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1146 has memoization hash 81000f2986489ada7a9d3028a30aca80 parsl.dataflow.memoization INFO: Task 1146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/1a958d77-8bf9-425a-86d4-802881926d34_isrForCrosstalkSources_855390_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/1a958d77-8bf9-425a-86d4-802881926d34_isrForCrosstalkSources_855390_35.stderr parsl.dataflow.dflow INFO: Parsl task 1146 try 0 launched on executor multi with executor id 1147 parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d29abc-4cc2-4f82-96f8-c02fff4da0a5_isrForCrosstalkSources_855448_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d29abc-4cc2-4f82-96f8-c02fff4da0a5_isrForCrosstalkSources_855448_55.stderr parsl.dataflow.dflow DEBUG: Task 1147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1147 has memoization hash 15a046f467531d98fc5916ffd7715115 parsl.dataflow.memoization INFO: Task 1147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1147 try 0 launched on executor multi with executor id 1148 parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/373a00b9-a18b-4cbc-bd99-2126c83db6f0_isrForCrosstalkSources_855391_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/373a00b9-a18b-4cbc-bd99-2126c83db6f0_isrForCrosstalkSources_855391_54.stderr parsl.dataflow.dflow DEBUG: Task 1148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1148 has memoization hash 2b863bcd95c897530b7f98ec54b84ea0 parsl.dataflow.memoization INFO: Task 1148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1148 try 0 launched on executor multi with executor id 1149 parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/cb7cec95-694f-47f9-9ac4-75655d98dce7_isrForCrosstalkSources_855413_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/cb7cec95-694f-47f9-9ac4-75655d98dce7_isrForCrosstalkSources_855413_33.stderr parsl.dataflow.dflow DEBUG: Task 1149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1149 has memoization hash 8a25eaa038a64e9a0148d1afa01fa136 parsl.dataflow.memoization INFO: Task 1149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1149 try 0 launched on executor multi with executor id 1150 parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f738f891-4c7b-4b99-acbe-10c13e3f3fe1_isrForCrosstalkSources_855395_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f738f891-4c7b-4b99-acbe-10c13e3f3fe1_isrForCrosstalkSources_855395_31.stderr parsl.dataflow.dflow DEBUG: Task 1150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1150 has memoization hash 84d8470bdb224b9abe958270c859aab3 parsl.dataflow.memoization INFO: Task 1150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1150 try 0 launched on executor multi with executor id 1151 parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0936f4c1-e53e-4d93-9166-286cb9ebb051_isrForCrosstalkSources_855408_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0936f4c1-e53e-4d93-9166-286cb9ebb051_isrForCrosstalkSources_855408_5.stderr parsl.dataflow.dflow DEBUG: Task 1151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1151 has memoization hash 5af56ae5cef2d2f2375865bd0d66b7ef parsl.dataflow.memoization INFO: Task 1151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1151 try 0 launched on executor multi with executor id 1152 parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e29cffcc-a015-466e-948c-92d0c19d6953_isrForCrosstalkSources_855385_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e29cffcc-a015-466e-948c-92d0c19d6953_isrForCrosstalkSources_855385_7.stderr parsl.dataflow.dflow DEBUG: Task 1152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1152 has memoization hash d311e186ebb547f2cacaa3b790cd91f3 parsl.dataflow.memoization INFO: Task 1152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1152 try 0 launched on executor multi with executor id 1153 parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/41a88532-7ec6-464d-a274-aced7224ea91_isrForCrosstalkSources_855429_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/41a88532-7ec6-464d-a274-aced7224ea91_isrForCrosstalkSources_855429_38.stderr parsl.dataflow.dflow DEBUG: Task 1153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1153 has memoization hash 49ce91d94bf371472ed91e5fd547950f parsl.dataflow.memoization INFO: Task 1153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1153 try 0 launched on executor multi with executor id 1154 parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/02ec537e-9b30-41c9-8525-a621c87d8ec9_isrForCrosstalkSources_855386_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/02ec537e-9b30-41c9-8525-a621c87d8ec9_isrForCrosstalkSources_855386_20.stderr parsl.dataflow.dflow DEBUG: Task 1154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1154 has memoization hash 59cf88589955df486bd27b454f5b94b5 parsl.dataflow.memoization INFO: Task 1154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1154 try 0 launched on executor multi with executor id 1155 parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5cf7617f-65ca-42ce-8d4e-8fae13830690_isrForCrosstalkSources_855441_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5cf7617f-65ca-42ce-8d4e-8fae13830690_isrForCrosstalkSources_855441_14.stderr parsl.dataflow.dflow DEBUG: Task 1155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1155 has memoization hash 2d05f63667e36afd3cc1f0ddf0e200be parsl.dataflow.memoization INFO: Task 1155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:11 proc_lsst.multi:170 [INFO] [multi] job_id 16964057 proc_lsst.multi INFO: [multi] job_id 16964057 2024-03-09 14:50:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:50:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:11 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16964057 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16964057 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16964057 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-09 14:50:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-09 14:50:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1155 try 0 launched on executor multi with executor id 1156 parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/36d64beb-f580-421d-a1a1-2495aae50c22_isrForCrosstalkSources_855432_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/36d64beb-f580-421d-a1a1-2495aae50c22_isrForCrosstalkSources_855432_28.stderr parsl.dataflow.dflow DEBUG: Task 1156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1156 has memoization hash dec89b0b2929b7edb8bce791ea26b20d parsl.dataflow.memoization INFO: Task 1156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1156 try 0 launched on executor multi with executor id 1157 parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a7ff794-7428-4c73-9b54-90700e07396f_isrForCrosstalkSources_855449_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a7ff794-7428-4c73-9b54-90700e07396f_isrForCrosstalkSources_855449_54.stderr parsl.dataflow.dflow DEBUG: Task 1157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1157 has memoization hash 478e87550c39c9f783de89fc43240fa5 parsl.dataflow.memoization INFO: Task 1157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1157 try 0 launched on executor multi with executor id 1158 parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bfdefd54-6547-49a8-baec-41fd306b4fb0_isrForCrosstalkSources_855424_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bfdefd54-6547-49a8-baec-41fd306b4fb0_isrForCrosstalkSources_855424_34.stderr parsl.dataflow.dflow DEBUG: Task 1158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1158 has memoization hash 53b779d0e189bc1b1802095977b32906 parsl.dataflow.memoization INFO: Task 1158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1158 try 0 launched on executor multi with executor id 1159 parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/11c86f94-1d59-43c2-a085-a73ed4aa7f15_isrForCrosstalkSources_855453_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/11c86f94-1d59-43c2-a085-a73ed4aa7f15_isrForCrosstalkSources_855453_18.stderr parsl.dataflow.dflow DEBUG: Task 1159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1159 has memoization hash 06471e2e904e7b6027037de9028cdfa9 parsl.dataflow.memoization INFO: Task 1159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1159 try 0 launched on executor multi with executor id 1160 parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1a5afa20-2f38-4804-95b6-574ec21ddd0d_isrForCrosstalkSources_855391_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1a5afa20-2f38-4804-95b6-574ec21ddd0d_isrForCrosstalkSources_855391_25.stderr parsl.dataflow.dflow DEBUG: Task 1160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1160 has memoization hash af2bfabfab43be27b2f8f0df15f6934f parsl.dataflow.memoization INFO: Task 1160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1160 try 0 launched on executor multi with executor id 1161 parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/41529b21-29a6-44e8-b8b1-4e5d96c875eb_isrForCrosstalkSources_855428_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/41529b21-29a6-44e8-b8b1-4e5d96c875eb_isrForCrosstalkSources_855428_34.stderr parsl.dataflow.dflow DEBUG: Task 1161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1161 has memoization hash 14b951cb9e640aecf6caa91693d05d72 parsl.dataflow.memoization INFO: Task 1161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1161 try 0 launched on executor multi with executor id 1162 parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91074912-ef4d-451b-b1bf-b329d637f442_isrForCrosstalkSources_855390_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91074912-ef4d-451b-b1bf-b329d637f442_isrForCrosstalkSources_855390_28.stderr parsl.dataflow.dflow DEBUG: Task 1162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1162 has memoization hash 35d2856bd243274ecebb4128e17b21a0 parsl.dataflow.memoization INFO: Task 1162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1162 try 0 launched on executor multi with executor id 1163 parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c8169b75-cbb6-4e37-b0d8-6b3c90acab5e_isrForCrosstalkSources_855392_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c8169b75-cbb6-4e37-b0d8-6b3c90acab5e_isrForCrosstalkSources_855392_46.stderr parsl.dataflow.dflow DEBUG: Task 1163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1163 has memoization hash b5ea4814dd084cf22573b5f9c777520a parsl.dataflow.memoization INFO: Task 1163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1163 try 0 launched on executor multi with executor id 1164 parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1fb59a67-8969-4266-952a-dd44381c8a12_isrForCrosstalkSources_855422_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1fb59a67-8969-4266-952a-dd44381c8a12_isrForCrosstalkSources_855422_18.stderr parsl.dataflow.dflow DEBUG: Task 1164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1164 has memoization hash 442036f90bfb0750069dbbb1a1052d2d parsl.dataflow.memoization INFO: Task 1164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1164 try 0 launched on executor multi with executor id 1165 parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/db6a928e-ba82-4e45-930d-9bff60392aee_isrForCrosstalkSources_855432_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/db6a928e-ba82-4e45-930d-9bff60392aee_isrForCrosstalkSources_855432_26.stderr parsl.dataflow.dflow DEBUG: Task 1165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1165 has memoization hash 141409f1be9b2d2ea4ab427b52c10af4 parsl.dataflow.memoization INFO: Task 1165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1165 try 0 launched on executor multi with executor id 1166 parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b04168c0-c285-41c3-b294-610644a98f09_isrForCrosstalkSources_855451_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b04168c0-c285-41c3-b294-610644a98f09_isrForCrosstalkSources_855451_48.stderr parsl.dataflow.dflow DEBUG: Task 1166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1166 has memoization hash 17d59c59f02ce183fb469da23a09c2bb parsl.dataflow.memoization INFO: Task 1166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1166 try 0 launched on executor multi with executor id 1167 parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ad663e9-9367-4f30-87f2-fe6f725223bf_isrForCrosstalkSources_855445_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ad663e9-9367-4f30-87f2-fe6f725223bf_isrForCrosstalkSources_855445_26.stderr parsl.dataflow.dflow DEBUG: Task 1167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1167 has memoization hash 98e23ca56c9423dac4a6fed6ad318a17 parsl.dataflow.memoization INFO: Task 1167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1167 try 0 launched on executor multi with executor id 1168 parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c11e2b42-ec9e-46f7-be3c-f94bba256568_isrForCrosstalkSources_855429_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c11e2b42-ec9e-46f7-be3c-f94bba256568_isrForCrosstalkSources_855429_26.stderr parsl.dataflow.dflow DEBUG: Task 1168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1168 has memoization hash b23aa6c9107655d09415d430afb4fafd parsl.dataflow.memoization INFO: Task 1168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1168 try 0 launched on executor multi with executor id 1169 parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/2e2a5419-434e-4447-976a-7f33379f7b1d_isrForCrosstalkSources_855439_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/2e2a5419-434e-4447-976a-7f33379f7b1d_isrForCrosstalkSources_855439_61.stderr parsl.dataflow.dflow DEBUG: Task 1169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1169 has memoization hash bbe67a72913ccedc2c356b4fbca8a81b parsl.dataflow.memoization INFO: Task 1169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1169 try 0 launched on executor multi with executor id 1170 parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fd2f93c7-3e99-4ea1-9d8b-485323632824_isrForCrosstalkSources_855401_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fd2f93c7-3e99-4ea1-9d8b-485323632824_isrForCrosstalkSources_855401_28.stderr parsl.dataflow.dflow DEBUG: Task 1170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1170 has memoization hash ba564039454c3a280009cda4ac41920b parsl.dataflow.memoization INFO: Task 1170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1170 try 0 launched on executor multi with executor id 1171 parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c9aa0789-a218-4cd1-900c-563e4bf436b4_isrForCrosstalkSources_855388_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c9aa0789-a218-4cd1-900c-563e4bf436b4_isrForCrosstalkSources_855388_32.stderr parsl.dataflow.dflow DEBUG: Task 1171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1171 has memoization hash d39a1294de0a2995ad7242f8d092256f parsl.dataflow.memoization INFO: Task 1171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1171 try 0 launched on executor multi with executor id 1172 parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4257af37-396b-4205-9858-57777ecb8d37_isrForCrosstalkSources_855409_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4257af37-396b-4205-9858-57777ecb8d37_isrForCrosstalkSources_855409_33.stderr parsl.dataflow.dflow DEBUG: Task 1172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1172 has memoization hash 065c9cba9b419c700ba04a9cdcd4908d parsl.dataflow.memoization INFO: Task 1172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1172 try 0 launched on executor multi with executor id 1173 parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e7377bf9-01e3-435d-875e-7496a1f70252_isrForCrosstalkSources_855419_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e7377bf9-01e3-435d-875e-7496a1f70252_isrForCrosstalkSources_855419_49.stderr parsl.dataflow.dflow DEBUG: Task 1173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1173 has memoization hash a18e420695e2b8ee670f36e349d49cb6 parsl.dataflow.memoization INFO: Task 1173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1173 try 0 launched on executor multi with executor id 1174 parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5e493692-ebd7-434b-b20b-ed83268c8305_isrForCrosstalkSources_855427_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5e493692-ebd7-434b-b20b-ed83268c8305_isrForCrosstalkSources_855427_2.stderr parsl.dataflow.dflow DEBUG: Task 1174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1174 has memoization hash 73282b2470ef17254375996306fb7a2c parsl.dataflow.memoization INFO: Task 1174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1174 try 0 launched on executor multi with executor id 1175 parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3595cb71-35f4-4b7b-bf04-7b80baa24194_isrForCrosstalkSources_855425_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3595cb71-35f4-4b7b-bf04-7b80baa24194_isrForCrosstalkSources_855425_18.stderr parsl.dataflow.dflow DEBUG: Task 1175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1175 has memoization hash 06dda96dc385b8e774334110e4704a41 parsl.dataflow.memoization INFO: Task 1175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1175 try 0 launched on executor multi with executor id 1176 parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/be647352-b0f6-495f-bd20-f795d5c3d258_isrForCrosstalkSources_855383_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/be647352-b0f6-495f-bd20-f795d5c3d258_isrForCrosstalkSources_855383_14.stderr parsl.dataflow.dflow DEBUG: Task 1176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1176 has memoization hash 4fd48f5f6869dfa81b5c1c572f5c89ef parsl.dataflow.memoization INFO: Task 1176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1176 try 0 launched on executor multi with executor id 1177 parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7b3424ee-145a-4d68-af9f-cbcbd0dc3228_isrForCrosstalkSources_855380_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7b3424ee-145a-4d68-af9f-cbcbd0dc3228_isrForCrosstalkSources_855380_6.stderr parsl.dataflow.dflow DEBUG: Task 1177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1177 has memoization hash bb8ff660a3d90fadda77fa97cf34f5d9 parsl.dataflow.memoization INFO: Task 1177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1177 try 0 launched on executor multi with executor id 1178 parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fab20f97-03cd-41e1-936c-1a8730238e42_isrForCrosstalkSources_855439_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fab20f97-03cd-41e1-936c-1a8730238e42_isrForCrosstalkSources_855439_51.stderr parsl.dataflow.dflow DEBUG: Task 1178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1178 has memoization hash 5812165e70eeeeeddac31a0f2af2d670 parsl.dataflow.memoization INFO: Task 1178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1178 try 0 launched on executor multi with executor id 1179 parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ce26efc5-5cbd-48d3-8afb-53f9cd8e56c9_isrForCrosstalkSources_855395_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ce26efc5-5cbd-48d3-8afb-53f9cd8e56c9_isrForCrosstalkSources_855395_6.stderr parsl.dataflow.dflow DEBUG: Task 1179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1179 has memoization hash e5edf563e21511dedd5f3c0f744f3742 parsl.dataflow.memoization INFO: Task 1179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1179 try 0 launched on executor multi with executor id 1180 parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c18b06ad-522f-460a-b717-070b78e1ed2e_isrForCrosstalkSources_855458_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c18b06ad-522f-460a-b717-070b78e1ed2e_isrForCrosstalkSources_855458_39.stderr parsl.dataflow.dflow DEBUG: Task 1180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1180 has memoization hash 2f287408744b0deac110176cbfb0dabf parsl.dataflow.memoization INFO: Task 1180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1180 try 0 launched on executor multi with executor id 1181 parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4a8be49b-23f7-4ff8-a3d0-0cbe5c009de1_isrForCrosstalkSources_855404_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4a8be49b-23f7-4ff8-a3d0-0cbe5c009de1_isrForCrosstalkSources_855404_50.stderr parsl.dataflow.dflow DEBUG: Task 1181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1181 has memoization hash 584368854afa4e8b185723668ea8dc29 parsl.dataflow.memoization INFO: Task 1181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1181 try 0 launched on executor multi with executor id 1182 parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/81faaf59-1024-40a9-b8c6-e8bdef12984e_isrForCrosstalkSources_855381_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/81faaf59-1024-40a9-b8c6-e8bdef12984e_isrForCrosstalkSources_855381_5.stderr parsl.dataflow.dflow DEBUG: Task 1182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1182 has memoization hash c13c7811c36e632acf066e70b179651d parsl.dataflow.memoization INFO: Task 1182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1182 try 0 launched on executor multi with executor id 1183 parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c3c87755-a3b8-4f8e-925f-9e21d8c4a121_isrForCrosstalkSources_855439_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c3c87755-a3b8-4f8e-925f-9e21d8c4a121_isrForCrosstalkSources_855439_14.stderr parsl.dataflow.dflow DEBUG: Task 1183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1183 has memoization hash 1560c7f133a4839605ed500cab247d46 parsl.dataflow.memoization INFO: Task 1183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1183 try 0 launched on executor multi with executor id 1184 parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6235e50a-ec5a-4b28-8e27-161faff1c473_isrForCrosstalkSources_855425_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6235e50a-ec5a-4b28-8e27-161faff1c473_isrForCrosstalkSources_855425_25.stderr parsl.dataflow.dflow DEBUG: Task 1184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1184 has memoization hash d1c80f16e9ecd744df141042008bec23 parsl.dataflow.memoization INFO: Task 1184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1184 try 0 launched on executor multi with executor id 1185 parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d20d862f-31a0-4727-82d4-b9e92c59e45f_isrForCrosstalkSources_855402_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d20d862f-31a0-4727-82d4-b9e92c59e45f_isrForCrosstalkSources_855402_11.stderr parsl.dataflow.dflow DEBUG: Task 1185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1185 has memoization hash f75e82f5cf4f854da1e2abee7dd9b922 parsl.dataflow.memoization INFO: Task 1185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1185 try 0 launched on executor multi with executor id 1186 parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9d93a122-c6f5-4c30-8fc4-8881fa70bc26_isrForCrosstalkSources_855400_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9d93a122-c6f5-4c30-8fc4-8881fa70bc26_isrForCrosstalkSources_855400_16.stderr parsl.dataflow.dflow DEBUG: Task 1186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1186 has memoization hash 02b0b790cbc1168f60cc51815c3cb959 parsl.dataflow.memoization INFO: Task 1186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2089c2fd78b9627b6d8c87769f35e9f with result from task 2 2024-03-09 14:50:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964058 proc_lsst.multi INFO: [multi] job_id 16964058 2024-03-09 14:50:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:50:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:12 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16964058 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16964058 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16964058 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-09 14:50:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1186 try 0 launched on executor multi with executor id 1187 parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3af24d6e-bb65-4423-9ddd-97543fb87ca8_isrForCrosstalkSources_855433_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3af24d6e-bb65-4423-9ddd-97543fb87ca8_isrForCrosstalkSources_855433_47.stderr parsl.dataflow.dflow DEBUG: Task 1187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1187 has memoization hash 567449b1339431d577b18a2bcfddd46f parsl.dataflow.memoization INFO: Task 1187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3f410c72-2667-4e61-a5b7-3cc997248cf6_isrForCrosstalkSources_855448_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3f410c72-2667-4e61-a5b7-3cc997248cf6_isrForCrosstalkSources_855448_6.stderr parsl.dataflow.dflow INFO: Parsl task 1187 try 0 launched on executor multi with executor id 1188 parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b65b6b3e-aa32-4945-a847-9a5627525381_isrForCrosstalkSources_855422_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b65b6b3e-aa32-4945-a847-9a5627525381_isrForCrosstalkSources_855422_57.stderr parsl.dataflow.dflow DEBUG: Task 1188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1188 has memoization hash 1e065be288de888d003671c43e792013 parsl.dataflow.memoization INFO: Task 1188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1188 try 0 launched on executor multi with executor id 1189 parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a497bf4d-79b5-47d1-a69d-2bc2a246df28_isrForCrosstalkSources_855405_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a497bf4d-79b5-47d1-a69d-2bc2a246df28_isrForCrosstalkSources_855405_37.stderr parsl.dataflow.dflow DEBUG: Task 1189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1189 has memoization hash 3beb8edda1df68d3794eed7134b8af97 parsl.dataflow.memoization INFO: Task 1189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1189 try 0 launched on executor multi with executor id 1190 parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8982692a-4a36-4902-9e91-36b00d56d5a9_isrForCrosstalkSources_855412_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8982692a-4a36-4902-9e91-36b00d56d5a9_isrForCrosstalkSources_855412_20.stderr parsl.dataflow.dflow DEBUG: Task 1190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1190 has memoization hash 48d1c03b4d11477e1f22e9eebecef10d parsl.dataflow.memoization INFO: Task 1190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1190 try 0 launched on executor multi with executor id 1191 parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d28d27a7-f59d-4014-abfa-9a243af49d0a_isrForCrosstalkSources_855388_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d28d27a7-f59d-4014-abfa-9a243af49d0a_isrForCrosstalkSources_855388_21.stderr parsl.dataflow.dflow DEBUG: Task 1191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1191 has memoization hash eb01295f4bfe35c4a300ef42aeb693f2 parsl.dataflow.memoization INFO: Task 1191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1191 try 0 launched on executor multi with executor id 1192 parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/868d5c9f-f70c-4ac7-a4ae-963041d7478b_isrForCrosstalkSources_855395_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/868d5c9f-f70c-4ac7-a4ae-963041d7478b_isrForCrosstalkSources_855395_36.stderr parsl.dataflow.dflow DEBUG: Task 1192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1192 has memoization hash 69a9abb2b799fecf02e40dce306767e2 parsl.dataflow.memoization INFO: Task 1192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1192 try 0 launched on executor multi with executor id 1193 parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/79e28332-e8c6-41bf-ae0c-e583bb410fd4_isrForCrosstalkSources_855385_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/79e28332-e8c6-41bf-ae0c-e583bb410fd4_isrForCrosstalkSources_855385_33.stderr parsl.dataflow.dflow DEBUG: Task 1193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1193 has memoization hash 6e7cb949215b23671255fb1249415038 parsl.dataflow.memoization INFO: Task 1193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1193 try 0 launched on executor multi with executor id 1194 parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/bab3f2ea-a0b7-4951-b32f-118b5f9b5334_isrForCrosstalkSources_855409_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/bab3f2ea-a0b7-4951-b32f-118b5f9b5334_isrForCrosstalkSources_855409_17.stderr parsl.dataflow.dflow DEBUG: Task 1194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1194 has memoization hash 1d2e524753ee47c78f642979dcba6350 parsl.dataflow.memoization INFO: Task 1194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1194 try 0 launched on executor multi with executor id 1195 parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cfa45cba-8a51-4a81-988c-f80eb521c9b2_isrForCrosstalkSources_855418_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cfa45cba-8a51-4a81-988c-f80eb521c9b2_isrForCrosstalkSources_855418_13.stderr parsl.dataflow.dflow DEBUG: Task 1195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1195 has memoization hash 4f1ccf371eaf725bb49b6f69843bea56 parsl.dataflow.memoization INFO: Task 1195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1195 try 0 launched on executor multi with executor id 1196 parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/10bd851f-d5d4-46b3-9de8-0920fe0fe860_isrForCrosstalkSources_855386_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/10bd851f-d5d4-46b3-9de8-0920fe0fe860_isrForCrosstalkSources_855386_31.stderr parsl.dataflow.dflow DEBUG: Task 1196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1196 has memoization hash a8c1910b3d58d72e51180b2caff4b40d parsl.dataflow.memoization INFO: Task 1196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1196 try 0 launched on executor multi with executor id 1197 parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/aa4ac66e-9bae-494e-a39b-594004925371_isrForCrosstalkSources_855405_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/aa4ac66e-9bae-494e-a39b-594004925371_isrForCrosstalkSources_855405_61.stderr parsl.dataflow.dflow DEBUG: Task 1197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1197 has memoization hash 541c68c9473e81e1516c41b30c25c74c parsl.dataflow.memoization INFO: Task 1197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1197 try 0 launched on executor multi with executor id 1198 parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a61ff73a-d552-4e2e-9b88-69a4b3ea2c73_isrForCrosstalkSources_855391_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a61ff73a-d552-4e2e-9b88-69a4b3ea2c73_isrForCrosstalkSources_855391_50.stderr parsl.dataflow.dflow DEBUG: Task 1198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1198 has memoization hash 8addfd2fe864ced18bc36a77d5984130 parsl.dataflow.memoization INFO: Task 1198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1198 try 0 launched on executor multi with executor id 1199 parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f6b3e8ea-e1e3-44b8-a4fc-c603c1487823_isrForCrosstalkSources_855424_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f6b3e8ea-e1e3-44b8-a4fc-c603c1487823_isrForCrosstalkSources_855424_39.stderr parsl.dataflow.dflow DEBUG: Task 1199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1199 has memoization hash a1a14a223f82f686d19401120d19ed36 parsl.dataflow.memoization INFO: Task 1199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1199 try 0 launched on executor multi with executor id 1200 parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/425ccc48-9d68-41c6-8738-801192920fe9_isrForCrosstalkSources_855458_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/425ccc48-9d68-41c6-8738-801192920fe9_isrForCrosstalkSources_855458_50.stderr parsl.dataflow.dflow DEBUG: Task 1200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1200 has memoization hash fd00c7b5e21416e52a951e86c921bccb parsl.dataflow.memoization INFO: Task 1200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1200 try 0 launched on executor multi with executor id 1201 parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72b5e654-d85e-4acd-862a-a9fcdf0bc7a8_isrForCrosstalkSources_855437_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72b5e654-d85e-4acd-862a-a9fcdf0bc7a8_isrForCrosstalkSources_855437_53.stderr parsl.dataflow.dflow DEBUG: Task 1201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1201 has memoization hash 70ddbaa6bd21b0d16424e40e6d5d4ad7 parsl.dataflow.memoization INFO: Task 1201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1201 try 0 launched on executor multi with executor id 1202 parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fea916fe-485c-4a2c-8e02-210bc1e5cc8e_isrForCrosstalkSources_855387_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fea916fe-485c-4a2c-8e02-210bc1e5cc8e_isrForCrosstalkSources_855387_12.stderr parsl.dataflow.dflow DEBUG: Task 1202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1202 has memoization hash 4af98d32b13db471be23b8980173f1ee parsl.dataflow.memoization INFO: Task 1202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1202 try 0 launched on executor multi with executor id 1203 parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/97ff951f-02b1-46be-9bc9-f8042a70e81d_isrForCrosstalkSources_855436_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/97ff951f-02b1-46be-9bc9-f8042a70e81d_isrForCrosstalkSources_855436_11.stderr parsl.dataflow.dflow DEBUG: Task 1203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1203 has memoization hash 2a7788273652058d448dfd1813cf9285 parsl.dataflow.memoization INFO: Task 1203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1203 try 0 launched on executor multi with executor id 1204 parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b584af14-bfd2-4bb5-890a-c29d71e2405f_isrForCrosstalkSources_855444_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b584af14-bfd2-4bb5-890a-c29d71e2405f_isrForCrosstalkSources_855444_35.stderr parsl.dataflow.dflow DEBUG: Task 1204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1204 has memoization hash 489e670cac84a44cb72e45e34ef99c19 parsl.dataflow.memoization INFO: Task 1204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1204 try 0 launched on executor multi with executor id 1205 parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/63e030b0-ecce-4ada-a602-91eee322c74d_isrForCrosstalkSources_855414_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/63e030b0-ecce-4ada-a602-91eee322c74d_isrForCrosstalkSources_855414_16.stderr parsl.dataflow.dflow DEBUG: Task 1205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1205 has memoization hash 2e26da2313f6811d11786f91e6d09e53 parsl.dataflow.memoization INFO: Task 1205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1205 try 0 launched on executor multi with executor id 1206 parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7a0896fa-fe54-4228-a11b-a6401e718cec_isrForCrosstalkSources_855422_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7a0896fa-fe54-4228-a11b-a6401e718cec_isrForCrosstalkSources_855422_12.stderr parsl.dataflow.dflow DEBUG: Task 1206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1206 has memoization hash 550a47a2aa1fc294b4e09c2df3fe551f parsl.dataflow.memoization INFO: Task 1206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1206 try 0 launched on executor multi with executor id 1207 parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/2d81d349-c386-487b-8912-43b8aa375a71_isrForCrosstalkSources_855423_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/2d81d349-c386-487b-8912-43b8aa375a71_isrForCrosstalkSources_855423_34.stderr parsl.dataflow.dflow DEBUG: Task 1207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1207 has memoization hash c16d216fce3dc2a39e6842f58ccc0078 parsl.dataflow.memoization INFO: Task 1207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964059 proc_lsst.multi INFO: [multi] job_id 16964059 2024-03-09 14:50:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:50:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:12 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16964059 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16964059 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16964059 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-09 14:50:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1207 try 0 launched on executor multi with executor id 1208 parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/192735ff-a08b-43e1-b9ee-a1150b687f47_isrForCrosstalkSources_855451_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/192735ff-a08b-43e1-b9ee-a1150b687f47_isrForCrosstalkSources_855451_4.stderr parsl.dataflow.dflow DEBUG: Task 1208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1208 has memoization hash 1f249e6fb2eb82a454f7c60e25d3a312 parsl.dataflow.memoization INFO: Task 1208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1208 try 0 launched on executor multi with executor id 1209 parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8613e3a1-c37b-4514-a38a-020de96ef597_isrForCrosstalkSources_855410_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8613e3a1-c37b-4514-a38a-020de96ef597_isrForCrosstalkSources_855410_44.stderr parsl.dataflow.dflow DEBUG: Task 1209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1209 has memoization hash 42fed2058d186953d3946fd5cdb2cc0d parsl.dataflow.memoization INFO: Task 1209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1209 try 0 launched on executor multi with executor id 1210 parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d683273d-47e1-4111-8d50-6c1e5045959d_isrForCrosstalkSources_855432_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d683273d-47e1-4111-8d50-6c1e5045959d_isrForCrosstalkSources_855432_55.stderr parsl.dataflow.dflow DEBUG: Task 1210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1210 has memoization hash 862c1ed4d08b971fc6e877a5e3ba5d44 parsl.dataflow.memoization INFO: Task 1210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aabb2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1210 try 0 launched on executor multi with executor id 1211 parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f4744bcf-b883-4bde-a77b-be1f55433f2f_isrForCrosstalkSources_855434_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f4744bcf-b883-4bde-a77b-be1f55433f2f_isrForCrosstalkSources_855434_49.stderr parsl.dataflow.dflow DEBUG: Task 1211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1211 has memoization hash d4410f52ceaa5c731f0c9eb33efb7b09 parsl.dataflow.memoization INFO: Task 1211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1211 try 0 launched on executor multi with executor id 1212 parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e0fae397-6983-47ac-afc7-15bc45c0e0d1_isrForCrosstalkSources_855401_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e0fae397-6983-47ac-afc7-15bc45c0e0d1_isrForCrosstalkSources_855401_20.stderr parsl.dataflow.dflow DEBUG: Task 1212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1212 has memoization hash 7e6370d85165c10af27a53ed2b2ad2dd parsl.dataflow.memoization INFO: Task 1212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1212 try 0 launched on executor multi with executor id 1213 parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/cabc9533-9833-400a-92df-7079a9599de9_isrForCrosstalkSources_855397_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/cabc9533-9833-400a-92df-7079a9599de9_isrForCrosstalkSources_855397_54.stderr parsl.dataflow.dflow DEBUG: Task 1213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1213 has memoization hash db2df1abb3c3ff0a1b29d5465f3fdb7c parsl.dataflow.memoization INFO: Task 1213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1213 try 0 launched on executor multi with executor id 1214 parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/84b6eeea-257b-4946-858d-f36cde3bd42d_isrForCrosstalkSources_855417_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/84b6eeea-257b-4946-858d-f36cde3bd42d_isrForCrosstalkSources_855417_6.stderr parsl.dataflow.dflow DEBUG: Task 1214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1214 has memoization hash 7e71c8d515ab0b746bb4fcfcd66e1d0b parsl.dataflow.memoization INFO: Task 1214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1214 try 0 launched on executor multi with executor id 1215 parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7518cfeb-27cc-4b13-b03a-e87027f3c5d3_isrForCrosstalkSources_855436_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7518cfeb-27cc-4b13-b03a-e87027f3c5d3_isrForCrosstalkSources_855436_57.stderr parsl.dataflow.dflow DEBUG: Task 1215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1215 has memoization hash 4dd5f3c3da8ca721d4853e38f8df1fd1 parsl.dataflow.memoization INFO: Task 1215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1215 try 0 launched on executor multi with executor id 1216 parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c12f0398-413f-4d7d-8f65-7eee3bab327c_isrForCrosstalkSources_855458_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c12f0398-413f-4d7d-8f65-7eee3bab327c_isrForCrosstalkSources_855458_40.stderr parsl.dataflow.dflow DEBUG: Task 1216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1216 has memoization hash 998ad9d1af4d908a328487506048e907 parsl.dataflow.memoization INFO: Task 1216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1216 try 0 launched on executor multi with executor id 1217 parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8cf3fa29-ea68-4ae7-aaed-16e7fc07c740_isrForCrosstalkSources_855391_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8cf3fa29-ea68-4ae7-aaed-16e7fc07c740_isrForCrosstalkSources_855391_52.stderr parsl.dataflow.dflow DEBUG: Task 1217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1217 has memoization hash de71658815225ee5a6fc365f787c9473 parsl.dataflow.memoization INFO: Task 1217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1217 try 0 launched on executor multi with executor id 1218 parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5dd04427-c96d-4b8a-823d-7b2f67234240_isrForCrosstalkSources_855400_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5dd04427-c96d-4b8a-823d-7b2f67234240_isrForCrosstalkSources_855400_13.stderr parsl.dataflow.dflow DEBUG: Task 1218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1218 has memoization hash aebd0bba9caba884add3b4b662f99962 parsl.dataflow.memoization INFO: Task 1218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1218 try 0 launched on executor multi with executor id 1219 parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5ff4434c-f6fe-4d10-a5c6-efd8ae013619_isrForCrosstalkSources_855450_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5ff4434c-f6fe-4d10-a5c6-efd8ae013619_isrForCrosstalkSources_855450_27.stderr parsl.dataflow.dflow DEBUG: Task 1219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1219 has memoization hash 41a201d0a3b81222cb4c743f2d0487df parsl.dataflow.memoization INFO: Task 1219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1219 try 0 launched on executor multi with executor id 1220 parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/36802aad-889d-4f8f-86d5-45143952a23c_isrForCrosstalkSources_855451_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/36802aad-889d-4f8f-86d5-45143952a23c_isrForCrosstalkSources_855451_21.stderr parsl.dataflow.dflow DEBUG: Task 1220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1220 has memoization hash c4d6aba6bfbe97e21569913209bc0eaf parsl.dataflow.memoization INFO: Task 1220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1220 try 0 launched on executor multi with executor id 1221 parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/572795c9-8ba3-4066-845b-cdd714519b7f_isrForCrosstalkSources_855408_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/572795c9-8ba3-4066-845b-cdd714519b7f_isrForCrosstalkSources_855408_48.stderr parsl.dataflow.dflow DEBUG: Task 1221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1221 has memoization hash add435c544ce7a8cde3639169952955e parsl.dataflow.memoization INFO: Task 1221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1221 try 0 launched on executor multi with executor id 1222 parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c7a425de-d49b-42ef-a3a4-3b242dab0721_isrForCrosstalkSources_855444_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c7a425de-d49b-42ef-a3a4-3b242dab0721_isrForCrosstalkSources_855444_1.stderr parsl.dataflow.dflow DEBUG: Task 1222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1222 has memoization hash 9d333e2a0e1369ae23a40ee1505986d0 parsl.dataflow.memoization INFO: Task 1222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1222 try 0 launched on executor multi with executor id 1223 parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3b87263d-ed4f-4e33-906e-8819f0622d3c_isrForCrosstalkSources_855450_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3b87263d-ed4f-4e33-906e-8819f0622d3c_isrForCrosstalkSources_855450_44.stderr parsl.dataflow.dflow DEBUG: Task 1223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1223 has memoization hash 65037fba7ec2cd4864c3b36900e8fcf8 parsl.dataflow.memoization INFO: Task 1223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1223 try 0 launched on executor multi with executor id 1224 parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/1bc03eb4-02bf-4af8-ae49-3b01a5a4c720_isrForCrosstalkSources_855433_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/1bc03eb4-02bf-4af8-ae49-3b01a5a4c720_isrForCrosstalkSources_855433_21.stderr parsl.dataflow.dflow DEBUG: Task 1224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1224 has memoization hash b632be506b2be26b576a6ba810ebce55 parsl.dataflow.memoization INFO: Task 1224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1224 try 0 launched on executor multi with executor id 1225 parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/82751ce6-69c1-40aa-bce7-84f433669989_isrForCrosstalkSources_855427_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/82751ce6-69c1-40aa-bce7-84f433669989_isrForCrosstalkSources_855427_25.stderr parsl.dataflow.dflow DEBUG: Task 1225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1225 has memoization hash e200266c29d916ad6103656ccb0bf7ff parsl.dataflow.memoization INFO: Task 1225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1225 try 0 launched on executor multi with executor id 1226 parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a5db1e0c-4f3a-4260-a680-9d6ee9338bd3_isrForCrosstalkSources_855457_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a5db1e0c-4f3a-4260-a680-9d6ee9338bd3_isrForCrosstalkSources_855457_22.stderr parsl.dataflow.dflow DEBUG: Task 1226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1226 has memoization hash 63b756e5fb953db9c356f5ba493e44a9 parsl.dataflow.memoization INFO: Task 1226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1226 try 0 launched on executor multi with executor id 1227 parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0a36c45d-f532-44d5-8cd8-8d84afdd7872_isrForCrosstalkSources_855387_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0a36c45d-f532-44d5-8cd8-8d84afdd7872_isrForCrosstalkSources_855387_56.stderr parsl.dataflow.dflow DEBUG: Task 1227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1227 has memoization hash ed33819195f57dbd8a558d3ca857d64f parsl.dataflow.memoization INFO: Task 1227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964060 proc_lsst.multi INFO: [multi] job_id 16964060 2024-03-09 14:50:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:12 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16964060 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16964060 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16964060 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-09 14:50:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 14:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1227 try 0 launched on executor multi with executor id 1228 parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/878a2737-0235-427b-9d5c-372b0c718b28_isrForCrosstalkSources_855433_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/878a2737-0235-427b-9d5c-372b0c718b28_isrForCrosstalkSources_855433_23.stderr parsl.dataflow.dflow DEBUG: Task 1228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1228 has memoization hash abada59f25577c20e3bd85de6f03a396 parsl.dataflow.memoization INFO: Task 1228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1228 try 0 launched on executor multi with executor id 1229 parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/af4953b5-fc74-4f0e-b26f-8d735356a68b_isrForCrosstalkSources_855391_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/af4953b5-fc74-4f0e-b26f-8d735356a68b_isrForCrosstalkSources_855391_45.stderr parsl.dataflow.dflow DEBUG: Task 1229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1229 has memoization hash 1b64ca2f54749989b6173dce0df34119 parsl.dataflow.memoization INFO: Task 1229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1229 try 0 launched on executor multi with executor id 1230 parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e94b448e-74f3-49b9-bebb-590c056f5126_isrForCrosstalkSources_855398_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e94b448e-74f3-49b9-bebb-590c056f5126_isrForCrosstalkSources_855398_52.stderr parsl.dataflow.dflow DEBUG: Task 1230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1230 has memoization hash 975fae408489f778ba9ad93aca7cde37 parsl.dataflow.memoization INFO: Task 1230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1230 try 0 launched on executor multi with executor id 1231 parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/37b1ab62-3e07-448f-b869-822d1b1d7aa7_isrForCrosstalkSources_855456_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/37b1ab62-3e07-448f-b869-822d1b1d7aa7_isrForCrosstalkSources_855456_42.stderr parsl.dataflow.dflow DEBUG: Task 1231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1231 has memoization hash 7fc060141bfe54b6b73ceebce1c0fca5 parsl.dataflow.memoization INFO: Task 1231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1231 try 0 launched on executor multi with executor id 1232 parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0e016ba9-3800-4ec4-83f5-88969b772915_isrForCrosstalkSources_855403_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0e016ba9-3800-4ec4-83f5-88969b772915_isrForCrosstalkSources_855403_60.stderr parsl.dataflow.dflow DEBUG: Task 1232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1232 has memoization hash 55e06c104b675eb80c69425066d4e4f2 parsl.dataflow.memoization INFO: Task 1232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1232 try 0 launched on executor multi with executor id 1233 parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a8c1046-6c15-4319-9a47-b8a9deec866c_isrForCrosstalkSources_855393_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a8c1046-6c15-4319-9a47-b8a9deec866c_isrForCrosstalkSources_855393_22.stderr parsl.dataflow.dflow DEBUG: Task 1233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1233 has memoization hash f2082e3e90d3ec1407f91f4dd70b06a7 parsl.dataflow.memoization INFO: Task 1233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1233 try 0 launched on executor multi with executor id 1234 parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7b3ad21f-31a5-47b2-928d-39f05841e196_isrForCrosstalkSources_855439_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7b3ad21f-31a5-47b2-928d-39f05841e196_isrForCrosstalkSources_855439_53.stderr parsl.dataflow.dflow DEBUG: Task 1234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1234 has memoization hash a16ce97ef6ad4c16978016c6149e29a8 parsl.dataflow.memoization INFO: Task 1234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1234 try 0 launched on executor multi with executor id 1235 parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/bd4634f3-e64f-4b5e-b664-ac90290ac434_isrForCrosstalkSources_855419_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/bd4634f3-e64f-4b5e-b664-ac90290ac434_isrForCrosstalkSources_855419_51.stderr parsl.dataflow.dflow DEBUG: Task 1235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1235 has memoization hash 7d8b8331b231b7282cbfbc2679ffcfdb parsl.dataflow.memoization INFO: Task 1235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1235 try 0 launched on executor multi with executor id 1236 parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/684a58ff-156b-4ed3-9c35-285da086a8ad_isrForCrosstalkSources_855436_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/684a58ff-156b-4ed3-9c35-285da086a8ad_isrForCrosstalkSources_855436_15.stderr parsl.dataflow.dflow DEBUG: Task 1236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1236 has memoization hash af9ede8186d17fb2c7fffa5c8add067b parsl.dataflow.memoization INFO: Task 1236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1236 try 0 launched on executor multi with executor id 1237 parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/caa0b4f8-ec6f-4e66-935f-cdec40b5ac03_isrForCrosstalkSources_855445_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/caa0b4f8-ec6f-4e66-935f-cdec40b5ac03_isrForCrosstalkSources_855445_1.stderr parsl.dataflow.dflow DEBUG: Task 1237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1237 has memoization hash 7d57e80e3b3e062d7cd2d12c02ae1514 parsl.dataflow.memoization INFO: Task 1237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1237 try 0 launched on executor multi with executor id 1238 parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/9a283e5b-17dd-47fa-b159-044bd9995f77_isrForCrosstalkSources_855417_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/9a283e5b-17dd-47fa-b159-044bd9995f77_isrForCrosstalkSources_855417_37.stderr parsl.dataflow.dflow DEBUG: Task 1238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1238 has memoization hash e3f2fad297082ebce7d64c300f4642b5 parsl.dataflow.memoization INFO: Task 1238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1238 try 0 launched on executor multi with executor id 1239 parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3989f1cc-6bc6-4d77-b871-6cc215c4b3af_isrForCrosstalkSources_855391_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3989f1cc-6bc6-4d77-b871-6cc215c4b3af_isrForCrosstalkSources_855391_26.stderr parsl.dataflow.dflow DEBUG: Task 1239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1239 has memoization hash 4d20c5265a55511dcddae5fc1ea21964 parsl.dataflow.memoization INFO: Task 1239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1239 try 0 launched on executor multi with executor id 1240 parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/87fba31a-90c1-4a64-bd24-0f06218bcc3b_isrForCrosstalkSources_855428_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/87fba31a-90c1-4a64-bd24-0f06218bcc3b_isrForCrosstalkSources_855428_29.stderr parsl.dataflow.dflow DEBUG: Task 1240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1240 has memoization hash 476551efaf90a4d00c66f0cd8052613c parsl.dataflow.memoization INFO: Task 1240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1240 try 0 launched on executor multi with executor id 1241 parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/fb9c569a-fb8f-4806-90c6-20d02f4fdf13_isrForCrosstalkSources_855380_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/fb9c569a-fb8f-4806-90c6-20d02f4fdf13_isrForCrosstalkSources_855380_18.stderr parsl.dataflow.dflow DEBUG: Task 1241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1241 has memoization hash 691d75fe48d11fe8304e55b7f1da3292 parsl.dataflow.memoization INFO: Task 1241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1241 try 0 launched on executor multi with executor id 1242 parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ee7204a3-5ccc-45ef-b4dd-52e1b1216fda_isrForCrosstalkSources_855414_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ee7204a3-5ccc-45ef-b4dd-52e1b1216fda_isrForCrosstalkSources_855414_11.stderr parsl.dataflow.dflow DEBUG: Task 1242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1242 has memoization hash fc8dcbb7ceff68da15fc74ba0f8168a9 parsl.dataflow.memoization INFO: Task 1242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1242 try 0 launched on executor multi with executor id 1243 parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5069b9c3-e25d-482a-b8f9-b6216483de2d_isrForCrosstalkSources_855400_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5069b9c3-e25d-482a-b8f9-b6216483de2d_isrForCrosstalkSources_855400_52.stderr parsl.dataflow.dflow DEBUG: Task 1243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1243 has memoization hash 270f80c5e5e820816bf9288ee8d1264a parsl.dataflow.memoization INFO: Task 1243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1243 try 0 launched on executor multi with executor id 1244 parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/134b8198-6e1c-458b-9c45-d816ac1b7346_isrForCrosstalkSources_855441_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/134b8198-6e1c-458b-9c45-d816ac1b7346_isrForCrosstalkSources_855441_25.stderr parsl.dataflow.dflow DEBUG: Task 1244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1244 has memoization hash fca0526ed7cba5e54e33aec9c5d91830 parsl.dataflow.memoization INFO: Task 1244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1244 try 0 launched on executor multi with executor id 1245 parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4d64aee1-cc78-44c1-9c7d-3706f0857989_isrForCrosstalkSources_855400_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4d64aee1-cc78-44c1-9c7d-3706f0857989_isrForCrosstalkSources_855400_33.stderr parsl.dataflow.dflow DEBUG: Task 1245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1245 has memoization hash 329a55b0c46595ca1076fd3070b72dd4 parsl.dataflow.memoization INFO: Task 1245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1245 try 0 launched on executor multi with executor id 1246 parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cba499f4-7ba6-43ef-8065-dcfda4e70006_isrForCrosstalkSources_855435_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cba499f4-7ba6-43ef-8065-dcfda4e70006_isrForCrosstalkSources_855435_6.stderr parsl.dataflow.dflow DEBUG: Task 1246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1246 has memoization hash ce330b30e3d9bc6e34bbc337d3ff7ee2 parsl.dataflow.memoization INFO: Task 1246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1246 try 0 launched on executor multi with executor id 1247 parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9e8d44ca-693a-4f5b-a5d5-c802c901a4b7_isrForCrosstalkSources_855437_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9e8d44ca-693a-4f5b-a5d5-c802c901a4b7_isrForCrosstalkSources_855437_26.stderr parsl.dataflow.dflow DEBUG: Task 1247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1247 has memoization hash f1cb210ced759c60af30f3e367ae44c5 parsl.dataflow.memoization INFO: Task 1247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1247 try 0 launched on executor multi with executor id 1248 parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c940a931-48da-4e0e-a6a6-d5c7d4e5af15_isrForCrosstalkSources_855391_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c940a931-48da-4e0e-a6a6-d5c7d4e5af15_isrForCrosstalkSources_855391_30.stderr parsl.dataflow.dflow DEBUG: Task 1248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1248 has memoization hash c31ff5afd7e2b0dd85a312a4b168c633 parsl.dataflow.memoization INFO: Task 1248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1248 try 0 launched on executor multi with executor id 1249 parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/11fefe37-4beb-4319-8aa5-5982944206fb_isrForCrosstalkSources_855390_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/11fefe37-4beb-4319-8aa5-5982944206fb_isrForCrosstalkSources_855390_12.stderr parsl.dataflow.dflow DEBUG: Task 1249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1249 has memoization hash 4ac8d07e425f5f7e3d0992727fb3ed67 parsl.dataflow.memoization INFO: Task 1249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1249 try 0 launched on executor multi with executor id 1250 parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b8778b36-6991-4f90-8f98-f25641afa329_isrForCrosstalkSources_855421_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b8778b36-6991-4f90-8f98-f25641afa329_isrForCrosstalkSources_855421_21.stderr parsl.dataflow.dflow DEBUG: Task 1250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1250 has memoization hash b39485797f5cdaba5b9300ee989df993 parsl.dataflow.memoization INFO: Task 1250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1250 try 0 launched on executor multi with executor id 1251 parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8f2b549e-0f28-462e-aad7-5ef2b33e6e4d_isrForCrosstalkSources_855412_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8f2b549e-0f28-462e-aad7-5ef2b33e6e4d_isrForCrosstalkSources_855412_57.stderr parsl.dataflow.dflow DEBUG: Task 1251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1251 has memoization hash 9a91564580602ba592b4595ad9118792 parsl.dataflow.memoization INFO: Task 1251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1251 try 0 launched on executor multi with executor id 1252 parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2e7dd9a7-ed06-4bf7-8e31-63c5860d4031_isrForCrosstalkSources_855408_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2e7dd9a7-ed06-4bf7-8e31-63c5860d4031_isrForCrosstalkSources_855408_2.stderr parsl.dataflow.dflow DEBUG: Task 1252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1252 has memoization hash 26ee661bf66d80a540e8e4a33d84a9a5 parsl.dataflow.memoization INFO: Task 1252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:13 proc_lsst.multi:170 [INFO] [multi] job_id 16964061 proc_lsst.multi INFO: [multi] job_id 16964061 2024-03-09 14:50:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964061 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964061 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16964061 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-09 14:50:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-09 14:50:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1252 try 0 launched on executor multi with executor id 1253 parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d9829a2a-bd89-46e6-b11c-8a2d06cc689e_isrForCrosstalkSources_855430_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d9829a2a-bd89-46e6-b11c-8a2d06cc689e_isrForCrosstalkSources_855430_28.stderr parsl.dataflow.dflow DEBUG: Task 1253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1253 has memoization hash 20deb78122750428266dfa1a0830478d parsl.dataflow.memoization INFO: Task 1253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1253 try 0 launched on executor multi with executor id 1254 parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5e5567ac-b4f1-4c90-a7d4-d09bd8c7bb75_isrForCrosstalkSources_855407_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5e5567ac-b4f1-4c90-a7d4-d09bd8c7bb75_isrForCrosstalkSources_855407_34.stderr parsl.dataflow.dflow DEBUG: Task 1254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1254 has memoization hash 467b91f74e91e974ac99d37ef49c1407 parsl.dataflow.memoization INFO: Task 1254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1254 try 0 launched on executor multi with executor id 1255 parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/91e30c68-cba3-4211-9c30-863901ef8f65_isrForCrosstalkSources_855423_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/91e30c68-cba3-4211-9c30-863901ef8f65_isrForCrosstalkSources_855423_20.stderr parsl.dataflow.dflow DEBUG: Task 1255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1255 has memoization hash 7d6d79eac74fd74810e182ec97196a7b parsl.dataflow.memoization INFO: Task 1255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1255 try 0 launched on executor multi with executor id 1256 parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b1acab89-497c-419d-8195-cb47176a9303_isrForCrosstalkSources_855435_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b1acab89-497c-419d-8195-cb47176a9303_isrForCrosstalkSources_855435_57.stderr parsl.dataflow.dflow DEBUG: Task 1256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1256 has memoization hash df93cd285f0b722cb8cde74e7833e01a parsl.dataflow.memoization INFO: Task 1256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1256 try 0 launched on executor multi with executor id 1257 parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aab08b9-d77b-472a-85ae-688c221b66a7_isrForCrosstalkSources_855407_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aab08b9-d77b-472a-85ae-688c221b66a7_isrForCrosstalkSources_855407_62.stderr parsl.dataflow.dflow DEBUG: Task 1257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1257 has memoization hash 92246350372cb78aa278c828d3acfcf7 parsl.dataflow.memoization INFO: Task 1257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1257 try 0 launched on executor multi with executor id 1258 parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32436abd-1f2b-4e0b-9ad9-5d7c11e52fea_isrForCrosstalkSources_855448_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32436abd-1f2b-4e0b-9ad9-5d7c11e52fea_isrForCrosstalkSources_855448_42.stderr parsl.dataflow.dflow DEBUG: Task 1258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1258 has memoization hash fca8354383ea2d50d5de6b68221551b5 parsl.dataflow.memoization INFO: Task 1258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1258 try 0 launched on executor multi with executor id 1259 parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/435b7919-834e-4815-8b81-06de05c3df97_isrForCrosstalkSources_855406_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/435b7919-834e-4815-8b81-06de05c3df97_isrForCrosstalkSources_855406_62.stderr parsl.dataflow.dflow DEBUG: Task 1259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1259 has memoization hash 590877f57967da9bab95515ea9e28962 parsl.dataflow.memoization INFO: Task 1259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1259 try 0 launched on executor multi with executor id 1260 parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ef3779b7-9f46-46f0-aadb-66770c6e689a_isrForCrosstalkSources_855386_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ef3779b7-9f46-46f0-aadb-66770c6e689a_isrForCrosstalkSources_855386_51.stderr parsl.dataflow.dflow DEBUG: Task 1260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1260 has memoization hash 760bd98144ec8327f2d830ce93ff6cfd parsl.dataflow.memoization INFO: Task 1260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1260 try 0 launched on executor multi with executor id 1261 parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5dba5c03-1145-4e30-810b-b5cf7630aec9_isrForCrosstalkSources_855389_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5dba5c03-1145-4e30-810b-b5cf7630aec9_isrForCrosstalkSources_855389_33.stderr parsl.dataflow.dflow DEBUG: Task 1261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1261 has memoization hash dd0549cf0aac57d572f3a74e2b455e68 parsl.dataflow.memoization INFO: Task 1261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1261 try 0 launched on executor multi with executor id 1262 parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/abe0bc68-b1da-46a5-a17a-558e9c27eb13_isrForCrosstalkSources_855388_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/abe0bc68-b1da-46a5-a17a-558e9c27eb13_isrForCrosstalkSources_855388_61.stderr parsl.dataflow.dflow DEBUG: Task 1262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1262 has memoization hash 7653fd5bf0bb77212d968b8fdaaebbea parsl.dataflow.memoization INFO: Task 1262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1262 try 0 launched on executor multi with executor id 1263 parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4719d837-3dbb-4df0-be0f-a99fe1a9b0d9_isrForCrosstalkSources_855443_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4719d837-3dbb-4df0-be0f-a99fe1a9b0d9_isrForCrosstalkSources_855443_53.stderr parsl.dataflow.dflow DEBUG: Task 1263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1263 has memoization hash d5cc42f481d207ff92af7730af958eb7 parsl.dataflow.memoization INFO: Task 1263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1263 try 0 launched on executor multi with executor id 1264 parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/65b79be9-f467-4450-9eea-7e284bf2fde2_isrForCrosstalkSources_855448_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/65b79be9-f467-4450-9eea-7e284bf2fde2_isrForCrosstalkSources_855448_29.stderr parsl.dataflow.dflow DEBUG: Task 1264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1264 has memoization hash 034d355e74ed77f113593ee881f29e29 parsl.dataflow.memoization INFO: Task 1264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1264 try 0 launched on executor multi with executor id 1265 parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3907cb54-d165-4e79-b5ba-d61788582ca8_isrForCrosstalkSources_855411_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3907cb54-d165-4e79-b5ba-d61788582ca8_isrForCrosstalkSources_855411_9.stderr parsl.dataflow.dflow DEBUG: Task 1265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1265 has memoization hash b1ef4d928b98aa29fc8799c31eeb9d89 parsl.dataflow.memoization INFO: Task 1265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1265 try 0 launched on executor multi with executor id 1266 parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e2013fcb-b0d8-4fd7-898d-2caba0fd2192_isrForCrosstalkSources_855402_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e2013fcb-b0d8-4fd7-898d-2caba0fd2192_isrForCrosstalkSources_855402_42.stderr parsl.dataflow.dflow DEBUG: Task 1266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1266 has memoization hash 62f498704092d656a3535bc8b6850e97 parsl.dataflow.memoization INFO: Task 1266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1266 try 0 launched on executor multi with executor id 1267 parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3ef72b83-df55-4511-ab74-b9c9866af7f3_isrForCrosstalkSources_855427_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3ef72b83-df55-4511-ab74-b9c9866af7f3_isrForCrosstalkSources_855427_11.stderr parsl.dataflow.dflow DEBUG: Task 1267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1267 has memoization hash 10a6122df700ff1af1357b1cd28c5d55 parsl.dataflow.memoization INFO: Task 1267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1267 try 0 launched on executor multi with executor id 1268 parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/50e6e099-4753-4014-90b1-5588a4b81fde_isrForCrosstalkSources_855410_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/50e6e099-4753-4014-90b1-5588a4b81fde_isrForCrosstalkSources_855410_53.stderr parsl.dataflow.dflow DEBUG: Task 1268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1268 has memoization hash 9fbaded75a2fa5d4a21079d02b439d81 parsl.dataflow.memoization INFO: Task 1268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1268 try 0 launched on executor multi with executor id 1269 parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cbb8fb59-c55a-4a0e-b46b-02ff60d65002_isrForCrosstalkSources_855411_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cbb8fb59-c55a-4a0e-b46b-02ff60d65002_isrForCrosstalkSources_855411_6.stderr parsl.dataflow.dflow DEBUG: Task 1269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1269 has memoization hash e3478b1b4c27441e1b226c1442c58578 parsl.dataflow.memoization INFO: Task 1269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1269 try 0 launched on executor multi with executor id 1270 parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9e864875-c95a-4508-9213-cc41c538f83f_isrForCrosstalkSources_855439_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9e864875-c95a-4508-9213-cc41c538f83f_isrForCrosstalkSources_855439_52.stderr parsl.dataflow.dflow DEBUG: Task 1270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1270 has memoization hash fca6c2507c844e0bd147de17e059f30b parsl.dataflow.memoization INFO: Task 1270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1270 try 0 launched on executor multi with executor id 1271 parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b512c51b-8a1a-47f2-bd8d-180980fde0ee_isrForCrosstalkSources_855416_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b512c51b-8a1a-47f2-bd8d-180980fde0ee_isrForCrosstalkSources_855416_18.stderr parsl.dataflow.dflow DEBUG: Task 1271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1271 has memoization hash 5d7aaecaf4086014e83fcb14a40baa3f parsl.dataflow.memoization INFO: Task 1271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1271 try 0 launched on executor multi with executor id 1272 parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a350a02b-c7bf-4e4a-a892-43335800ae36_isrForCrosstalkSources_855452_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a350a02b-c7bf-4e4a-a892-43335800ae36_isrForCrosstalkSources_855452_35.stderr parsl.dataflow.dflow DEBUG: Task 1272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1272 has memoization hash 224bb6feccc05cea1699a5dd5d969c34 parsl.dataflow.memoization INFO: Task 1272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1272 try 0 launched on executor multi with executor id 1273 parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c8bd9403-a3b1-4f7e-9e0b-346dd96f248e_isrForCrosstalkSources_855391_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c8bd9403-a3b1-4f7e-9e0b-346dd96f248e_isrForCrosstalkSources_855391_17.stderr parsl.dataflow.dflow DEBUG: Task 1273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1273 has memoization hash f9cd1f02f05c11c315ad9c7e625b25ca parsl.dataflow.memoization INFO: Task 1273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1273 try 0 launched on executor multi with executor id 1274 parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/695162b8-afdd-4095-a445-282465669bef_isrForCrosstalkSources_855427_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/695162b8-afdd-4095-a445-282465669bef_isrForCrosstalkSources_855427_33.stderr parsl.dataflow.dflow DEBUG: Task 1274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1274 has memoization hash 2fa474cc6d1bb0f334b6e4dd43fff571 parsl.dataflow.memoization INFO: Task 1274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1274 try 0 launched on executor multi with executor id 1275 parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/958208e4-18bf-48ce-9422-bcbd95dd5a1f_isrForCrosstalkSources_855416_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/958208e4-18bf-48ce-9422-bcbd95dd5a1f_isrForCrosstalkSources_855416_32.stderr parsl.dataflow.dflow DEBUG: Task 1275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1275 has memoization hash 78e6a638db57d044188892d4c3171095 parsl.dataflow.memoization INFO: Task 1275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1275 try 0 launched on executor multi with executor id 1276 parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2f3fd13a-428a-4afa-8ef9-4c353ae7c8db_isrForCrosstalkSources_855411_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2f3fd13a-428a-4afa-8ef9-4c353ae7c8db_isrForCrosstalkSources_855411_17.stderr parsl.dataflow.dflow DEBUG: Task 1276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1276 has memoization hash 540e4a9f16d3913d291f08138106f809 parsl.dataflow.memoization INFO: Task 1276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1276 try 0 launched on executor multi with executor id 1277 parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0825954f-f40a-4eaa-acc9-4ad9fe09ab7c_isrForCrosstalkSources_855439_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0825954f-f40a-4eaa-acc9-4ad9fe09ab7c_isrForCrosstalkSources_855439_30.stderr parsl.dataflow.dflow DEBUG: Task 1277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1277 has memoization hash b808380ea4f1c89f8ab8c9a74ce32956 parsl.dataflow.memoization INFO: Task 1277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1277 try 0 launched on executor multi with executor id 1278 parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f05fd4f0-6b1f-4540-adb9-303eb0bcf69c_isrForCrosstalkSources_855396_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f05fd4f0-6b1f-4540-adb9-303eb0bcf69c_isrForCrosstalkSources_855396_49.stderr parsl.dataflow.dflow DEBUG: Task 1278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1278 has memoization hash 2bccde0956314ca1c2a7e8faa9d1ffc1 parsl.dataflow.memoization INFO: Task 1278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1278 try 0 launched on executor multi with executor id 1279 parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d6cc6044-2a7e-4a01-b02e-ddaa17aed0a0_isrForCrosstalkSources_855384_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d6cc6044-2a7e-4a01-b02e-ddaa17aed0a0_isrForCrosstalkSources_855384_59.stderr parsl.dataflow.dflow DEBUG: Task 1279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1279 has memoization hash 734fb6b3dab56eef7070fa2fe2d16919 parsl.dataflow.memoization INFO: Task 1279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1279 try 0 launched on executor multi with executor id 1280 parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1ad3f91-1ee0-41a4-b3d3-6ca93fa41397_isrForCrosstalkSources_855395_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1ad3f91-1ee0-41a4-b3d3-6ca93fa41397_isrForCrosstalkSources_855395_62.stderr parsl.dataflow.dflow DEBUG: Task 1280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1280 has memoization hash da96e29ef240041d0bd664eee71bd8c5 parsl.dataflow.memoization INFO: Task 1280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1280 try 0 launched on executor multi with executor id 1281 parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9d449ebf-d4f4-4ff1-987d-8ca57b0ba138_isrForCrosstalkSources_855441_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9d449ebf-d4f4-4ff1-987d-8ca57b0ba138_isrForCrosstalkSources_855441_12.stderr parsl.dataflow.dflow DEBUG: Task 1281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1281 has memoization hash 4cf62f5bfd6a77653dff9a29c6623f89 parsl.dataflow.memoization INFO: Task 1281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1281 try 0 launched on executor multi with executor id 1282 parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/36240c6f-d100-46aa-af24-cfff02026b8e_isrForCrosstalkSources_855402_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/36240c6f-d100-46aa-af24-cfff02026b8e_isrForCrosstalkSources_855402_58.stderr parsl.dataflow.dflow DEBUG: Task 1282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1282 has memoization hash 8ddc8a2631a8b691f4e30b8049d5a166 parsl.dataflow.memoization INFO: Task 1282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1282 try 0 launched on executor multi with executor id 1283 parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29333641-4b6d-4a47-882a-b308347aceba_isrForCrosstalkSources_855447_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29333641-4b6d-4a47-882a-b308347aceba_isrForCrosstalkSources_855447_7.stderr parsl.dataflow.dflow DEBUG: Task 1283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1283 has memoization hash ae32aab056d801804389aff132542bdf parsl.dataflow.memoization INFO: Task 1283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1283 try 0 launched on executor multi with executor id 1284 parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ad6c8989-4394-4259-9567-b77d43318e0e_isrForCrosstalkSources_855437_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ad6c8989-4394-4259-9567-b77d43318e0e_isrForCrosstalkSources_855437_54.stderr parsl.dataflow.dflow DEBUG: Task 1284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1284 has memoization hash 30a5b5f37c416eb5d1c0d99e1320df0a parsl.dataflow.memoization INFO: Task 1284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1284 try 0 launched on executor multi with executor id 1285 parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5002c368-60c1-4754-90a1-e00547876651_isrForCrosstalkSources_855434_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5002c368-60c1-4754-90a1-e00547876651_isrForCrosstalkSources_855434_16.stderr parsl.dataflow.dflow DEBUG: Task 1285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1285 has memoization hash c821fdbffe62967bc9eba051fb9bac7c parsl.dataflow.memoization INFO: Task 1285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1285 try 0 launched on executor multi with executor id 1286 parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0efc5ed8-72a6-4e6d-a16c-f44afebf2d99_isrForCrosstalkSources_855457_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0efc5ed8-72a6-4e6d-a16c-f44afebf2d99_isrForCrosstalkSources_855457_45.stderr parsl.dataflow.dflow DEBUG: Task 1286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1286 has memoization hash 12ef7d321d2f8718f4063695911d382d parsl.dataflow.memoization INFO: Task 1286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1286 try 0 launched on executor multi with executor id 1287 parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/aa5a47e5-3de3-407f-9ab8-72cf09b19cf1_isrForCrosstalkSources_855398_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/aa5a47e5-3de3-407f-9ab8-72cf09b19cf1_isrForCrosstalkSources_855398_2.stderr parsl.dataflow.dflow DEBUG: Task 1287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1287 has memoization hash dc6f65bfa1ac189f07091096e1781535 parsl.dataflow.memoization INFO: Task 1287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1287 try 0 launched on executor multi with executor id 1288 parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4f1965fc-13be-4b7d-94a7-e331a31fc51c_isrForCrosstalkSources_855413_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4f1965fc-13be-4b7d-94a7-e331a31fc51c_isrForCrosstalkSources_855413_17.stderr parsl.dataflow.dflow DEBUG: Task 1288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1288 has memoization hash dfb1277a01588ebfbfc81101a932ee75 parsl.dataflow.memoization INFO: Task 1288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1288 try 0 launched on executor multi with executor id 1289 parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/9f9c534a-6878-4915-b900-c846a2a85848_isrForCrosstalkSources_855435_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/9f9c534a-6878-4915-b900-c846a2a85848_isrForCrosstalkSources_855435_29.stderr parsl.dataflow.dflow DEBUG: Task 1289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1289 has memoization hash 074e5e72c54854dd22c801d0354863fe parsl.dataflow.memoization INFO: Task 1289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1289 try 0 launched on executor multi with executor id 1290 parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/89499d58-dde5-48fb-83e6-bc7e2e3e592c_isrForCrosstalkSources_855448_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/89499d58-dde5-48fb-83e6-bc7e2e3e592c_isrForCrosstalkSources_855448_2.stderr parsl.dataflow.dflow DEBUG: Task 1290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1290 has memoization hash e4e47769de0327c9bb34101b1c2e6b47 parsl.dataflow.memoization INFO: Task 1290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:14 proc_lsst.multi:170 [INFO] [multi] job_id 16964062 proc_lsst.multi INFO: [multi] job_id 16964062 2024-03-09 14:50:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:50:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964062 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964062 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16964062 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-09 14:50:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1290 try 0 launched on executor multi with executor id 1291 parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/69c811f6-5c4f-409e-b85f-999edfc51c1e_isrForCrosstalkSources_855388_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/69c811f6-5c4f-409e-b85f-999edfc51c1e_isrForCrosstalkSources_855388_17.stderr parsl.dataflow.dflow DEBUG: Task 1291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1291 has memoization hash f922797f58e4bff9570c044dccd1eb98 parsl.dataflow.memoization INFO: Task 1291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1291 try 0 launched on executor multi with executor id 1292 parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/66d4495f-f947-45cd-b601-25f8e60772a8_isrForCrosstalkSources_855423_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/66d4495f-f947-45cd-b601-25f8e60772a8_isrForCrosstalkSources_855423_6.stderr parsl.dataflow.dflow DEBUG: Task 1292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1292 has memoization hash 8e41f72b033614310a51989418da064a parsl.dataflow.memoization INFO: Task 1292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1292 try 0 launched on executor multi with executor id 1293 parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e700dd17-da2d-4931-a63c-419dc8e9d769_isrForCrosstalkSources_855413_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e700dd17-da2d-4931-a63c-419dc8e9d769_isrForCrosstalkSources_855413_46.stderr parsl.dataflow.dflow DEBUG: Task 1293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1293 has memoization hash da09a8f95b40b92c787737af7187af4b parsl.dataflow.memoization INFO: Task 1293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1293 try 0 launched on executor multi with executor id 1294 parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/febf0934-de7f-4dd6-9047-02fd4cb13244_isrForCrosstalkSources_855453_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/febf0934-de7f-4dd6-9047-02fd4cb13244_isrForCrosstalkSources_855453_15.stderr parsl.dataflow.dflow DEBUG: Task 1294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1294 has memoization hash b73cb6c92e3a6ee614fa87fa4f8f7703 parsl.dataflow.memoization INFO: Task 1294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1294 try 0 launched on executor multi with executor id 1295 parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a7e87389-be4c-462f-8cad-bbae3ab9ef96_isrForCrosstalkSources_855453_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a7e87389-be4c-462f-8cad-bbae3ab9ef96_isrForCrosstalkSources_855453_19.stderr parsl.dataflow.dflow DEBUG: Task 1295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1295 has memoization hash e91315702fa3e2d29cfa82c52243fc01 parsl.dataflow.memoization INFO: Task 1295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1295 try 0 launched on executor multi with executor id 1296 parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fc8b863c-f4b3-4a19-91a8-27d1085bd03f_isrForCrosstalkSources_855425_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fc8b863c-f4b3-4a19-91a8-27d1085bd03f_isrForCrosstalkSources_855425_58.stderr parsl.dataflow.dflow DEBUG: Task 1296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1296 has memoization hash 68846659de6255d445a7893baa67e2b1 parsl.dataflow.memoization INFO: Task 1296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1296 try 0 launched on executor multi with executor id 1297 parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b855dccf-8243-406c-ab3d-0632aa139dbf_isrForCrosstalkSources_855453_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b855dccf-8243-406c-ab3d-0632aa139dbf_isrForCrosstalkSources_855453_20.stderr parsl.dataflow.dflow DEBUG: Task 1297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1297 has memoization hash 4f2b45d455e0a402631a22d5a3c603c8 parsl.dataflow.memoization INFO: Task 1297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1297 try 0 launched on executor multi with executor id 1298 parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/20146c4b-1c2a-4229-a9d4-9f3c121e9003_isrForCrosstalkSources_855453_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/20146c4b-1c2a-4229-a9d4-9f3c121e9003_isrForCrosstalkSources_855453_28.stderr parsl.dataflow.dflow DEBUG: Task 1298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1298 has memoization hash 408578cfc88a0dd039833b01b1a39bbb parsl.dataflow.memoization INFO: Task 1298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1298 try 0 launched on executor multi with executor id 1299 parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6f59c259-b387-405b-808d-de0d69819e63_isrForCrosstalkSources_855392_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6f59c259-b387-405b-808d-de0d69819e63_isrForCrosstalkSources_855392_57.stderr parsl.dataflow.dflow DEBUG: Task 1299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1299 has memoization hash 46bba9df017a529be602e73e2eabaeb9 parsl.dataflow.memoization INFO: Task 1299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1299 try 0 launched on executor multi with executor id 1300 parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e678a845-4e12-4bd2-9704-7bc59f58429b_isrForCrosstalkSources_855451_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e678a845-4e12-4bd2-9704-7bc59f58429b_isrForCrosstalkSources_855451_43.stderr parsl.dataflow.dflow DEBUG: Task 1300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1300 has memoization hash 99e69664053b4c9e7cc56e7b0e823033 parsl.dataflow.memoization INFO: Task 1300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1300 try 0 launched on executor multi with executor id 1301 parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/da71700c-0fab-4941-9bb3-d4da42d5d72b_isrForCrosstalkSources_855420_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/da71700c-0fab-4941-9bb3-d4da42d5d72b_isrForCrosstalkSources_855420_59.stderr parsl.dataflow.dflow DEBUG: Task 1301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1301 has memoization hash e54c16bce6dc06d06a931f479c440f19 parsl.dataflow.memoization INFO: Task 1301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1301 try 0 launched on executor multi with executor id 1302 parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08bb7590-ada6-4aac-bb8a-f28688484c0c_isrForCrosstalkSources_855437_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08bb7590-ada6-4aac-bb8a-f28688484c0c_isrForCrosstalkSources_855437_1.stderr parsl.dataflow.dflow DEBUG: Task 1302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1302 has memoization hash 690b565e799c2213d5f69b221667b829 parsl.dataflow.memoization INFO: Task 1302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1302 try 0 launched on executor multi with executor id 1303 parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2225b674-ad77-4c5f-ad33-1337d06c3e71_isrForCrosstalkSources_855449_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2225b674-ad77-4c5f-ad33-1337d06c3e71_isrForCrosstalkSources_855449_18.stderr parsl.dataflow.dflow DEBUG: Task 1303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1303 has memoization hash d67d20a818874eb265bf97950c389c04 parsl.dataflow.memoization INFO: Task 1303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1303 try 0 launched on executor multi with executor id 1304 parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/557bcb82-074d-4c98-8cba-b66d1165bcae_isrForCrosstalkSources_855425_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/557bcb82-074d-4c98-8cba-b66d1165bcae_isrForCrosstalkSources_855425_47.stderr parsl.dataflow.dflow DEBUG: Task 1304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1304 has memoization hash 589c9b4ae4ad357dbbf8bf88a41d19ba parsl.dataflow.memoization INFO: Task 1304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1304 try 0 launched on executor multi with executor id 1305 parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/62c95f0e-802c-4ee0-a27a-858301a32a7d_isrForCrosstalkSources_855430_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/62c95f0e-802c-4ee0-a27a-858301a32a7d_isrForCrosstalkSources_855430_41.stderr parsl.dataflow.dflow DEBUG: Task 1305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1305 has memoization hash 683a8f38b3c518187818139647f3d3f3 parsl.dataflow.memoization INFO: Task 1305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1305 try 0 launched on executor multi with executor id 1306 parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/66b8cc81-7233-4e0d-bbe8-765c558ab114_isrForCrosstalkSources_855429_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/66b8cc81-7233-4e0d-bbe8-765c558ab114_isrForCrosstalkSources_855429_31.stderr parsl.dataflow.dflow DEBUG: Task 1306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1306 has memoization hash de4c4d5cf0e559648f68155ff7db3930 parsl.dataflow.memoization INFO: Task 1306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1306 try 0 launched on executor multi with executor id 1307 parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e479013a-0584-48da-a843-208637635a8e_isrForCrosstalkSources_855394_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e479013a-0584-48da-a843-208637635a8e_isrForCrosstalkSources_855394_42.stderr parsl.dataflow.dflow DEBUG: Task 1307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1307 has memoization hash 19a9bea44c0fff2e7beade2788679385 parsl.dataflow.memoization INFO: Task 1307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1307 try 0 launched on executor multi with executor id 1308 parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6b242225-49dc-43d2-80b3-acb96d5602ec_isrForCrosstalkSources_855401_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6b242225-49dc-43d2-80b3-acb96d5602ec_isrForCrosstalkSources_855401_16.stderr parsl.dataflow.dflow DEBUG: Task 1308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1308 has memoization hash 1f569bf108948972212f022c3986e631 parsl.dataflow.memoization INFO: Task 1308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1308 try 0 launched on executor multi with executor id 1309 parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b347679f-030f-409c-b40f-f987a3839c93_isrForCrosstalkSources_855385_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b347679f-030f-409c-b40f-f987a3839c93_isrForCrosstalkSources_855385_17.stderr parsl.dataflow.dflow DEBUG: Task 1309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1309 has memoization hash fb507540bbb0a93d57ba5c4ae140f54f parsl.dataflow.memoization INFO: Task 1309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1309 try 0 launched on executor multi with executor id 1310 parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/87003133-5e0d-49f4-be9e-f775dddd51e2_isrForCrosstalkSources_855395_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/87003133-5e0d-49f4-be9e-f775dddd51e2_isrForCrosstalkSources_855395_50.stderr parsl.dataflow.dflow DEBUG: Task 1310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1310 has memoization hash ab0f689152f923b711c39c0cc867aba8 parsl.dataflow.memoization INFO: Task 1310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1310 try 0 launched on executor multi with executor id 1311 parsl.dataflow.dflow INFO: Standard output for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1c19a13e-e764-4cab-bf82-0120b9bd3c53_isrForCrosstalkSources_855455_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1c19a13e-e764-4cab-bf82-0120b9bd3c53_isrForCrosstalkSources_855455_19.stderr parsl.dataflow.dflow DEBUG: Task 1311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1311 has memoization hash 5990672721d51aa64e71c8fc8a5575ac parsl.dataflow.memoization INFO: Task 1311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:14 proc_lsst.multi:170 [INFO] [multi] job_id 16964063 proc_lsst.multi INFO: [multi] job_id 16964063 2024-03-09 14:50:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:50:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964063 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964063 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16964063 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-09 14:50:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1311 try 0 launched on executor multi with executor id 1312 parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b976d34a-cd82-420c-8404-1f68db4f912a_isrForCrosstalkSources_855425_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b976d34a-cd82-420c-8404-1f68db4f912a_isrForCrosstalkSources_855425_5.stderr parsl.dataflow.dflow DEBUG: Task 1312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1312 has memoization hash 39203d0b8486e887f01d21a40072898a parsl.dataflow.memoization INFO: Task 1312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1312 try 0 launched on executor multi with executor id 1313 parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e02e7ba3-8aa4-4ae2-a99e-93dab5feca81_isrForCrosstalkSources_855407_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e02e7ba3-8aa4-4ae2-a99e-93dab5feca81_isrForCrosstalkSources_855407_26.stderr parsl.dataflow.dflow DEBUG: Task 1313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1313 has memoization hash e70e920b5a72763613ea5d57a983273a parsl.dataflow.memoization INFO: Task 1313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1313 try 0 launched on executor multi with executor id 1314 parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c7574d87-d8ed-4555-9b6a-79a4a782f6f3_isrForCrosstalkSources_855392_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c7574d87-d8ed-4555-9b6a-79a4a782f6f3_isrForCrosstalkSources_855392_7.stderr parsl.dataflow.dflow DEBUG: Task 1314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1314 has memoization hash 81699f1ed8cb86b0f0c5dead88a758c5 parsl.dataflow.memoization INFO: Task 1314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1314 try 0 launched on executor multi with executor id 1315 parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5cfd9b92-3f22-4387-89de-cb30b0b2057e_isrForCrosstalkSources_855427_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5cfd9b92-3f22-4387-89de-cb30b0b2057e_isrForCrosstalkSources_855427_39.stderr parsl.dataflow.dflow DEBUG: Task 1315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1315 has memoization hash ce203baccf08f176e62989baf4bcf53d parsl.dataflow.memoization INFO: Task 1315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1315 try 0 launched on executor multi with executor id 1316 parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da3ecc14-1e9a-4b80-b5ae-d243d38fc598_isrForCrosstalkSources_855404_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da3ecc14-1e9a-4b80-b5ae-d243d38fc598_isrForCrosstalkSources_855404_62.stderr parsl.dataflow.dflow DEBUG: Task 1316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1316 has memoization hash 04ab7bb3f7fc25d2156635ee65031ef2 parsl.dataflow.memoization INFO: Task 1316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1316 try 0 launched on executor multi with executor id 1317 parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/86c94024-e976-4d32-8d0a-0048321b3825_isrForCrosstalkSources_855381_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/86c94024-e976-4d32-8d0a-0048321b3825_isrForCrosstalkSources_855381_38.stderr parsl.dataflow.dflow DEBUG: Task 1317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1317 has memoization hash 223039c26ef68c93d58c5905fb981df3 parsl.dataflow.memoization INFO: Task 1317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1317 try 0 launched on executor multi with executor id 1318 parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/0b30c9b2-3269-487c-bc31-196ae62addf8_isrForCrosstalkSources_855421_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/0b30c9b2-3269-487c-bc31-196ae62addf8_isrForCrosstalkSources_855421_20.stderr parsl.dataflow.dflow DEBUG: Task 1318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1318 has memoization hash 2b445f04bb24eecc65209224bdbcad7b parsl.dataflow.memoization INFO: Task 1318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1318 try 0 launched on executor multi with executor id 1319 parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/098b044f-9cff-48a1-ab6e-3f64e5a8dcd6_isrForCrosstalkSources_855437_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/098b044f-9cff-48a1-ab6e-3f64e5a8dcd6_isrForCrosstalkSources_855437_2.stderr parsl.dataflow.dflow DEBUG: Task 1319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1319 has memoization hash beae336106f4114d4e6afbbb5dee4fce parsl.dataflow.memoization INFO: Task 1319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1319 try 0 launched on executor multi with executor id 1320 parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c96de887-7906-416a-8580-0064b3231660_isrForCrosstalkSources_855425_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c96de887-7906-416a-8580-0064b3231660_isrForCrosstalkSources_855425_30.stderr parsl.dataflow.dflow DEBUG: Task 1320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1320 has memoization hash 0083101376a9cbe547325a809ac55cd0 parsl.dataflow.memoization INFO: Task 1320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1320 try 0 launched on executor multi with executor id 1321 parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/a60e20fc-f449-4489-b8fa-990b53f07d74_isrForCrosstalkSources_855389_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/a60e20fc-f449-4489-b8fa-990b53f07d74_isrForCrosstalkSources_855389_14.stderr parsl.dataflow.dflow DEBUG: Task 1321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1321 has memoization hash 952fd26600d659852eba2b0555428f40 parsl.dataflow.memoization INFO: Task 1321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1321 try 0 launched on executor multi with executor id 1322 parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/66258046-064e-45d3-834a-cc3974227351_isrForCrosstalkSources_855413_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/66258046-064e-45d3-834a-cc3974227351_isrForCrosstalkSources_855413_8.stderr parsl.dataflow.dflow DEBUG: Task 1322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1322 has memoization hash b666a375291914178646b5fb1e55ca82 parsl.dataflow.memoization INFO: Task 1322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1322 try 0 launched on executor multi with executor id 1323 parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b4f326f9-37af-41a3-bd7c-9b8ead0294c7_isrForCrosstalkSources_855446_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b4f326f9-37af-41a3-bd7c-9b8ead0294c7_isrForCrosstalkSources_855446_56.stderr parsl.dataflow.dflow DEBUG: Task 1323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1323 has memoization hash aa7a9053d5733e8fd79efe67127be21b parsl.dataflow.memoization INFO: Task 1323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1323 try 0 launched on executor multi with executor id 1324 parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53e0f64b-f39b-4fdb-81cc-654da276a321_isrForCrosstalkSources_855389_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53e0f64b-f39b-4fdb-81cc-654da276a321_isrForCrosstalkSources_855389_4.stderr parsl.dataflow.dflow DEBUG: Task 1324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1324 has memoization hash 0475895b4f99d3b0d1ae28f2a99552f1 parsl.dataflow.memoization INFO: Task 1324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1324 try 0 launched on executor multi with executor id 1325 parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8be47b2b-715a-45f2-af8d-8f78dd25f25e_isrForCrosstalkSources_855399_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8be47b2b-715a-45f2-af8d-8f78dd25f25e_isrForCrosstalkSources_855399_17.stderr parsl.dataflow.dflow DEBUG: Task 1325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1325 has memoization hash 7cf0d92ed826bb4d6ad13707e7f940f5 parsl.dataflow.memoization INFO: Task 1325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1325 try 0 launched on executor multi with executor id 1326 parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/66ac8fa6-220a-41ed-a47b-df9b0797929e_isrForCrosstalkSources_855414_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/66ac8fa6-220a-41ed-a47b-df9b0797929e_isrForCrosstalkSources_855414_21.stderr parsl.dataflow.dflow DEBUG: Task 1326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1326 has memoization hash 3c5ca5d1ff0629fa0e992ce0c50ce648 parsl.dataflow.memoization INFO: Task 1326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1326 try 0 launched on executor multi with executor id 1327 parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36360b6b-5220-44ca-ad27-1f524aff7bc5_isrForCrosstalkSources_855388_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36360b6b-5220-44ca-ad27-1f524aff7bc5_isrForCrosstalkSources_855388_12.stderr parsl.dataflow.dflow DEBUG: Task 1327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1327 has memoization hash d27ebf4f6bb2ce6d8b006cef96d7fd23 parsl.dataflow.memoization INFO: Task 1327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1327 try 0 launched on executor multi with executor id 1328 parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5e296d09-2400-4317-81b0-38927d6b8965_isrForCrosstalkSources_855403_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5e296d09-2400-4317-81b0-38927d6b8965_isrForCrosstalkSources_855403_22.stderr parsl.dataflow.dflow DEBUG: Task 1328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1328 has memoization hash 6bf16430c25dda243d1b306b7754705d parsl.dataflow.memoization INFO: Task 1328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1328 try 0 launched on executor multi with executor id 1329 parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/5ee26fa8-0da0-4f33-b24d-ddbdc04ea898_isrForCrosstalkSources_855392_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/5ee26fa8-0da0-4f33-b24d-ddbdc04ea898_isrForCrosstalkSources_855392_3.stderr parsl.dataflow.dflow DEBUG: Task 1329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1329 has memoization hash df43278fcb476eac4f2ba6ebaffbb103 parsl.dataflow.memoization INFO: Task 1329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1329 try 0 launched on executor multi with executor id 1330 parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ec5a2743-9a3b-4a3d-b22d-afe7352b502a_isrForCrosstalkSources_855439_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ec5a2743-9a3b-4a3d-b22d-afe7352b502a_isrForCrosstalkSources_855439_47.stderr parsl.dataflow.dflow DEBUG: Task 1330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1330 has memoization hash e5f12f7f3f7456f37c077eadb382b1e8 parsl.dataflow.memoization INFO: Task 1330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1330 try 0 launched on executor multi with executor id 1331 parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2a7a15f1-d2f9-44c7-bd2a-611dceccfc0e_isrForCrosstalkSources_855403_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2a7a15f1-d2f9-44c7-bd2a-611dceccfc0e_isrForCrosstalkSources_855403_58.stderr parsl.dataflow.dflow DEBUG: Task 1331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1331 has memoization hash 900ca1f7e2b935c87534a796d8c96497 parsl.dataflow.memoization INFO: Task 1331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1331 try 0 launched on executor multi with executor id 1332 parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/c1c93ab4-95f3-4cec-b833-013f84663acc_isrForCrosstalkSources_855407_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/c1c93ab4-95f3-4cec-b833-013f84663acc_isrForCrosstalkSources_855407_61.stderr parsl.dataflow.dflow DEBUG: Task 1332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1332 has memoization hash d9299b5d22bbf5c7d604915b8d7e939e parsl.dataflow.memoization INFO: Task 1332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1332 try 0 launched on executor multi with executor id 1333 parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ac6e2c7a-55bf-4c62-aaa6-a9c329fbbf25_isrForCrosstalkSources_855454_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ac6e2c7a-55bf-4c62-aaa6-a9c329fbbf25_isrForCrosstalkSources_855454_51.stderr parsl.dataflow.dflow DEBUG: Task 1333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1333 has memoization hash 56a9b79e6829280839a6a0cc19eacb73 parsl.dataflow.memoization INFO: Task 1333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1333 try 0 launched on executor multi with executor id 1334 parsl.dataflow.dflow INFO: Standard output for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/a7f6d45f-f725-43c0-ab9a-f35d26368438_isrForCrosstalkSources_855446_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/a7f6d45f-f725-43c0-ab9a-f35d26368438_isrForCrosstalkSources_855446_16.stderr parsl.dataflow.dflow DEBUG: Task 1334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1334 has memoization hash ccab38fa119d1eb31244571c4372b047 parsl.dataflow.memoization INFO: Task 1334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1334 try 0 launched on executor multi with executor id 1335 parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2e0cb3ff-e5a4-4e29-8a9e-3b1f7c9e3116_isrForCrosstalkSources_855447_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2e0cb3ff-e5a4-4e29-8a9e-3b1f7c9e3116_isrForCrosstalkSources_855447_45.stderr parsl.dataflow.dflow DEBUG: Task 1335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1335 has memoization hash bf375c9ffde13854a8f895b7a3e254b9 parsl.dataflow.memoization INFO: Task 1335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1335 try 0 launched on executor multi with executor id 1336 parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/971d7850-438c-453f-915a-ac157922f84e_isrForCrosstalkSources_855399_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/971d7850-438c-453f-915a-ac157922f84e_isrForCrosstalkSources_855399_50.stderr parsl.dataflow.dflow DEBUG: Task 1336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1336 has memoization hash cb46aa299c867909ee25e2808dc5e033 parsl.dataflow.memoization INFO: Task 1336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964064 proc_lsst.multi INFO: [multi] job_id 16964064 2024-03-09 14:50:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:50:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964064 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964064 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16964064 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-09 14:50:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1336 try 0 launched on executor multi with executor id 1337 parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f7da6ec1-b172-45fc-ad93-17f0e034f252_isrForCrosstalkSources_855419_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f7da6ec1-b172-45fc-ad93-17f0e034f252_isrForCrosstalkSources_855419_5.stderr parsl.dataflow.dflow DEBUG: Task 1337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1337 has memoization hash 8afb320335ed1b3ef6c94fec0799968e parsl.dataflow.memoization INFO: Task 1337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaa1ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1337 try 0 launched on executor multi with executor id 1338 parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/62f0b021-4263-4e10-b35b-1e80741ab1e5_isrForCrosstalkSources_855445_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/62f0b021-4263-4e10-b35b-1e80741ab1e5_isrForCrosstalkSources_855445_13.stderr parsl.dataflow.dflow DEBUG: Task 1338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1338 has memoization hash 6e12341fbaff432f035649708ee31045 parsl.dataflow.memoization INFO: Task 1338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1338 try 0 launched on executor multi with executor id 1339 parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/450f2d9a-3fc4-45a2-b381-dcac0725f567_isrForCrosstalkSources_855411_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/450f2d9a-3fc4-45a2-b381-dcac0725f567_isrForCrosstalkSources_855411_26.stderr parsl.dataflow.dflow DEBUG: Task 1339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1339 has memoization hash 23a7cb1ed1c4ec7bf297908ff7354692 parsl.dataflow.memoization INFO: Task 1339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1339 try 0 launched on executor multi with executor id 1340 parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/75a5e783-2258-43c0-ac84-ad827e5d847a_isrForCrosstalkSources_855430_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/75a5e783-2258-43c0-ac84-ad827e5d847a_isrForCrosstalkSources_855430_54.stderr parsl.dataflow.dflow DEBUG: Task 1340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1340 has memoization hash a4b3bf0f3db2339b2702b6f49b2dff1d parsl.dataflow.memoization INFO: Task 1340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1340 try 0 launched on executor multi with executor id 1341 parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2cba4a96-0593-45ee-8440-010b4ac57327_isrForCrosstalkSources_855400_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2cba4a96-0593-45ee-8440-010b4ac57327_isrForCrosstalkSources_855400_41.stderr parsl.dataflow.dflow DEBUG: Task 1341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1341 has memoization hash a68f7a5dfeb2edf38a094dc5da502709 parsl.dataflow.memoization INFO: Task 1341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1341 try 0 launched on executor multi with executor id 1342 parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b5455a3f-abad-4a64-9336-5c37c2124f5f_isrForCrosstalkSources_855411_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b5455a3f-abad-4a64-9336-5c37c2124f5f_isrForCrosstalkSources_855411_48.stderr parsl.dataflow.dflow DEBUG: Task 1342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1342 has memoization hash 0a40d651e09ad2deac00410a7822691e parsl.dataflow.memoization INFO: Task 1342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1342 try 0 launched on executor multi with executor id 1343 parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c7b1fb3f-8b0f-471c-bb43-31c2da8ee99a_isrForCrosstalkSources_855448_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c7b1fb3f-8b0f-471c-bb43-31c2da8ee99a_isrForCrosstalkSources_855448_21.stderr parsl.dataflow.dflow DEBUG: Task 1343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1343 has memoization hash 03a5249d1b06ad6073d076a3de62081f parsl.dataflow.memoization INFO: Task 1343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1343 try 0 launched on executor multi with executor id 1344 parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/32327c0e-04f0-4d3e-a0ab-db0ae59538b2_isrForCrosstalkSources_855439_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/32327c0e-04f0-4d3e-a0ab-db0ae59538b2_isrForCrosstalkSources_855439_20.stderr parsl.dataflow.dflow DEBUG: Task 1344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1344 has memoization hash 9cce492c83618c428d64b4fbe819d59c parsl.dataflow.memoization INFO: Task 1344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1344 try 0 launched on executor multi with executor id 1345 parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bb88b41f-fb98-4025-b945-e221ba11a15a_isrForCrosstalkSources_855420_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bb88b41f-fb98-4025-b945-e221ba11a15a_isrForCrosstalkSources_855420_45.stderr parsl.dataflow.dflow DEBUG: Task 1345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1345 has memoization hash a42649c415d41ba6447f08c604e82c34 parsl.dataflow.memoization INFO: Task 1345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1345 try 0 launched on executor multi with executor id 1346 parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b8e94b91-557e-4253-8892-0b05dd3d0ff2_isrForCrosstalkSources_855459_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b8e94b91-557e-4253-8892-0b05dd3d0ff2_isrForCrosstalkSources_855459_38.stderr parsl.dataflow.dflow DEBUG: Task 1346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1346 has memoization hash f011f76cbeb7f0162e8109addd8977ba parsl.dataflow.memoization INFO: Task 1346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1346 try 0 launched on executor multi with executor id 1347 parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/990b30a9-6bc7-45db-a10a-e0fd10b173c6_isrForCrosstalkSources_855382_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/990b30a9-6bc7-45db-a10a-e0fd10b173c6_isrForCrosstalkSources_855382_48.stderr parsl.dataflow.dflow DEBUG: Task 1347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1347 has memoization hash 085c119dc573374cea21541ef449f98c parsl.dataflow.memoization INFO: Task 1347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1347 try 0 launched on executor multi with executor id 1348 parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3fdf4877-f0b2-48d9-98d2-88de3b89dbf4_isrForCrosstalkSources_855385_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3fdf4877-f0b2-48d9-98d2-88de3b89dbf4_isrForCrosstalkSources_855385_55.stderr parsl.dataflow.dflow DEBUG: Task 1348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1348 has memoization hash e745900141c44ec7aa500fd2fbf8d80f parsl.dataflow.memoization INFO: Task 1348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1348 try 0 launched on executor multi with executor id 1349 parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d634ac82-289c-4924-a14b-212fdd20d983_isrForCrosstalkSources_855389_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d634ac82-289c-4924-a14b-212fdd20d983_isrForCrosstalkSources_855389_51.stderr parsl.dataflow.dflow DEBUG: Task 1349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1349 has memoization hash 932f60fc74ace2ef895cb9cef8edc49e parsl.dataflow.memoization INFO: Task 1349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1349 try 0 launched on executor multi with executor id 1350 parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/4ed6a71c-ecc9-4314-b28d-32dd516f0a28_isrForCrosstalkSources_855407_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/4ed6a71c-ecc9-4314-b28d-32dd516f0a28_isrForCrosstalkSources_855407_10.stderr parsl.dataflow.dflow DEBUG: Task 1350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1350 has memoization hash 403e5387278958e2c63057135a2231f2 parsl.dataflow.memoization INFO: Task 1350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1350 try 0 launched on executor multi with executor id 1351 parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7a51e99e-84d9-42fc-8f64-b2c260b90ae1_isrForCrosstalkSources_855410_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7a51e99e-84d9-42fc-8f64-b2c260b90ae1_isrForCrosstalkSources_855410_21.stderr parsl.dataflow.dflow DEBUG: Task 1351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1351 has memoization hash 45e4b2bf3847b14b61b3954084532c6b parsl.dataflow.memoization INFO: Task 1351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1351 try 0 launched on executor multi with executor id 1352 parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bc0b0985-388d-416d-b211-184dc00d975d_isrForCrosstalkSources_855438_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bc0b0985-388d-416d-b211-184dc00d975d_isrForCrosstalkSources_855438_34.stderr parsl.dataflow.dflow DEBUG: Task 1352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1352 has memoization hash 4388850d38be09dfb9a8911682d97438 parsl.dataflow.memoization INFO: Task 1352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1352 try 0 launched on executor multi with executor id 1353 parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/893f394b-8d19-4c2f-988a-982506614165_isrForCrosstalkSources_855440_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/893f394b-8d19-4c2f-988a-982506614165_isrForCrosstalkSources_855440_2.stderr parsl.dataflow.dflow DEBUG: Task 1353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1353 has memoization hash 7b73f6bb7942783a1f395b779bf87a50 parsl.dataflow.memoization INFO: Task 1353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1353 try 0 launched on executor multi with executor id 1354 parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ae0cf2ba-cce6-414d-853c-ef06a59b1872_isrForCrosstalkSources_855411_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ae0cf2ba-cce6-414d-853c-ef06a59b1872_isrForCrosstalkSources_855411_13.stderr parsl.dataflow.dflow DEBUG: Task 1354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1354 has memoization hash a8bcca8fc5eb4690addef791b4d87792 parsl.dataflow.memoization INFO: Task 1354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1354 try 0 launched on executor multi with executor id 1355 parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a3dd926f-6197-46ee-86ad-03a39fb76b6d_isrForCrosstalkSources_855434_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a3dd926f-6197-46ee-86ad-03a39fb76b6d_isrForCrosstalkSources_855434_56.stderr parsl.dataflow.dflow DEBUG: Task 1355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1355 has memoization hash 164b5c251e509f27699779b204538374 parsl.dataflow.memoization INFO: Task 1355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1355 try 0 launched on executor multi with executor id 1356 parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/41c385a2-5807-417c-972f-0d528d121718_isrForCrosstalkSources_855444_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/41c385a2-5807-417c-972f-0d528d121718_isrForCrosstalkSources_855444_10.stderr parsl.dataflow.dflow DEBUG: Task 1356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1356 has memoization hash 12eb1b0d2999cb89a16066447082dc97 parsl.dataflow.memoization INFO: Task 1356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1356 try 0 launched on executor multi with executor id 1357 parsl.dataflow.dflow INFO: Standard output for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/16ffbede-ca4c-47b1-94bb-26e90584e587_isrForCrosstalkSources_855420_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/16ffbede-ca4c-47b1-94bb-26e90584e587_isrForCrosstalkSources_855420_54.stderr parsl.dataflow.dflow DEBUG: Task 1357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1357 has memoization hash 73082f0226473e4e7421184fe3fd8954 parsl.dataflow.memoization INFO: Task 1357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1357 try 0 launched on executor multi with executor id 1358 parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce4930e6-40aa-46c2-9794-a442c009ecec_isrForCrosstalkSources_855445_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce4930e6-40aa-46c2-9794-a442c009ecec_isrForCrosstalkSources_855445_30.stderr parsl.dataflow.dflow DEBUG: Task 1358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1358 has memoization hash 9d2c3eb7a7728adf974a88d97629dc6a parsl.dataflow.memoization INFO: Task 1358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1358 try 0 launched on executor multi with executor id 1359 parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8ece8090-74a7-4c73-a4b6-6ff6c7abbf75_isrForCrosstalkSources_855453_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8ece8090-74a7-4c73-a4b6-6ff6c7abbf75_isrForCrosstalkSources_855453_14.stderr parsl.dataflow.dflow DEBUG: Task 1359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1359 has memoization hash 5ed88b6ee918f8692e5740a61ad1ea68 parsl.dataflow.memoization INFO: Task 1359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1359 try 0 launched on executor multi with executor id 1360 parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9110c907-f64f-4025-bf22-cddef1f9a1f1_isrForCrosstalkSources_855418_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9110c907-f64f-4025-bf22-cddef1f9a1f1_isrForCrosstalkSources_855418_25.stderr parsl.dataflow.dflow DEBUG: Task 1360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1360 has memoization hash e3e0b32088d493abf0c52fb2d8576654 parsl.dataflow.memoization INFO: Task 1360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1360 try 0 launched on executor multi with executor id 1361 parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f8e392ee-535d-4fe5-aa4f-7a19f73523da_isrForCrosstalkSources_855438_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f8e392ee-535d-4fe5-aa4f-7a19f73523da_isrForCrosstalkSources_855438_60.stderr parsl.dataflow.dflow DEBUG: Task 1361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1361 has memoization hash c58b34d30c9c6de2f3748c4b79787443 parsl.dataflow.memoization INFO: Task 1361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1361 try 0 launched on executor multi with executor id 1362 parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2d4d7d9e-1457-4548-81ff-b5ff576278fb_isrForCrosstalkSources_855379_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2d4d7d9e-1457-4548-81ff-b5ff576278fb_isrForCrosstalkSources_855379_32.stderr parsl.dataflow.dflow DEBUG: Task 1362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1362 has memoization hash f7ee2741fabf885ec65ecec2b63d7685 parsl.dataflow.memoization INFO: Task 1362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1362 try 0 launched on executor multi with executor id 1363 parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/35fc9e0a-95d6-42dd-9eba-186d09ed0c51_isrForCrosstalkSources_855386_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/35fc9e0a-95d6-42dd-9eba-186d09ed0c51_isrForCrosstalkSources_855386_29.stderr parsl.dataflow.dflow DEBUG: Task 1363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1363 has memoization hash 609f2b1d4edf4f1cd7a987d132c8f544 parsl.dataflow.memoization INFO: Task 1363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964065 proc_lsst.multi INFO: [multi] job_id 16964065 2024-03-09 14:50:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964065 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964065 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16964065 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-09 14:50:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1363 try 0 launched on executor multi with executor id 1364 parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9486a6e4-734b-4f06-b620-b29712722af2_isrForCrosstalkSources_855387_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9486a6e4-734b-4f06-b620-b29712722af2_isrForCrosstalkSources_855387_53.stderr parsl.dataflow.dflow DEBUG: Task 1364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1364 has memoization hash 1453f334805afb56c4bdfb26aa421f7c parsl.dataflow.memoization INFO: Task 1364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1364 try 0 launched on executor multi with executor id 1365 parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/d09e157e-6c81-49c2-aa64-b2765fb8550f_isrForCrosstalkSources_855411_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/d09e157e-6c81-49c2-aa64-b2765fb8550f_isrForCrosstalkSources_855411_1.stderr parsl.dataflow.dflow DEBUG: Task 1365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1365 has memoization hash e294bc1bce9043ed3ffdff81fa1bcd7f parsl.dataflow.memoization INFO: Task 1365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1365 try 0 launched on executor multi with executor id 1366 parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a0f2c5e-403d-434d-82f9-fe023a5488c8_isrForCrosstalkSources_855447_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a0f2c5e-403d-434d-82f9-fe023a5488c8_isrForCrosstalkSources_855447_14.stderr parsl.dataflow.dflow DEBUG: Task 1366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1366 has memoization hash f67b123c01eac931a6330877581021bd parsl.dataflow.memoization INFO: Task 1366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1366 try 0 launched on executor multi with executor id 1367 parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/024dac83-2a09-4146-910c-d122fbf29926_isrForCrosstalkSources_855435_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/024dac83-2a09-4146-910c-d122fbf29926_isrForCrosstalkSources_855435_40.stderr parsl.dataflow.dflow DEBUG: Task 1367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1367 has memoization hash 2edb702030135a5c54fb9b4a4209cff1 parsl.dataflow.memoization INFO: Task 1367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1367 try 0 launched on executor multi with executor id 1368 parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a253ab8d-1274-4d48-9ac8-0ba0369238f0_isrForCrosstalkSources_855406_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a253ab8d-1274-4d48-9ac8-0ba0369238f0_isrForCrosstalkSources_855406_58.stderr parsl.dataflow.dflow DEBUG: Task 1368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1368 has memoization hash 45b0797b394ed68d0b6bd6c24de92f8a parsl.dataflow.memoization INFO: Task 1368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1368 try 0 launched on executor multi with executor id 1369 parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f50d54e4-a262-4d58-b46c-397244d0b85c_isrForCrosstalkSources_855449_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f50d54e4-a262-4d58-b46c-397244d0b85c_isrForCrosstalkSources_855449_6.stderr parsl.dataflow.dflow DEBUG: Task 1369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1369 has memoization hash de93359db71808c262083334cb2603d6 parsl.dataflow.memoization INFO: Task 1369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1369 try 0 launched on executor multi with executor id 1370 parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/ce1bc7e1-801a-46b2-b9db-33f603c56d9a_isrForCrosstalkSources_855458_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/ce1bc7e1-801a-46b2-b9db-33f603c56d9a_isrForCrosstalkSources_855458_46.stderr parsl.dataflow.dflow DEBUG: Task 1370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1370 has memoization hash e7ee53bb87916e651484b3572d4c848e parsl.dataflow.memoization INFO: Task 1370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1370 try 0 launched on executor multi with executor id 1371 parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b00a077a-c1be-4c28-b312-8d52de0f41f5_isrForCrosstalkSources_855432_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b00a077a-c1be-4c28-b312-8d52de0f41f5_isrForCrosstalkSources_855432_40.stderr parsl.dataflow.dflow DEBUG: Task 1371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1371 has memoization hash 3df768a111db13a13177de1ca9706f67 parsl.dataflow.memoization INFO: Task 1371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1371 try 0 launched on executor multi with executor id 1372 parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b2330561-5f1a-4f00-ba40-effc69392b36_isrForCrosstalkSources_855408_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b2330561-5f1a-4f00-ba40-effc69392b36_isrForCrosstalkSources_855408_9.stderr parsl.dataflow.dflow DEBUG: Task 1372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1372 has memoization hash 8f140adb3cb847e3788325153ebfd946 parsl.dataflow.memoization INFO: Task 1372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1372 try 0 launched on executor multi with executor id 1373 parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ae3ac075-a6d6-4f80-8bc6-4a7e9e1d273d_isrForCrosstalkSources_855399_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ae3ac075-a6d6-4f80-8bc6-4a7e9e1d273d_isrForCrosstalkSources_855399_26.stderr parsl.dataflow.dflow DEBUG: Task 1373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1373 has memoization hash bc8b8e74aab035069b37fb354af8afc0 parsl.dataflow.memoization INFO: Task 1373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1373 try 0 launched on executor multi with executor id 1374 parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fdbc26c7-6d6f-4cfb-a236-1f1fa25e3b52_isrForCrosstalkSources_855394_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fdbc26c7-6d6f-4cfb-a236-1f1fa25e3b52_isrForCrosstalkSources_855394_35.stderr parsl.dataflow.dflow DEBUG: Task 1374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1374 has memoization hash c74743f7b5bd776b9441d5dd497710db parsl.dataflow.memoization INFO: Task 1374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1374 try 0 launched on executor multi with executor id 1375 parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/4f5f74dc-2d7a-497f-95ca-72cf50425764_isrForCrosstalkSources_855437_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/4f5f74dc-2d7a-497f-95ca-72cf50425764_isrForCrosstalkSources_855437_56.stderr parsl.dataflow.dflow DEBUG: Task 1375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1375 has memoization hash 74144739686562911e675a4b15234cd4 parsl.dataflow.memoization INFO: Task 1375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1375 try 0 launched on executor multi with executor id 1376 parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46f900e5-e42c-44cb-870d-2c5bcf107d08_isrForCrosstalkSources_855448_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46f900e5-e42c-44cb-870d-2c5bcf107d08_isrForCrosstalkSources_855448_19.stderr parsl.dataflow.dflow DEBUG: Task 1376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1376 has memoization hash ef2b2b886b7cf694c21430adb8ae0693 parsl.dataflow.memoization INFO: Task 1376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1376 try 0 launched on executor multi with executor id 1377 parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f40266f1-d0f1-430c-90d6-058063de0ad9_isrForCrosstalkSources_855380_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f40266f1-d0f1-430c-90d6-058063de0ad9_isrForCrosstalkSources_855380_44.stderr parsl.dataflow.dflow DEBUG: Task 1377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1377 has memoization hash 7149e27344b35b367a8d7db77b5acba3 parsl.dataflow.memoization INFO: Task 1377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1377 try 0 launched on executor multi with executor id 1378 parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d2ec1941-9f0e-4d4e-a351-6a862f8638d2_isrForCrosstalkSources_855397_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d2ec1941-9f0e-4d4e-a351-6a862f8638d2_isrForCrosstalkSources_855397_44.stderr parsl.dataflow.dflow DEBUG: Task 1378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1378 has memoization hash f4c4925dcd06fbbe416aace81ce485d4 parsl.dataflow.memoization INFO: Task 1378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1378 try 0 launched on executor multi with executor id 1379 parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4c66839d-9c8f-481a-bfaf-f118dd7d629c_isrForCrosstalkSources_855391_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4c66839d-9c8f-481a-bfaf-f118dd7d629c_isrForCrosstalkSources_855391_39.stderr parsl.dataflow.dflow DEBUG: Task 1379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1379 has memoization hash 6cda824fce314c8292bdff1c60c0e7f2 parsl.dataflow.memoization INFO: Task 1379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1379 try 0 launched on executor multi with executor id 1380 parsl.dataflow.dflow INFO: Standard output for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6b9bac88-2825-45ab-a621-fb2c0fe27619_isrForCrosstalkSources_855452_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6b9bac88-2825-45ab-a621-fb2c0fe27619_isrForCrosstalkSources_855452_58.stderr parsl.dataflow.dflow DEBUG: Task 1380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1380 has memoization hash 5fa69480c85f4d6a2f58595d16e3217f parsl.dataflow.memoization INFO: Task 1380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1380 try 0 launched on executor multi with executor id 1381 parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/32c8a22b-2d08-4da6-8bf2-d2e7a5b13653_isrForCrosstalkSources_855436_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/32c8a22b-2d08-4da6-8bf2-d2e7a5b13653_isrForCrosstalkSources_855436_18.stderr parsl.dataflow.dflow DEBUG: Task 1381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1381 has memoization hash e185057692941c61e1aa0091530198a5 parsl.dataflow.memoization INFO: Task 1381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1381 try 0 launched on executor multi with executor id 1382 parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a8a51654-0177-4d7a-99b8-795615b237a2_isrForCrosstalkSources_855387_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a8a51654-0177-4d7a-99b8-795615b237a2_isrForCrosstalkSources_855387_35.stderr parsl.dataflow.dflow DEBUG: Task 1382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1382 has memoization hash 2d70d010886f9554287d54d6f9efcd47 parsl.dataflow.memoization INFO: Task 1382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1382 try 0 launched on executor multi with executor id 1383 parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/e11bcf25-4766-44c1-b620-a62860267d5c_isrForCrosstalkSources_855440_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/e11bcf25-4766-44c1-b620-a62860267d5c_isrForCrosstalkSources_855440_34.stderr parsl.dataflow.dflow DEBUG: Task 1383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1383 has memoization hash 1d4dadea399f90882acd8f243e80809f parsl.dataflow.memoization INFO: Task 1383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1383 try 0 launched on executor multi with executor id 1384 parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/34e690e0-9ac8-4371-9840-ffa047bf8809_isrForCrosstalkSources_855407_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/34e690e0-9ac8-4371-9840-ffa047bf8809_isrForCrosstalkSources_855407_40.stderr parsl.dataflow.dflow DEBUG: Task 1384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1384 has memoization hash a51e039eb28229999d7c5c6609e02f5e parsl.dataflow.memoization INFO: Task 1384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1384 try 0 launched on executor multi with executor id 1385 parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ab4ffbb6-0541-485d-939c-8677e00ee08e_isrForCrosstalkSources_855431_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ab4ffbb6-0541-485d-939c-8677e00ee08e_isrForCrosstalkSources_855431_41.stderr parsl.dataflow.dflow DEBUG: Task 1385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1385 has memoization hash 5158be2c1cb0a9278d1f7262b3b9936f parsl.dataflow.memoization INFO: Task 1385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1385 try 0 launched on executor multi with executor id 1386 parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2a6dfab8-1e22-4930-82b5-7bf255a4da18_isrForCrosstalkSources_855456_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2a6dfab8-1e22-4930-82b5-7bf255a4da18_isrForCrosstalkSources_855456_31.stderr parsl.dataflow.dflow DEBUG: Task 1386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1386 has memoization hash 1c0a7b067595d6b952de31bfd4d5bc49 parsl.dataflow.memoization INFO: Task 1386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1386 try 0 launched on executor multi with executor id 1387 parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/23c391d2-930e-413b-a7e8-72f22a0799db_isrForCrosstalkSources_855409_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/23c391d2-930e-413b-a7e8-72f22a0799db_isrForCrosstalkSources_855409_54.stderr parsl.dataflow.dflow DEBUG: Task 1387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1387 has memoization hash 81f1548eecc0a023ade9b17e51678c97 parsl.dataflow.memoization INFO: Task 1387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1387 try 0 launched on executor multi with executor id 1388 parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b96cf070-4bfe-4c7b-b4c6-c17e6c1a3fbd_isrForCrosstalkSources_855419_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b96cf070-4bfe-4c7b-b4c6-c17e6c1a3fbd_isrForCrosstalkSources_855419_16.stderr parsl.dataflow.dflow DEBUG: Task 1388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1388 has memoization hash f8d952adcdc67af079ad97d8f28a79f7 parsl.dataflow.memoization INFO: Task 1388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1388 try 0 launched on executor multi with executor id 1389 parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6ec44280-9aaf-433d-bed1-de9ba89909c9_isrForCrosstalkSources_855425_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6ec44280-9aaf-433d-bed1-de9ba89909c9_isrForCrosstalkSources_855425_3.stderr parsl.dataflow.dflow DEBUG: Task 1389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1389 has memoization hash c8e0ddbe042155223e85119583163059 parsl.dataflow.memoization INFO: Task 1389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1389 try 0 launched on executor multi with executor id 1390 parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/94bfd977-d507-41f9-8bf0-21c463ea13e7_isrForCrosstalkSources_855436_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/94bfd977-d507-41f9-8bf0-21c463ea13e7_isrForCrosstalkSources_855436_41.stderr parsl.dataflow.dflow DEBUG: Task 1390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1390 has memoization hash 392e8afd15ea884951416898f52e3b52 parsl.dataflow.memoization INFO: Task 1390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1390 try 0 launched on executor multi with executor id 1391 parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cee2e386-00df-445d-b2e3-0e51b96e80c2_isrForCrosstalkSources_855454_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cee2e386-00df-445d-b2e3-0e51b96e80c2_isrForCrosstalkSources_855454_5.stderr parsl.dataflow.dflow DEBUG: Task 1391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1391 has memoization hash 32e89b0c7a5e871891892ffc3c660066 parsl.dataflow.memoization INFO: Task 1391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1391 try 0 launched on executor multi with executor id 1392 parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/171081e6-b651-4257-a18b-f1fbe6481126_isrForCrosstalkSources_855452_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/171081e6-b651-4257-a18b-f1fbe6481126_isrForCrosstalkSources_855452_27.stderr parsl.dataflow.dflow DEBUG: Task 1392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1392 has memoization hash db179ffa4fcaf1a759361a2282de67a7 parsl.dataflow.memoization INFO: Task 1392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1392 try 0 launched on executor multi with executor id 1393 parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c2f3024b-59e8-46bb-ac91-0d140ca1ede1_isrForCrosstalkSources_855395_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c2f3024b-59e8-46bb-ac91-0d140ca1ede1_isrForCrosstalkSources_855395_40.stderr parsl.dataflow.dflow DEBUG: Task 1393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1393 has memoization hash 86622a357660518c9ad24e78716dc87a parsl.dataflow.memoization INFO: Task 1393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1393 try 0 launched on executor multi with executor id 1394 parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a73b5a88-4d39-4e82-a486-eb5e4dc82fb5_isrForCrosstalkSources_855384_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a73b5a88-4d39-4e82-a486-eb5e4dc82fb5_isrForCrosstalkSources_855384_23.stderr parsl.dataflow.dflow DEBUG: Task 1394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1394 has memoization hash 114960dae12685aab9762ebd54ab1b1e parsl.dataflow.memoization INFO: Task 1394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:16 proc_lsst.multi:170 [INFO] [multi] job_id 16964066 proc_lsst.multi INFO: [multi] job_id 16964066 2024-03-09 14:50:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 14:50:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964066 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964066 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16964066 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-09 14:50:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1394 try 0 launched on executor multi with executor id 1395 parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/dfe31f11-c25a-472a-8f18-ab2f69edea6c_isrForCrosstalkSources_855386_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/dfe31f11-c25a-472a-8f18-ab2f69edea6c_isrForCrosstalkSources_855386_62.stderr parsl.dataflow.dflow DEBUG: Task 1395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1395 has memoization hash de8cfebd20ed5d5abb1e11dc328519b9 parsl.dataflow.memoization INFO: Task 1395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1395 try 0 launched on executor multi with executor id 1396 parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/356131d0-ecdc-4d27-91c7-1bd46bf63aaf_isrForCrosstalkSources_855439_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/356131d0-ecdc-4d27-91c7-1bd46bf63aaf_isrForCrosstalkSources_855439_18.stderr parsl.dataflow.dflow DEBUG: Task 1396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1396 has memoization hash b8b6ac0b81de78eacfde41861f5c148e parsl.dataflow.memoization INFO: Task 1396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1396 try 0 launched on executor multi with executor id 1397 parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/251eff64-b67b-422d-82bb-dd36410d1285_isrForCrosstalkSources_855386_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/251eff64-b67b-422d-82bb-dd36410d1285_isrForCrosstalkSources_855386_14.stderr parsl.dataflow.dflow DEBUG: Task 1397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1397 has memoization hash 87700f0f91f5334c5576ec6fcccf11e1 parsl.dataflow.memoization INFO: Task 1397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1397 try 0 launched on executor multi with executor id 1398 parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5884aea8-df75-4b38-9a13-09f81f1cca22_isrForCrosstalkSources_855400_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5884aea8-df75-4b38-9a13-09f81f1cca22_isrForCrosstalkSources_855400_7.stderr parsl.dataflow.dflow DEBUG: Task 1398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1398 has memoization hash b0ee663ab7736e72e3ea12db54131daf parsl.dataflow.memoization INFO: Task 1398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1398 try 0 launched on executor multi with executor id 1399 parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/df18b367-4ead-498c-9dde-e47849cc576c_isrForCrosstalkSources_855381_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/df18b367-4ead-498c-9dde-e47849cc576c_isrForCrosstalkSources_855381_48.stderr parsl.dataflow.dflow DEBUG: Task 1399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1399 has memoization hash 34e7cb32b0b4e24277731be74ad2c714 parsl.dataflow.memoization INFO: Task 1399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1399 try 0 launched on executor multi with executor id 1400 parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/759741aa-3fb6-4fe8-8efb-2b18188424b1_isrForCrosstalkSources_855380_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/759741aa-3fb6-4fe8-8efb-2b18188424b1_isrForCrosstalkSources_855380_34.stderr parsl.dataflow.dflow DEBUG: Task 1400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1400 has memoization hash 07e1573495c486e63409972bb3803ead parsl.dataflow.memoization INFO: Task 1400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1400 try 0 launched on executor multi with executor id 1401 parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e5357042-1f36-4717-b602-b12d431aeb1f_isrForCrosstalkSources_855394_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e5357042-1f36-4717-b602-b12d431aeb1f_isrForCrosstalkSources_855394_55.stderr parsl.dataflow.dflow DEBUG: Task 1401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1401 has memoization hash 25972dfea27500b4b9ae91af22d06687 parsl.dataflow.memoization INFO: Task 1401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1401 try 0 launched on executor multi with executor id 1402 parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8735d1e3-413e-41ce-b9c8-80ed72829584_isrForCrosstalkSources_855445_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8735d1e3-413e-41ce-b9c8-80ed72829584_isrForCrosstalkSources_855445_11.stderr parsl.dataflow.dflow DEBUG: Task 1402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1402 has memoization hash 8c6ae5c2ff938e10b398bb802332c9c4 parsl.dataflow.memoization INFO: Task 1402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1402 try 0 launched on executor multi with executor id 1403 parsl.dataflow.dflow INFO: Standard output for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5d59860f-5ab8-48d6-ad44-1ec6dab65d51_isrForCrosstalkSources_855396_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5d59860f-5ab8-48d6-ad44-1ec6dab65d51_isrForCrosstalkSources_855396_27.stderr parsl.dataflow.dflow DEBUG: Task 1403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1403 has memoization hash 2729de422831ff3a4709eebb178a4cb2 parsl.dataflow.memoization INFO: Task 1403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1403 try 0 launched on executor multi with executor id 1404 parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/00c0b93c-abeb-491a-b34c-034dc96bdb4d_isrForCrosstalkSources_855389_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/00c0b93c-abeb-491a-b34c-034dc96bdb4d_isrForCrosstalkSources_855389_56.stderr parsl.dataflow.dflow DEBUG: Task 1404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1404 has memoization hash 34cde2d1bb3cf26075fd64c34ef0a146 parsl.dataflow.memoization INFO: Task 1404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1404 try 0 launched on executor multi with executor id 1405 parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/acc540fb-726f-4fc8-b52e-33a2edf5a5b8_isrForCrosstalkSources_855401_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/acc540fb-726f-4fc8-b52e-33a2edf5a5b8_isrForCrosstalkSources_855401_40.stderr parsl.dataflow.dflow DEBUG: Task 1405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1405 has memoization hash 691cb9d422c67202b8604acf28acf5d2 parsl.dataflow.memoization INFO: Task 1405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1405 try 0 launched on executor multi with executor id 1406 parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/28e1eccf-ec8c-406a-ad9f-ead44e5820cb_isrForCrosstalkSources_855443_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/28e1eccf-ec8c-406a-ad9f-ead44e5820cb_isrForCrosstalkSources_855443_46.stderr parsl.dataflow.dflow DEBUG: Task 1406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1406 has memoization hash b057ab0e388eac72094434440f8edbe5 parsl.dataflow.memoization INFO: Task 1406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1406 try 0 launched on executor multi with executor id 1407 parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/600f190f-7e70-4533-b14c-11f4c7347a86_isrForCrosstalkSources_855402_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/600f190f-7e70-4533-b14c-11f4c7347a86_isrForCrosstalkSources_855402_18.stderr parsl.dataflow.dflow DEBUG: Task 1407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1407 has memoization hash b0a8bb4534d7acd9e43bb1d3b013f994 parsl.dataflow.memoization INFO: Task 1407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1407 try 0 launched on executor multi with executor id 1408 parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a772475f-96db-490b-8c76-f81f1fd1dbb9_isrForCrosstalkSources_855407_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a772475f-96db-490b-8c76-f81f1fd1dbb9_isrForCrosstalkSources_855407_15.stderr parsl.dataflow.dflow DEBUG: Task 1408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1408 has memoization hash 1af7b1a8c0238c2bd4e359b1b9bd0ab2 parsl.dataflow.memoization INFO: Task 1408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1408 try 0 launched on executor multi with executor id 1409 parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/606f616d-6a86-4a34-90c7-68440eee528b_isrForCrosstalkSources_855442_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/606f616d-6a86-4a34-90c7-68440eee528b_isrForCrosstalkSources_855442_11.stderr parsl.dataflow.dflow DEBUG: Task 1409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1409 has memoization hash 74f3245c7d8d56576bba9a1abc132fcb parsl.dataflow.memoization INFO: Task 1409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1409 try 0 launched on executor multi with executor id 1410 parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4d47d873-2b85-410b-a79d-3ceee7871bec_isrForCrosstalkSources_855409_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4d47d873-2b85-410b-a79d-3ceee7871bec_isrForCrosstalkSources_855409_62.stderr parsl.dataflow.dflow DEBUG: Task 1410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1410 has memoization hash 2ea967f3c11115dc46df2afc9308e56c parsl.dataflow.memoization INFO: Task 1410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1410 try 0 launched on executor multi with executor id 1411 parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9a51e723-3b71-4c2c-9a43-abba468e42d1_isrForCrosstalkSources_855396_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9a51e723-3b71-4c2c-9a43-abba468e42d1_isrForCrosstalkSources_855396_23.stderr parsl.dataflow.dflow DEBUG: Task 1411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1411 has memoization hash c3d16f3547f809adc28713df7aceeb54 parsl.dataflow.memoization INFO: Task 1411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1411 try 0 launched on executor multi with executor id 1412 parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/93ab04a7-da49-442c-aa83-4a20284602d7_isrForCrosstalkSources_855425_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/93ab04a7-da49-442c-aa83-4a20284602d7_isrForCrosstalkSources_855425_40.stderr parsl.dataflow.dflow DEBUG: Task 1412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1412 has memoization hash f12f37ff06b6fab9856cff04131af120 parsl.dataflow.memoization INFO: Task 1412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1412 try 0 launched on executor multi with executor id 1413 parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2356c3a5-39e7-40f0-b1a4-e7f6993e1cd4_isrForCrosstalkSources_855441_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2356c3a5-39e7-40f0-b1a4-e7f6993e1cd4_isrForCrosstalkSources_855441_35.stderr parsl.dataflow.dflow DEBUG: Task 1413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1413 has memoization hash 9edd5ab7782316ea6d99e66d33359fef parsl.dataflow.memoization INFO: Task 1413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1413 try 0 launched on executor multi with executor id 1414 parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/400b5e4e-c33f-4cf5-bf95-06cb7b0fbca5_isrForCrosstalkSources_855386_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/400b5e4e-c33f-4cf5-bf95-06cb7b0fbca5_isrForCrosstalkSources_855386_10.stderr parsl.dataflow.dflow DEBUG: Task 1414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1414 has memoization hash 7fc459cc93ece955470aeb23b96d982e parsl.dataflow.memoization INFO: Task 1414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1414 try 0 launched on executor multi with executor id 1415 parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9baa78df-503a-4ae8-8662-e201e750ccfe_isrForCrosstalkSources_855449_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9baa78df-503a-4ae8-8662-e201e750ccfe_isrForCrosstalkSources_855449_13.stderr parsl.dataflow.dflow DEBUG: Task 1415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1415 has memoization hash 6b44687a8ab8fa5c70d112beb2e66841 parsl.dataflow.memoization INFO: Task 1415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1415 try 0 launched on executor multi with executor id 1416 parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f6ebd18a-38a4-4a87-af22-32ccd1294fef_isrForCrosstalkSources_855418_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f6ebd18a-38a4-4a87-af22-32ccd1294fef_isrForCrosstalkSources_855418_55.stderr parsl.dataflow.dflow DEBUG: Task 1416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1416 has memoization hash f6221bd606b80e2ed815027fa8bbb59f parsl.dataflow.memoization INFO: Task 1416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1416 try 0 launched on executor multi with executor id 1417 parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dc40f4f8-50a5-4837-90e3-726ede50627b_isrForCrosstalkSources_855396_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dc40f4f8-50a5-4837-90e3-726ede50627b_isrForCrosstalkSources_855396_30.stderr parsl.dataflow.dflow DEBUG: Task 1417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1417 has memoization hash 2e747b594aa4d6390e23576cdafff3d9 parsl.dataflow.memoization INFO: Task 1417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1417 try 0 launched on executor multi with executor id 1418 parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cac2c76-d4e5-4730-8f4a-e352609beae2_isrForCrosstalkSources_855403_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cac2c76-d4e5-4730-8f4a-e352609beae2_isrForCrosstalkSources_855403_15.stderr parsl.dataflow.dflow DEBUG: Task 1418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1418 has memoization hash 67321dfb902375129886d36e75172954 parsl.dataflow.memoization INFO: Task 1418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1418 try 0 launched on executor multi with executor id 1419 parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/efc63e9b-12b0-46cb-921a-f4925213edec_isrForCrosstalkSources_855389_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/efc63e9b-12b0-46cb-921a-f4925213edec_isrForCrosstalkSources_855389_10.stderr parsl.dataflow.dflow DEBUG: Task 1419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1419 has memoization hash 1118e4c4f9f0a1a3f2c2c45f4ee8745b parsl.dataflow.memoization INFO: Task 1419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:16 proc_lsst.multi:170 [INFO] [multi] job_id 16964067 proc_lsst.multi INFO: [multi] job_id 16964067 2024-03-09 14:50:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 14:50:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964067 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964067 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16964067 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-09 14:50:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 14:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1419 try 0 launched on executor multi with executor id 1420 parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6630b181-bca6-4be3-9816-adc8cff6c554_isrForCrosstalkSources_855420_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6630b181-bca6-4be3-9816-adc8cff6c554_isrForCrosstalkSources_855420_30.stderr parsl.dataflow.dflow DEBUG: Task 1420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1420 has memoization hash 006904b1ff6c63589c2538e74360bab0 parsl.dataflow.memoization INFO: Task 1420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1420 try 0 launched on executor multi with executor id 1421 parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5bf1cd5d-a356-4f76-83fa-cf8590ac28d3_isrForCrosstalkSources_855456_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5bf1cd5d-a356-4f76-83fa-cf8590ac28d3_isrForCrosstalkSources_855456_8.stderr parsl.dataflow.dflow DEBUG: Task 1421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1421 has memoization hash 22fb50433cc1406e44f36140a5da31b3 parsl.dataflow.memoization INFO: Task 1421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1421 try 0 launched on executor multi with executor id 1422 parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0c8e370f-9ec6-4c4c-923e-aed0c012c23b_isrForCrosstalkSources_855451_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0c8e370f-9ec6-4c4c-923e-aed0c012c23b_isrForCrosstalkSources_855451_40.stderr parsl.dataflow.dflow DEBUG: Task 1422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1422 has memoization hash 84b3bf9247852ce20c492abf7e6d4a5b parsl.dataflow.memoization INFO: Task 1422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459b5b4e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1422 try 0 launched on executor multi with executor id 1423 parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f645d75c-7098-4478-8869-f1d7eb3d00a7_isrForCrosstalkSources_855436_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f645d75c-7098-4478-8869-f1d7eb3d00a7_isrForCrosstalkSources_855436_8.stderr parsl.dataflow.dflow DEBUG: Task 1423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1423 has memoization hash 130a6e835674bcea371b1f55e1b3e99b parsl.dataflow.memoization INFO: Task 1423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1423 try 0 launched on executor multi with executor id 1424 parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b84b73d7-1ca7-4e10-9507-fb4f9b0733f6_isrForCrosstalkSources_855442_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b84b73d7-1ca7-4e10-9507-fb4f9b0733f6_isrForCrosstalkSources_855442_53.stderr parsl.dataflow.dflow DEBUG: Task 1424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1424 has memoization hash 051d0ccae0497d50b76ec03ba03f8a41 parsl.dataflow.memoization INFO: Task 1424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1424 try 0 launched on executor multi with executor id 1425 parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cc40cc80-3120-49e3-a589-66017eb1d55e_isrForCrosstalkSources_855458_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cc40cc80-3120-49e3-a589-66017eb1d55e_isrForCrosstalkSources_855458_27.stderr parsl.dataflow.dflow DEBUG: Task 1425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1425 has memoization hash 8595ff973276d4a2e0ec848f35c35295 parsl.dataflow.memoization INFO: Task 1425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1425 try 0 launched on executor multi with executor id 1426 parsl.dataflow.dflow INFO: Standard output for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7f309441-ad53-4424-a701-48f407a9036e_isrForCrosstalkSources_855437_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7f309441-ad53-4424-a701-48f407a9036e_isrForCrosstalkSources_855437_31.stderr parsl.dataflow.dflow DEBUG: Task 1426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1426 has memoization hash d115ad7fd2cb36cae56d3cf5d36146a3 parsl.dataflow.memoization INFO: Task 1426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1426 try 0 launched on executor multi with executor id 1427 parsl.dataflow.dflow INFO: Standard output for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3d4d3f2d-12d2-4e7e-837d-cde10da9b0b6_isrForCrosstalkSources_855381_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3d4d3f2d-12d2-4e7e-837d-cde10da9b0b6_isrForCrosstalkSources_855381_57.stderr parsl.dataflow.dflow DEBUG: Task 1427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1427 has memoization hash 871f8e0127dd7dfbdd73093913cfc848 parsl.dataflow.memoization INFO: Task 1427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1427 try 0 launched on executor multi with executor id 1428 parsl.dataflow.dflow INFO: Standard output for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e8d18879-482a-4811-81f2-c3facf94e3f4_isrForCrosstalkSources_855380_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e8d18879-482a-4811-81f2-c3facf94e3f4_isrForCrosstalkSources_855380_29.stderr parsl.dataflow.dflow DEBUG: Task 1428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1428 has memoization hash 3006386aae5da76f3c83243ec7812e33 parsl.dataflow.memoization INFO: Task 1428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1428 try 0 launched on executor multi with executor id 1429 parsl.dataflow.dflow INFO: Standard output for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c115c1eb-43c4-483e-9e9a-5fc2139de29d_isrForCrosstalkSources_855451_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c115c1eb-43c4-483e-9e9a-5fc2139de29d_isrForCrosstalkSources_855451_26.stderr parsl.dataflow.dflow DEBUG: Task 1429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1429 has memoization hash ead8dad7f78166ba6ba706a7c8864f5d parsl.dataflow.memoization INFO: Task 1429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1429 try 0 launched on executor multi with executor id 1430 parsl.dataflow.dflow INFO: Standard output for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/d913ab73-bc61-4571-9c33-2068ced97910_isrForCrosstalkSources_855404_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/d913ab73-bc61-4571-9c33-2068ced97910_isrForCrosstalkSources_855404_8.stderr parsl.dataflow.dflow DEBUG: Task 1430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1430 has memoization hash 93515faeced4ea13ee507ec92120cddb parsl.dataflow.memoization INFO: Task 1430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1430 try 0 launched on executor multi with executor id 1431 parsl.dataflow.dflow INFO: Standard output for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/76bc1419-31dc-4366-8f01-0a699dc98987_isrForCrosstalkSources_855387_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/76bc1419-31dc-4366-8f01-0a699dc98987_isrForCrosstalkSources_855387_52.stderr parsl.dataflow.dflow DEBUG: Task 1431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1431 has memoization hash 119b5588753bd15df0d3fc7f05232383 parsl.dataflow.memoization INFO: Task 1431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1431 try 0 launched on executor multi with executor id 1432 parsl.dataflow.dflow INFO: Standard output for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/983ad80a-7048-442a-98d7-cd0a18558365_isrForCrosstalkSources_855457_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/983ad80a-7048-442a-98d7-cd0a18558365_isrForCrosstalkSources_855457_9.stderr parsl.dataflow.dflow DEBUG: Task 1432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1432 has memoization hash 4578bdeafc277f495bbdf8490cd734a4 parsl.dataflow.memoization INFO: Task 1432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1432 try 0 launched on executor multi with executor id 1433 parsl.dataflow.dflow INFO: Standard output for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/75765d28-45bb-4f75-989e-b660e83ba923_isrForCrosstalkSources_855436_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/75765d28-45bb-4f75-989e-b660e83ba923_isrForCrosstalkSources_855436_49.stderr parsl.dataflow.dflow DEBUG: Task 1433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1433 has memoization hash e4a4b911215198fabd558d49d3c54183 parsl.dataflow.memoization INFO: Task 1433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1433 try 0 launched on executor multi with executor id 1434 parsl.dataflow.dflow INFO: Standard output for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e65e8cde-ee91-4b49-9c36-73865d45ca68_isrForCrosstalkSources_855428_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e65e8cde-ee91-4b49-9c36-73865d45ca68_isrForCrosstalkSources_855428_53.stderr parsl.dataflow.dflow DEBUG: Task 1434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1434 has memoization hash c99f58a83fbed7719dc9be56f68c7c72 parsl.dataflow.memoization INFO: Task 1434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1434 try 0 launched on executor multi with executor id 1435 parsl.dataflow.dflow INFO: Standard output for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d21e8ebc-3c9d-4f19-8e1c-6be54468abee_isrForCrosstalkSources_855414_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d21e8ebc-3c9d-4f19-8e1c-6be54468abee_isrForCrosstalkSources_855414_39.stderr parsl.dataflow.dflow DEBUG: Task 1435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1435 has memoization hash 1cf89bf395d67208490e4e37c36c758e parsl.dataflow.memoization INFO: Task 1435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1435 try 0 launched on executor multi with executor id 1436 parsl.dataflow.dflow INFO: Standard output for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4a3d4e85-6951-475b-9879-b5d839a36a05_isrForCrosstalkSources_855393_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4a3d4e85-6951-475b-9879-b5d839a36a05_isrForCrosstalkSources_855393_30.stderr parsl.dataflow.dflow DEBUG: Task 1436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1436 has memoization hash 7886a76b13524a09f56bb17af7e68320 parsl.dataflow.memoization INFO: Task 1436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1436 try 0 launched on executor multi with executor id 1437 parsl.dataflow.dflow INFO: Standard output for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2af8eb3f-0fe0-446d-9bcc-fea862240180_isrForCrosstalkSources_855443_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2af8eb3f-0fe0-446d-9bcc-fea862240180_isrForCrosstalkSources_855443_42.stderr parsl.dataflow.dflow DEBUG: Task 1437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1437 has memoization hash a57ba0a6d2a825438d9710afe2ff8e1f parsl.dataflow.memoization INFO: Task 1437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1437 try 0 launched on executor multi with executor id 1438 parsl.dataflow.dflow INFO: Standard output for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cb745ba9-2227-4705-8172-4d0411986ca1_isrForCrosstalkSources_855387_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cb745ba9-2227-4705-8172-4d0411986ca1_isrForCrosstalkSources_855387_57.stderr parsl.dataflow.dflow DEBUG: Task 1438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1438 has memoization hash eeab1534aea73e34fa361e8c85c1dade parsl.dataflow.memoization INFO: Task 1438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1438 try 0 launched on executor multi with executor id 1439 parsl.dataflow.dflow INFO: Standard output for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eae4c1d2-35ef-454a-8bf6-f0bf4b51fa98_isrForCrosstalkSources_855396_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eae4c1d2-35ef-454a-8bf6-f0bf4b51fa98_isrForCrosstalkSources_855396_13.stderr parsl.dataflow.dflow DEBUG: Task 1439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1439 has memoization hash 01696d8c1a6febf9acaac1dcdd6fcaf9 parsl.dataflow.memoization INFO: Task 1439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1439 try 0 launched on executor multi with executor id 1440 parsl.dataflow.dflow INFO: Standard output for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e59df41b-5496-4527-8617-38ebc59b7bb2_isrForCrosstalkSources_855454_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e59df41b-5496-4527-8617-38ebc59b7bb2_isrForCrosstalkSources_855454_42.stderr parsl.dataflow.dflow DEBUG: Task 1440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1440 has memoization hash f2f379f91017a2112cbfe2fd5c5412b7 parsl.dataflow.memoization INFO: Task 1440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1440 try 0 launched on executor multi with executor id 1441 parsl.dataflow.dflow INFO: Standard output for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8a608126-3bfa-46b0-a30a-180fa0366f83_isrForCrosstalkSources_855440_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8a608126-3bfa-46b0-a30a-180fa0366f83_isrForCrosstalkSources_855440_10.stderr parsl.dataflow.dflow DEBUG: Task 1441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1441 has memoization hash db5355f8ad97b40d11e0d970c6965260 parsl.dataflow.memoization INFO: Task 1441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1441 try 0 launched on executor multi with executor id 1442 parsl.dataflow.dflow INFO: Standard output for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/140f7a41-1471-44fd-9014-37e161c8fd4b_isrForCrosstalkSources_855458_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/140f7a41-1471-44fd-9014-37e161c8fd4b_isrForCrosstalkSources_855458_18.stderr parsl.dataflow.dflow DEBUG: Task 1442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1442 has memoization hash 2fa345f00d82f17e05eddbe9be95df09 parsl.dataflow.memoization INFO: Task 1442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1442 try 0 launched on executor multi with executor id 1443 parsl.dataflow.dflow INFO: Standard output for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/888130f5-e922-48a3-84fe-cc92d9266001_isrForCrosstalkSources_855385_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/888130f5-e922-48a3-84fe-cc92d9266001_isrForCrosstalkSources_855385_19.stderr parsl.dataflow.dflow DEBUG: Task 1443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1443 has memoization hash 9a82142d1bf325e8540b23c58a45db0d parsl.dataflow.memoization INFO: Task 1443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1443 try 0 launched on executor multi with executor id 1444 parsl.dataflow.dflow INFO: Standard output for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/8a5de12b-16e7-475b-8721-38ef653f6a41_isrForCrosstalkSources_855420_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/8a5de12b-16e7-475b-8721-38ef653f6a41_isrForCrosstalkSources_855420_13.stderr parsl.dataflow.dflow DEBUG: Task 1444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1444 has memoization hash 0e8a7afbe9f10876b41c65c6b1bc90c1 parsl.dataflow.memoization INFO: Task 1444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1444 try 0 launched on executor multi with executor id 1445 parsl.dataflow.dflow INFO: Standard output for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1214e390-964b-4c48-8221-44f8dbba4300_isrForCrosstalkSources_855427_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1214e390-964b-4c48-8221-44f8dbba4300_isrForCrosstalkSources_855427_29.stderr parsl.dataflow.dflow DEBUG: Task 1445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1445 has memoization hash 53abf23bcf121e3012c170fe0cd78de4 parsl.dataflow.memoization INFO: Task 1445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1445 try 0 launched on executor multi with executor id 1446 parsl.dataflow.dflow INFO: Standard output for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0f194849-81fb-4e67-95cc-e1d7ec72b214_isrForCrosstalkSources_855445_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0f194849-81fb-4e67-95cc-e1d7ec72b214_isrForCrosstalkSources_855445_51.stderr parsl.dataflow.dflow DEBUG: Task 1446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1446 has memoization hash 00189ad4317562cd7487e0e62c7df7ab parsl.dataflow.memoization INFO: Task 1446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1446 try 0 launched on executor multi with executor id 1447 parsl.dataflow.dflow INFO: Standard output for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bb54b1ea-8904-4286-a9f5-08062e5590f4_isrForCrosstalkSources_855450_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bb54b1ea-8904-4286-a9f5-08062e5590f4_isrForCrosstalkSources_855450_15.stderr parsl.dataflow.dflow DEBUG: Task 1447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1447 has memoization hash 6cc8db8fe8df21449092e9f3823996a1 parsl.dataflow.memoization INFO: Task 1447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1447 try 0 launched on executor multi with executor id 1448 parsl.dataflow.dflow INFO: Standard output for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3f95a577-8ca7-4760-b556-53782ba8ea44_isrForCrosstalkSources_855457_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3f95a577-8ca7-4760-b556-53782ba8ea44_isrForCrosstalkSources_855457_33.stderr parsl.dataflow.dflow DEBUG: Task 1448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1448 has memoization hash 677d2e2fa0f7036b15b4cd8b45c0910d parsl.dataflow.memoization INFO: Task 1448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1448 try 0 launched on executor multi with executor id 1449 parsl.dataflow.dflow INFO: Standard output for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f1f48ae9-6994-4fde-b1ec-553bc39ca626_isrForCrosstalkSources_855417_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f1f48ae9-6994-4fde-b1ec-553bc39ca626_isrForCrosstalkSources_855417_47.stderr parsl.dataflow.dflow DEBUG: Task 1449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1449 has memoization hash 73fb52b0319f9bd4fe1685e21145bd4c parsl.dataflow.memoization INFO: Task 1449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1449 try 0 launched on executor multi with executor id 1450 parsl.dataflow.dflow INFO: Standard output for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a4133cd7-e61b-441a-86b5-ab667ecf0460_isrForCrosstalkSources_855409_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a4133cd7-e61b-441a-86b5-ab667ecf0460_isrForCrosstalkSources_855409_19.stderr parsl.dataflow.dflow DEBUG: Task 1450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1450 has memoization hash d10791a91b5379e7b2165e6d63a34e92 parsl.dataflow.memoization INFO: Task 1450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1450 try 0 launched on executor multi with executor id 1451 parsl.dataflow.dflow INFO: Standard output for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/24af258a-ea16-4be9-88b1-6414e946c60b_isrForCrosstalkSources_855443_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/24af258a-ea16-4be9-88b1-6414e946c60b_isrForCrosstalkSources_855443_35.stderr parsl.dataflow.dflow DEBUG: Task 1451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1451 has memoization hash 2280b578ac482229cc1544421e4c1a9f parsl.dataflow.memoization INFO: Task 1451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:17 proc_lsst.multi:170 [INFO] [multi] job_id 16964068 proc_lsst.multi INFO: [multi] job_id 16964068 2024-03-09 14:50:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 14:50:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964068 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964068 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16964068 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-09 14:50:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-09 14:50:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 14:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1451 try 0 launched on executor multi with executor id 1452 parsl.dataflow.dflow INFO: Standard output for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed45015e-8c66-45be-9117-f5197f167735_isrForCrosstalkSources_855419_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed45015e-8c66-45be-9117-f5197f167735_isrForCrosstalkSources_855419_17.stderr parsl.dataflow.dflow DEBUG: Task 1452 will be sent to executor multi parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1452 has memoization hash 81e52c04891f783049686c41d800e06d parsl.dataflow.memoization INFO: Task 1452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1452 try 0 launched on executor multi with executor id 1453 parsl.dataflow.dflow INFO: Standard output for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/589f6c3b-0f80-47aa-ab7f-7662f52567fb_isrForCrosstalkSources_855436_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/589f6c3b-0f80-47aa-ab7f-7662f52567fb_isrForCrosstalkSources_855436_40.stderr parsl.dataflow.dflow DEBUG: Task 1453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1453 has memoization hash bdb35f525966c4641c51f184b485f877 parsl.dataflow.memoization INFO: Task 1453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1453 try 0 launched on executor multi with executor id 1454 parsl.dataflow.dflow INFO: Standard output for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/964831aa-44bb-4e6d-b852-2a7b2f4631d4_isrForCrosstalkSources_855435_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/964831aa-44bb-4e6d-b852-2a7b2f4631d4_isrForCrosstalkSources_855435_16.stderr parsl.dataflow.dflow DEBUG: Task 1454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1454 has memoization hash f04ffa8615e24cc65ca6b5313c6ccbf0 parsl.dataflow.memoization INFO: Task 1454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1454 try 0 launched on executor multi with executor id 1455 parsl.dataflow.dflow INFO: Standard output for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/4d1af1f6-4db0-452e-8619-91fdbdcc7be1_isrForCrosstalkSources_855455_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/4d1af1f6-4db0-452e-8619-91fdbdcc7be1_isrForCrosstalkSources_855455_30.stderr parsl.dataflow.dflow DEBUG: Task 1455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1455 has memoization hash d9ba81aa59e74133efc810069a12471f parsl.dataflow.memoization INFO: Task 1455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1455 try 0 launched on executor multi with executor id 1456 parsl.dataflow.dflow INFO: Standard output for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/aaf9f718-1e71-464c-9a3f-dd5e84ff72e4_isrForCrosstalkSources_855400_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/aaf9f718-1e71-464c-9a3f-dd5e84ff72e4_isrForCrosstalkSources_855400_11.stderr parsl.dataflow.dflow DEBUG: Task 1456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1456 has memoization hash b4b3e17614fca04deeda2a6970434cd5 parsl.dataflow.memoization INFO: Task 1456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1456 try 0 launched on executor multi with executor id 1457 parsl.dataflow.dflow INFO: Standard output for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a03d39b8-26ce-4462-8e2f-51b4721f0239_isrForCrosstalkSources_855396_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a03d39b8-26ce-4462-8e2f-51b4721f0239_isrForCrosstalkSources_855396_40.stderr parsl.dataflow.dflow DEBUG: Task 1457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1457 has memoization hash e2ba32d6b8505720511def9e79330b74 parsl.dataflow.memoization INFO: Task 1457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1457 try 0 launched on executor multi with executor id 1458 parsl.dataflow.dflow INFO: Standard output for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/eff26f19-9cc3-4184-bf73-8503e5981914_isrForCrosstalkSources_855439_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/eff26f19-9cc3-4184-bf73-8503e5981914_isrForCrosstalkSources_855439_6.stderr parsl.dataflow.dflow DEBUG: Task 1458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1458 has memoization hash c078c5b4ab229af46fa5b92523df073a parsl.dataflow.memoization INFO: Task 1458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1458 try 0 launched on executor multi with executor id 1459 parsl.dataflow.dflow INFO: Standard output for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/90e610cc-7633-4ae1-85d3-fa29e122ce4b_isrForCrosstalkSources_855388_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/90e610cc-7633-4ae1-85d3-fa29e122ce4b_isrForCrosstalkSources_855388_46.stderr parsl.dataflow.dflow DEBUG: Task 1459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1459 has memoization hash 8d2ce938022daf6c847446038837af84 parsl.dataflow.memoization INFO: Task 1459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1459 try 0 launched on executor multi with executor id 1460 parsl.dataflow.dflow INFO: Standard output for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e3e4fe03-cdc9-478f-b625-1eab1095e8c8_isrForCrosstalkSources_855424_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e3e4fe03-cdc9-478f-b625-1eab1095e8c8_isrForCrosstalkSources_855424_27.stderr parsl.dataflow.dflow DEBUG: Task 1460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1460 has memoization hash 80a5b79bdc31890373cd6ed2a6a6fee7 parsl.dataflow.memoization INFO: Task 1460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1460 try 0 launched on executor multi with executor id 1461 parsl.dataflow.dflow INFO: Standard output for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c8864470-e990-4d27-93a4-b1e86e2d3c1f_isrForCrosstalkSources_855380_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c8864470-e990-4d27-93a4-b1e86e2d3c1f_isrForCrosstalkSources_855380_62.stderr parsl.dataflow.dflow DEBUG: Task 1461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1461 has memoization hash 949a71e8734258f9298fef2a1de311be parsl.dataflow.memoization INFO: Task 1461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1461 try 0 launched on executor multi with executor id 1462 parsl.dataflow.dflow INFO: Standard output for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3c36c0ab-723c-4b59-8cae-b8a530734ab6_isrForCrosstalkSources_855426_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3c36c0ab-723c-4b59-8cae-b8a530734ab6_isrForCrosstalkSources_855426_23.stderr parsl.dataflow.dflow DEBUG: Task 1462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1462 has memoization hash 1e7b09ca49ac8e11ba9addda4bb61dd4 parsl.dataflow.memoization INFO: Task 1462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1462 try 0 launched on executor multi with executor id 1463 parsl.dataflow.dflow INFO: Standard output for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/71392155-85fa-494d-8cbf-4494203ddb20_isrForCrosstalkSources_855443_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/71392155-85fa-494d-8cbf-4494203ddb20_isrForCrosstalkSources_855443_61.stderr parsl.dataflow.dflow DEBUG: Task 1463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1463 has memoization hash bbbc4d1fac633bcecfba440f95655444 parsl.dataflow.memoization INFO: Task 1463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1463 try 0 launched on executor multi with executor id 1464 parsl.dataflow.dflow INFO: Standard output for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8ca21f6-9695-46d9-8a65-9d00e305eea3_isrForCrosstalkSources_855449_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8ca21f6-9695-46d9-8a65-9d00e305eea3_isrForCrosstalkSources_855449_21.stderr parsl.dataflow.dflow DEBUG: Task 1464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1464 has memoization hash 130570862c65fbba3c1303bb7f6fcfc2 parsl.dataflow.memoization INFO: Task 1464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1464 try 0 launched on executor multi with executor id 1465 parsl.dataflow.dflow INFO: Standard output for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/502bcdce-8158-4a21-b841-13c02f26d487_isrForCrosstalkSources_855412_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/502bcdce-8158-4a21-b841-13c02f26d487_isrForCrosstalkSources_855412_36.stderr parsl.dataflow.dflow DEBUG: Task 1465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1465 has memoization hash c33fa6335a4933f61381e6d416f74550 parsl.dataflow.memoization INFO: Task 1465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1465 try 0 launched on executor multi with executor id 1466 parsl.dataflow.dflow INFO: Standard output for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7cdd2fa0-f152-4d7f-8a7f-9e7ef820b994_isrForCrosstalkSources_855450_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7cdd2fa0-f152-4d7f-8a7f-9e7ef820b994_isrForCrosstalkSources_855450_5.stderr parsl.dataflow.dflow DEBUG: Task 1466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1466 has memoization hash 82c833b38705bbec0c0a18cb02b1fa85 parsl.dataflow.memoization INFO: Task 1466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4028de1093ff467cfe5770721655bddf with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/83fc7c42-4a60-4eeb-a50c-32a027ff8897_isrForCrosstalkSources_855413_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/83fc7c42-4a60-4eeb-a50c-32a027ff8897_isrForCrosstalkSources_855413_37.stderr parsl.dataflow.dflow INFO: Parsl task 1466 try 0 launched on executor multi with executor id 1467 parsl.dataflow.dflow INFO: Standard output for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f7443be1-caf8-4054-ac34-f652c4d822d6_isrForCrosstalkSources_855421_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f7443be1-caf8-4054-ac34-f652c4d822d6_isrForCrosstalkSources_855421_27.stderr parsl.dataflow.dflow DEBUG: Task 1467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1467 has memoization hash eb7bdcf8c0cd3122147781c2d95f688d parsl.dataflow.memoization INFO: Task 1467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1467 try 0 launched on executor multi with executor id 1468 parsl.dataflow.dflow INFO: Standard output for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bafe2bf3-0373-4779-aadd-1e2fa1f4e9c1_isrForCrosstalkSources_855431_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bafe2bf3-0373-4779-aadd-1e2fa1f4e9c1_isrForCrosstalkSources_855431_15.stderr parsl.dataflow.dflow DEBUG: Task 1468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1468 has memoization hash 03778bd064035db39ae95e0149243052 parsl.dataflow.memoization INFO: Task 1468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1468 try 0 launched on executor multi with executor id 1469 parsl.dataflow.dflow INFO: Standard output for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f4d1c706-7978-4004-8168-075ad18f255c_isrForCrosstalkSources_855424_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f4d1c706-7978-4004-8168-075ad18f255c_isrForCrosstalkSources_855424_25.stderr parsl.dataflow.dflow DEBUG: Task 1469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1469 has memoization hash 74492a77f9e89c34af27601a99326faa parsl.dataflow.memoization INFO: Task 1469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1469 try 0 launched on executor multi with executor id 1470 parsl.dataflow.dflow INFO: Standard output for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/69488734-090d-43d8-84d9-62d0c9c286e3_isrForCrosstalkSources_855405_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/69488734-090d-43d8-84d9-62d0c9c286e3_isrForCrosstalkSources_855405_59.stderr parsl.dataflow.dflow DEBUG: Task 1470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1470 has memoization hash d125c2a82d7eee5f9bb19693b9190cf9 parsl.dataflow.memoization INFO: Task 1470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1470 try 0 launched on executor multi with executor id 1471 parsl.dataflow.dflow INFO: Standard output for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/73ad37f6-d4d4-4b72-a5cb-e4b5a11bca0f_isrForCrosstalkSources_855438_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/73ad37f6-d4d4-4b72-a5cb-e4b5a11bca0f_isrForCrosstalkSources_855438_27.stderr parsl.dataflow.dflow DEBUG: Task 1471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1471 has memoization hash cf528a8b8ccfcece26351c6b5e86a1b9 parsl.dataflow.memoization INFO: Task 1471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1471 try 0 launched on executor multi with executor id 1472 parsl.dataflow.dflow INFO: Standard output for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c806cd76-eaf4-449b-bb6d-5db845ba00b7_isrForCrosstalkSources_855452_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c806cd76-eaf4-449b-bb6d-5db845ba00b7_isrForCrosstalkSources_855452_21.stderr parsl.dataflow.dflow DEBUG: Task 1472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1472 has memoization hash 03fed52e14c32b0b61295b0021e14253 parsl.dataflow.memoization INFO: Task 1472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1472 try 0 launched on executor multi with executor id 1473 parsl.dataflow.dflow INFO: Standard output for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/15729995-4f17-44e7-b974-2ce57f727e90_isrForCrosstalkSources_855448_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/15729995-4f17-44e7-b974-2ce57f727e90_isrForCrosstalkSources_855448_58.stderr parsl.dataflow.dflow DEBUG: Task 1473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1473 has memoization hash 95420ca6ad3b2191f3e20ff6bb89c256 parsl.dataflow.memoization INFO: Task 1473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1473 try 0 launched on executor multi with executor id 1474 parsl.dataflow.dflow INFO: Standard output for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/633735e3-81cb-48bd-ad65-212978855fcd_isrForCrosstalkSources_855442_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/633735e3-81cb-48bd-ad65-212978855fcd_isrForCrosstalkSources_855442_34.stderr parsl.dataflow.dflow DEBUG: Task 1474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1474 has memoization hash cee0b70a1b8f72d02032892112585d2a parsl.dataflow.memoization INFO: Task 1474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1474 try 0 launched on executor multi with executor id 1475 parsl.dataflow.dflow INFO: Standard output for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/62d54c10-6479-42d8-817c-a197d0618efb_isrForCrosstalkSources_855440_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/62d54c10-6479-42d8-817c-a197d0618efb_isrForCrosstalkSources_855440_31.stderr parsl.dataflow.dflow DEBUG: Task 1475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1475 has memoization hash 420d6632698a98f852b021fbfd6cf46f parsl.dataflow.memoization INFO: Task 1475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1475 try 0 launched on executor multi with executor id 1476 parsl.dataflow.dflow INFO: Standard output for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4b5989dc-79cb-41b6-9eac-f92a95dc9826_isrForCrosstalkSources_855390_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4b5989dc-79cb-41b6-9eac-f92a95dc9826_isrForCrosstalkSources_855390_3.stderr parsl.dataflow.dflow DEBUG: Task 1476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1476 has memoization hash 4b9b84c8bca63b82a594c7cca1a33698 parsl.dataflow.memoization INFO: Task 1476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1476 try 0 launched on executor multi with executor id 1477 parsl.dataflow.dflow INFO: Standard output for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3bc14a05-08c0-417d-bbf9-791a2c5019f5_isrForCrosstalkSources_855413_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3bc14a05-08c0-417d-bbf9-791a2c5019f5_isrForCrosstalkSources_855413_53.stderr parsl.dataflow.dflow DEBUG: Task 1477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1477 has memoization hash 8454c45fd21443c69ec5467b733a391a parsl.dataflow.memoization INFO: Task 1477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1477 try 0 launched on executor multi with executor id 1478 parsl.dataflow.dflow INFO: Standard output for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7dd82a21-1e10-428e-a6f1-e9da3bc09160_isrForCrosstalkSources_855434_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7dd82a21-1e10-428e-a6f1-e9da3bc09160_isrForCrosstalkSources_855434_14.stderr parsl.dataflow.dflow DEBUG: Task 1478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1478 has memoization hash 234c3ec67ee49ec0d488b97fe73b50de parsl.dataflow.memoization INFO: Task 1478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1478 try 0 launched on executor multi with executor id 1479 parsl.dataflow.dflow INFO: Standard output for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/518b379a-199e-4da3-b02f-c9fc3df738c4_isrForCrosstalkSources_855432_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/518b379a-199e-4da3-b02f-c9fc3df738c4_isrForCrosstalkSources_855432_35.stderr parsl.dataflow.dflow DEBUG: Task 1479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1479 has memoization hash 545d70e73161340aeeaea6fa7f61cd64 parsl.dataflow.memoization INFO: Task 1479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1479 try 0 launched on executor multi with executor id 1480 parsl.dataflow.dflow INFO: Standard output for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ec137fdd-fb3d-4fd2-b168-843aa25e272b_isrForCrosstalkSources_855433_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ec137fdd-fb3d-4fd2-b168-843aa25e272b_isrForCrosstalkSources_855433_31.stderr parsl.dataflow.dflow DEBUG: Task 1480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1480 has memoization hash 02c69ceb02bf70075bf58ec616413b6a parsl.dataflow.memoization INFO: Task 1480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1480 try 0 launched on executor multi with executor id 1481 parsl.dataflow.dflow INFO: Standard output for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/87b67279-14a6-429d-a0b0-2c9998e690f5_isrForCrosstalkSources_855387_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/87b67279-14a6-429d-a0b0-2c9998e690f5_isrForCrosstalkSources_855387_27.stderr parsl.dataflow.dflow DEBUG: Task 1481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1481 has memoization hash 93658fef48cb95239e21567718f92ab4 parsl.dataflow.memoization INFO: Task 1481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964069 proc_lsst.multi INFO: [multi] job_id 16964069 2024-03-09 14:50:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 14:50:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964069 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964069 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16964069 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-09 14:50:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1481 try 0 launched on executor multi with executor id 1482 parsl.dataflow.dflow INFO: Standard output for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2ac88759-74a3-469d-bc5a-a688feba710c_isrForCrosstalkSources_855455_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2ac88759-74a3-469d-bc5a-a688feba710c_isrForCrosstalkSources_855455_21.stderr parsl.dataflow.dflow DEBUG: Task 1482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1482 has memoization hash 5f0028954ea8d9035b0144b4dfd927b5 parsl.dataflow.memoization INFO: Task 1482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1482 try 0 launched on executor multi with executor id 1483 parsl.dataflow.dflow INFO: Standard output for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/099901cf-0314-4994-8d12-9ddd77245969_isrForCrosstalkSources_855392_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/099901cf-0314-4994-8d12-9ddd77245969_isrForCrosstalkSources_855392_25.stderr parsl.dataflow.dflow DEBUG: Task 1483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1483 has memoization hash 7d9111dcef67c205a974cc538ae2400f parsl.dataflow.memoization INFO: Task 1483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1483 try 0 launched on executor multi with executor id 1484 parsl.dataflow.dflow INFO: Standard output for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/523bc2b1-f657-41f8-8f6a-90ddddc51644_isrForCrosstalkSources_855418_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/523bc2b1-f657-41f8-8f6a-90ddddc51644_isrForCrosstalkSources_855418_31.stderr parsl.dataflow.dflow DEBUG: Task 1484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1484 has memoization hash a9dbb8b209577d0bbe0e6faaecf3ac67 parsl.dataflow.memoization INFO: Task 1484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aaf3eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1484 try 0 launched on executor multi with executor id 1485 parsl.dataflow.dflow INFO: Standard output for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/07fabea5-0463-421b-b46c-8a34c5b99546_isrForCrosstalkSources_855440_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/07fabea5-0463-421b-b46c-8a34c5b99546_isrForCrosstalkSources_855440_11.stderr parsl.dataflow.dflow DEBUG: Task 1485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1485 has memoization hash 8cf862e942a2c5f252866503ac193914 parsl.dataflow.memoization INFO: Task 1485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1485 try 0 launched on executor multi with executor id 1486 parsl.dataflow.dflow INFO: Standard output for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/524f5dcd-21ae-42b5-a14f-2cb7b9ab1068_isrForCrosstalkSources_855409_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/524f5dcd-21ae-42b5-a14f-2cb7b9ab1068_isrForCrosstalkSources_855409_9.stderr parsl.dataflow.dflow DEBUG: Task 1486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1486 has memoization hash a9d0bfb5fc625fdd60eb73ddfe5401c2 parsl.dataflow.memoization INFO: Task 1486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d267a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1486 try 0 launched on executor multi with executor id 1487 parsl.dataflow.dflow INFO: Standard output for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/049e97b8-ca23-4afb-a482-088fd19fce66_isrForCrosstalkSources_855417_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/049e97b8-ca23-4afb-a482-088fd19fce66_isrForCrosstalkSources_855417_61.stderr parsl.dataflow.dflow DEBUG: Task 1487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1487 has memoization hash d324e914d9125c70695905d0a9a5f184 parsl.dataflow.memoization INFO: Task 1487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1487 try 0 launched on executor multi with executor id 1488 parsl.dataflow.dflow INFO: Standard output for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/23a2e29a-3745-483c-8aa7-000efd19e459_isrForCrosstalkSources_855384_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/23a2e29a-3745-483c-8aa7-000efd19e459_isrForCrosstalkSources_855384_56.stderr parsl.dataflow.dflow DEBUG: Task 1488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1488 has memoization hash 37a7898626f24c9477755bed599d1e6a parsl.dataflow.memoization INFO: Task 1488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1488 try 0 launched on executor multi with executor id 1489 parsl.dataflow.dflow INFO: Standard output for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e33deb42-1661-4653-a84d-1f458ce7f374_isrForCrosstalkSources_855402_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e33deb42-1661-4653-a84d-1f458ce7f374_isrForCrosstalkSources_855402_32.stderr parsl.dataflow.dflow DEBUG: Task 1489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1489 has memoization hash 7e930f09ab00a3a3f81f23ab85175db0 parsl.dataflow.memoization INFO: Task 1489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1489 try 0 launched on executor multi with executor id 1490 parsl.dataflow.dflow INFO: Standard output for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/41ce8129-76ce-4f45-9fc2-ed5d089bac12_isrForCrosstalkSources_855399_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/41ce8129-76ce-4f45-9fc2-ed5d089bac12_isrForCrosstalkSources_855399_25.stderr parsl.dataflow.dflow DEBUG: Task 1490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1490 has memoization hash c42c503475344d01286317a43794d1ce parsl.dataflow.memoization INFO: Task 1490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1490 try 0 launched on executor multi with executor id 1491 parsl.dataflow.dflow INFO: Standard output for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9c03fd4e-d4f5-40ef-9ed8-6a4f81bb9eac_isrForCrosstalkSources_855387_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9c03fd4e-d4f5-40ef-9ed8-6a4f81bb9eac_isrForCrosstalkSources_855387_39.stderr parsl.dataflow.dflow DEBUG: Task 1491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1491 has memoization hash b69cb842fdee62a39a0d9f948834f3fe parsl.dataflow.memoization INFO: Task 1491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1491 try 0 launched on executor multi with executor id 1492 parsl.dataflow.dflow INFO: Standard output for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/854cbe61-03ac-46d1-8e67-f13f43acefb2_isrForCrosstalkSources_855388_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/854cbe61-03ac-46d1-8e67-f13f43acefb2_isrForCrosstalkSources_855388_49.stderr parsl.dataflow.dflow DEBUG: Task 1492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1492 has memoization hash b225482f93e3fddd65b23a9ae3f1a55e parsl.dataflow.memoization INFO: Task 1492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1492 try 0 launched on executor multi with executor id 1493 parsl.dataflow.dflow INFO: Standard output for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3f6eb9b4-473d-4c1e-bf5a-51367fbdc56c_isrForCrosstalkSources_855420_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3f6eb9b4-473d-4c1e-bf5a-51367fbdc56c_isrForCrosstalkSources_855420_38.stderr parsl.dataflow.dflow DEBUG: Task 1493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1493 has memoization hash 4913b0e228b33bc1a17039202fc244f8 parsl.dataflow.memoization INFO: Task 1493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1493 try 0 launched on executor multi with executor id 1494 parsl.dataflow.dflow INFO: Standard output for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8b73a08d-622d-48f0-b0f9-befa7ee6dec9_isrForCrosstalkSources_855441_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8b73a08d-622d-48f0-b0f9-befa7ee6dec9_isrForCrosstalkSources_855441_40.stderr parsl.dataflow.dflow DEBUG: Task 1494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1494 has memoization hash a2fabd7438f0502ded01d01c1956ab5d parsl.dataflow.memoization INFO: Task 1494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1494 try 0 launched on executor multi with executor id 1495 parsl.dataflow.dflow INFO: Standard output for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7ebec094-6549-41fe-a868-06eee5b01662_isrForCrosstalkSources_855402_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7ebec094-6549-41fe-a868-06eee5b01662_isrForCrosstalkSources_855402_47.stderr parsl.dataflow.dflow DEBUG: Task 1495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1495 has memoization hash 863bab34a716bec640a6a93f0ff01c6a parsl.dataflow.memoization INFO: Task 1495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1495 try 0 launched on executor multi with executor id 1496 parsl.dataflow.dflow INFO: Standard output for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/81dd28fd-865e-4b7a-ab28-55e5cd6c4a81_isrForCrosstalkSources_855406_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/81dd28fd-865e-4b7a-ab28-55e5cd6c4a81_isrForCrosstalkSources_855406_8.stderr parsl.dataflow.dflow DEBUG: Task 1496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1496 has memoization hash 18c13244230ee0f2acfa12de7c703357 parsl.dataflow.memoization INFO: Task 1496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1496 try 0 launched on executor multi with executor id 1497 parsl.dataflow.dflow INFO: Standard output for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0132acac-04cd-4411-8ae0-2a2daceb05f1_isrForCrosstalkSources_855444_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0132acac-04cd-4411-8ae0-2a2daceb05f1_isrForCrosstalkSources_855444_39.stderr parsl.dataflow.dflow DEBUG: Task 1497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1497 has memoization hash 146e5fdbb6e2631baae6d13d90a2b1d2 parsl.dataflow.memoization INFO: Task 1497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1497 try 0 launched on executor multi with executor id 1498 parsl.dataflow.dflow INFO: Standard output for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/11e4b955-ced1-4c30-9e81-2818f1b5c4f8_isrForCrosstalkSources_855393_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/11e4b955-ced1-4c30-9e81-2818f1b5c4f8_isrForCrosstalkSources_855393_16.stderr parsl.dataflow.dflow DEBUG: Task 1498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1498 has memoization hash cfa7e2005a07dcf2fe0adef23434393a parsl.dataflow.memoization INFO: Task 1498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1498 try 0 launched on executor multi with executor id 1499 parsl.dataflow.dflow INFO: Standard output for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7e7b343c-d761-4ef5-8a79-206949752ebf_isrForCrosstalkSources_855406_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7e7b343c-d761-4ef5-8a79-206949752ebf_isrForCrosstalkSources_855406_16.stderr parsl.dataflow.dflow DEBUG: Task 1499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1499 has memoization hash 66fd5200dbeeb73929339ce8ab2913fa parsl.dataflow.memoization INFO: Task 1499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1499 try 0 launched on executor multi with executor id 1500 parsl.dataflow.dflow INFO: Standard output for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/29d8e73e-85d6-4ba8-bcdd-8cbbb768c93d_isrForCrosstalkSources_855411_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/29d8e73e-85d6-4ba8-bcdd-8cbbb768c93d_isrForCrosstalkSources_855411_53.stderr parsl.dataflow.dflow DEBUG: Task 1500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1500 has memoization hash b00feed41e53c5c709230724258bb336 parsl.dataflow.memoization INFO: Task 1500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1500 try 0 launched on executor multi with executor id 1501 parsl.dataflow.dflow INFO: Standard output for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4bb21ae8-d741-4a4d-9998-6eb3ebe0540f_isrForCrosstalkSources_855430_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4bb21ae8-d741-4a4d-9998-6eb3ebe0540f_isrForCrosstalkSources_855430_7.stderr parsl.dataflow.dflow DEBUG: Task 1501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1501 has memoization hash b68197d1d3c99c7678b981fbd0fbc895 parsl.dataflow.memoization INFO: Task 1501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1501 try 0 launched on executor multi with executor id 1502 parsl.dataflow.dflow INFO: Standard output for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/475b9a03-a092-43b6-9511-6d480d2c08c7_isrForCrosstalkSources_855389_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/475b9a03-a092-43b6-9511-6d480d2c08c7_isrForCrosstalkSources_855389_12.stderr parsl.dataflow.dflow DEBUG: Task 1502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1502 has memoization hash 69d0648962967274672c4d17b37f257a parsl.dataflow.memoization INFO: Task 1502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1502 try 0 launched on executor multi with executor id 1503 parsl.dataflow.dflow INFO: Standard output for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/13b4a2d7-d988-4b00-9a70-1a608501ce84_isrForCrosstalkSources_855424_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/13b4a2d7-d988-4b00-9a70-1a608501ce84_isrForCrosstalkSources_855424_3.stderr parsl.dataflow.dflow DEBUG: Task 1503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1503 has memoization hash a40368a11709f855ab195a2848402078 parsl.dataflow.memoization INFO: Task 1503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1503 try 0 launched on executor multi with executor id 1504 parsl.dataflow.dflow INFO: Standard output for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7337dd1e-dfac-40c3-a6ac-1d973e35205e_isrForCrosstalkSources_855445_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7337dd1e-dfac-40c3-a6ac-1d973e35205e_isrForCrosstalkSources_855445_29.stderr parsl.dataflow.dflow DEBUG: Task 1504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1504 has memoization hash ccb6873f2897830dcf9e2942448cbbca parsl.dataflow.memoization INFO: Task 1504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1504 try 0 launched on executor multi with executor id 1505 parsl.dataflow.dflow INFO: Standard output for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/474c0589-670d-458d-8ab9-6947d4e6fa1e_isrForCrosstalkSources_855407_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/474c0589-670d-458d-8ab9-6947d4e6fa1e_isrForCrosstalkSources_855407_20.stderr parsl.dataflow.dflow DEBUG: Task 1505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1505 has memoization hash 72b413566b74e0333d4a546ca6539606 parsl.dataflow.memoization INFO: Task 1505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1505 try 0 launched on executor multi with executor id 1506 parsl.dataflow.dflow INFO: Standard output for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6aa6f466-f412-4d7d-b982-bfe76e5dba98_isrForCrosstalkSources_855446_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6aa6f466-f412-4d7d-b982-bfe76e5dba98_isrForCrosstalkSources_855446_23.stderr parsl.dataflow.dflow DEBUG: Task 1506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1506 has memoization hash 97e2f3359a1906519d61227598dc1f38 parsl.dataflow.memoization INFO: Task 1506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1506 try 0 launched on executor multi with executor id 1507 parsl.dataflow.dflow INFO: Standard output for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/512e8d32-7314-466f-83ff-a69f3c96f315_isrForCrosstalkSources_855453_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/512e8d32-7314-466f-83ff-a69f3c96f315_isrForCrosstalkSources_855453_38.stderr parsl.dataflow.dflow DEBUG: Task 1507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1507 has memoization hash c78ce8d1c3b94a48962b2f0f30f99307 parsl.dataflow.memoization INFO: Task 1507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1507 try 0 launched on executor multi with executor id 1508 parsl.dataflow.dflow INFO: Standard output for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/08302ac3-3304-4dee-bf6c-9c66e39f3748_isrForCrosstalkSources_855396_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/08302ac3-3304-4dee-bf6c-9c66e39f3748_isrForCrosstalkSources_855396_52.stderr parsl.dataflow.dflow DEBUG: Task 1508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1508 has memoization hash 55b0e15fc1dadf01e1c399502d36d604 parsl.dataflow.memoization INFO: Task 1508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1508 try 0 launched on executor multi with executor id 1509 parsl.dataflow.dflow INFO: Standard output for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ba87233c-d026-4486-924c-a3b20163025f_isrForCrosstalkSources_855455_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ba87233c-d026-4486-924c-a3b20163025f_isrForCrosstalkSources_855455_2.stderr parsl.dataflow.dflow DEBUG: Task 1509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1509 has memoization hash bc42253840ace406d912212ecc004c8f parsl.dataflow.memoization INFO: Task 1509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1509 try 0 launched on executor multi with executor id 1510 parsl.dataflow.dflow INFO: Standard output for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/40ad6e44-7f78-4755-8ebb-35785b94ec12_isrForCrosstalkSources_855414_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/40ad6e44-7f78-4755-8ebb-35785b94ec12_isrForCrosstalkSources_855414_24.stderr parsl.dataflow.dflow DEBUG: Task 1510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1510 has memoization hash db1323b4e8ad7aee58f0fd0ab1b294a8 parsl.dataflow.memoization INFO: Task 1510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d242c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1510 try 0 launched on executor multi with executor id 1511 parsl.dataflow.dflow INFO: Standard output for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dd97e0fd-92e6-46ef-947d-2776c36f6f27_isrForCrosstalkSources_855434_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dd97e0fd-92e6-46ef-947d-2776c36f6f27_isrForCrosstalkSources_855434_38.stderr parsl.dataflow.dflow DEBUG: Task 1511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1511 has memoization hash 58a75ebc20db701dbd28e0fa597b213b parsl.dataflow.memoization INFO: Task 1511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d240e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1511 try 0 launched on executor multi with executor id 1512 parsl.dataflow.dflow INFO: Standard output for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c504f64-62fc-4220-a553-99cc6ab695cb_isrForCrosstalkSources_855400_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c504f64-62fc-4220-a553-99cc6ab695cb_isrForCrosstalkSources_855400_53.stderr parsl.dataflow.dflow DEBUG: Task 1512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1512 has memoization hash 6c6e456b057eeb6773932eb1cbb9e81a parsl.dataflow.memoization INFO: Task 1512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1512 try 0 launched on executor multi with executor id 1513 parsl.dataflow.dflow INFO: Standard output for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94cba6f2-d167-4756-9daf-468fd10b6c12_isrForCrosstalkSources_855448_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94cba6f2-d167-4756-9daf-468fd10b6c12_isrForCrosstalkSources_855448_30.stderr parsl.dataflow.dflow DEBUG: Task 1513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1513 has memoization hash d7b032c8e34703cdef20dbd307cacffc parsl.dataflow.memoization INFO: Task 1513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d244a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1513 try 0 launched on executor multi with executor id 1514 parsl.dataflow.dflow INFO: Standard output for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/86cc4c6b-b206-4e22-8b2d-a6a542e635ba_isrForCrosstalkSources_855436_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/86cc4c6b-b206-4e22-8b2d-a6a542e635ba_isrForCrosstalkSources_855436_50.stderr parsl.dataflow.dflow DEBUG: Task 1514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1514 has memoization hash 64d8564066a9470f27ca5238fe046b71 parsl.dataflow.memoization INFO: Task 1514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1514 try 0 launched on executor multi with executor id 1515 parsl.dataflow.dflow INFO: Standard output for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/477489cf-d7f4-45ea-8d97-e9fcf75af794_isrForCrosstalkSources_855442_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/477489cf-d7f4-45ea-8d97-e9fcf75af794_isrForCrosstalkSources_855442_46.stderr parsl.dataflow.dflow DEBUG: Task 1515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1515 has memoization hash b1e5cbdaeb6d4fbef8822b56e37a9d9c parsl.dataflow.memoization INFO: Task 1515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d245e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1515 try 0 launched on executor multi with executor id 1516 parsl.dataflow.dflow INFO: Standard output for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/99924625-b0c6-4890-8304-b145ffc16266_isrForCrosstalkSources_855432_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/99924625-b0c6-4890-8304-b145ffc16266_isrForCrosstalkSources_855432_20.stderr parsl.dataflow.dflow DEBUG: Task 1516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1516 has memoization hash b683c69265a3f81dec55bfac22e5b149 parsl.dataflow.memoization INFO: Task 1516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964070 proc_lsst.multi INFO: [multi] job_id 16964070 2024-03-09 14:50:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 14:50:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964070 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964070 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16964070 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-09 14:50:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 14:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1516 try 0 launched on executor multi with executor id 1517 parsl.dataflow.dflow INFO: Standard output for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ab2810ed-c4ed-4d49-8e97-018de2003ca5_isrForCrosstalkSources_855444_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ab2810ed-c4ed-4d49-8e97-018de2003ca5_isrForCrosstalkSources_855444_15.stderr parsl.dataflow.dflow DEBUG: Task 1517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1517 has memoization hash 1f04649c4079bb27079f1c013a4ccea8 parsl.dataflow.memoization INFO: Task 1517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1517 try 0 launched on executor multi with executor id 1518 parsl.dataflow.dflow INFO: Standard output for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1d3ec596-8447-40bc-b1da-540da8e4214b_isrForCrosstalkSources_855395_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1d3ec596-8447-40bc-b1da-540da8e4214b_isrForCrosstalkSources_855395_44.stderr parsl.dataflow.dflow DEBUG: Task 1518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1518 has memoization hash acfc4b90785080daecd112de36ecf4c4 parsl.dataflow.memoization INFO: Task 1518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1518 try 0 launched on executor multi with executor id 1519 parsl.dataflow.dflow INFO: Standard output for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d020cc23-cf07-4cb9-8114-95f9cebec142_isrForCrosstalkSources_855429_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d020cc23-cf07-4cb9-8114-95f9cebec142_isrForCrosstalkSources_855429_61.stderr parsl.dataflow.dflow DEBUG: Task 1519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1519 has memoization hash 5ec61dafc59af5a155fec0125d8b0381 parsl.dataflow.memoization INFO: Task 1519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1519 try 0 launched on executor multi with executor id 1520 parsl.dataflow.dflow INFO: Standard output for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa5fc20a-9ef8-4baa-a3ae-c183c9d249c3_isrForCrosstalkSources_855408_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa5fc20a-9ef8-4baa-a3ae-c183c9d249c3_isrForCrosstalkSources_855408_18.stderr parsl.dataflow.dflow DEBUG: Task 1520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1520 has memoization hash a4630fc0d533f718e12a67cacfba54c3 parsl.dataflow.memoization INFO: Task 1520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1520 try 0 launched on executor multi with executor id 1521 parsl.dataflow.dflow INFO: Standard output for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/07e7c45b-c608-46b1-9cc0-95067304f1ba_isrForCrosstalkSources_855383_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/07e7c45b-c608-46b1-9cc0-95067304f1ba_isrForCrosstalkSources_855383_4.stderr parsl.dataflow.dflow DEBUG: Task 1521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1521 has memoization hash 87a35b3f072966e91ff903b8d668ebf7 parsl.dataflow.memoization INFO: Task 1521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1521 try 0 launched on executor multi with executor id 1522 parsl.dataflow.dflow INFO: Standard output for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4eadf268-452f-46e1-8803-449c0c48703c_isrForCrosstalkSources_855415_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4eadf268-452f-46e1-8803-449c0c48703c_isrForCrosstalkSources_855415_24.stderr parsl.dataflow.dflow DEBUG: Task 1522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1522 has memoization hash 4e11222a91a5e591e5846cca6615a988 parsl.dataflow.memoization INFO: Task 1522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d247c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1522 try 0 launched on executor multi with executor id 1523 parsl.dataflow.dflow INFO: Standard output for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dddd2166-a442-42ff-98d4-9ecc39568f28_isrForCrosstalkSources_855432_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dddd2166-a442-42ff-98d4-9ecc39568f28_isrForCrosstalkSources_855432_21.stderr parsl.dataflow.dflow DEBUG: Task 1523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1523 has memoization hash 94ba7d72c265a9fb326c4ea1694efcec parsl.dataflow.memoization INFO: Task 1523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1523 try 0 launched on executor multi with executor id 1524 parsl.dataflow.dflow INFO: Standard output for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e96b8913-57c4-47fe-82d9-ad3bc1dee730_isrForCrosstalkSources_855413_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e96b8913-57c4-47fe-82d9-ad3bc1dee730_isrForCrosstalkSources_855413_31.stderr parsl.dataflow.dflow DEBUG: Task 1524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1524 has memoization hash 8127247d0ab217f8e03f66c1d846467f parsl.dataflow.memoization INFO: Task 1524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1524 try 0 launched on executor multi with executor id 1525 parsl.dataflow.dflow INFO: Standard output for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e356d3c7-f0e8-40e9-b540-3928ae359fdd_isrForCrosstalkSources_855380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e356d3c7-f0e8-40e9-b540-3928ae359fdd_isrForCrosstalkSources_855380_2.stderr parsl.dataflow.dflow DEBUG: Task 1525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1525 has memoization hash 02e0fb7e6deb4e7dbf47ef577973c175 parsl.dataflow.memoization INFO: Task 1525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1525 try 0 launched on executor multi with executor id 1526 parsl.dataflow.dflow INFO: Standard output for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/bdf22790-809d-4f82-b1ba-3553a647ab22_isrForCrosstalkSources_855398_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/bdf22790-809d-4f82-b1ba-3553a647ab22_isrForCrosstalkSources_855398_8.stderr parsl.dataflow.dflow DEBUG: Task 1526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1526 has memoization hash 5e71684c282cb5ba4eb41d29e3ced39d parsl.dataflow.memoization INFO: Task 1526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1526 try 0 launched on executor multi with executor id 1527 parsl.dataflow.dflow INFO: Standard output for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/75b7a3a0-3b2d-432c-8d2a-53dd7c6d0de9_isrForCrosstalkSources_855450_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/75b7a3a0-3b2d-432c-8d2a-53dd7c6d0de9_isrForCrosstalkSources_855450_28.stderr parsl.dataflow.dflow DEBUG: Task 1527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1527 has memoization hash e79800cfb614cf746ffec7a9a4b43b7f parsl.dataflow.memoization INFO: Task 1527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1527 try 0 launched on executor multi with executor id 1528 parsl.dataflow.dflow INFO: Standard output for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46346409-994f-4249-bfcf-7b49e55a950d_isrForCrosstalkSources_855419_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46346409-994f-4249-bfcf-7b49e55a950d_isrForCrosstalkSources_855419_24.stderr parsl.dataflow.dflow DEBUG: Task 1528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1528 has memoization hash 152dbe4752d6ccef72fc33a6a4f5572f parsl.dataflow.memoization INFO: Task 1528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d249a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1528 try 0 launched on executor multi with executor id 1529 parsl.dataflow.dflow INFO: Standard output for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7e8bb3f6-647c-49fc-885d-9f5cbaad5770_isrForCrosstalkSources_855441_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7e8bb3f6-647c-49fc-885d-9f5cbaad5770_isrForCrosstalkSources_855441_50.stderr parsl.dataflow.dflow DEBUG: Task 1529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1529 has memoization hash e91c7d40e40b698c3482528f8b19cfbf parsl.dataflow.memoization INFO: Task 1529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1529 try 0 launched on executor multi with executor id 1530 parsl.dataflow.dflow INFO: Standard output for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/020dcde1-e20f-4107-bd0b-e154235b1f45_isrForCrosstalkSources_855450_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/020dcde1-e20f-4107-bd0b-e154235b1f45_isrForCrosstalkSources_855450_51.stderr parsl.dataflow.dflow DEBUG: Task 1530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1530 has memoization hash f269f8cd08cc6dbfbca5a44418367493 parsl.dataflow.memoization INFO: Task 1530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1530 try 0 launched on executor multi with executor id 1531 parsl.dataflow.dflow INFO: Standard output for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a7ea0167-22d9-494a-ab33-b907b0655d07_isrForCrosstalkSources_855422_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a7ea0167-22d9-494a-ab33-b907b0655d07_isrForCrosstalkSources_855422_17.stderr parsl.dataflow.dflow DEBUG: Task 1531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1531 has memoization hash 859bd1a169a8a05810413d19b004609c parsl.dataflow.memoization INFO: Task 1531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1531 try 0 launched on executor multi with executor id 1532 parsl.dataflow.dflow INFO: Standard output for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/8cfe2e41-f295-4604-ab09-8bb165147745_isrForCrosstalkSources_855452_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/8cfe2e41-f295-4604-ab09-8bb165147745_isrForCrosstalkSources_855452_18.stderr parsl.dataflow.dflow DEBUG: Task 1532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1532 has memoization hash d3888059c14cb77ae0a257215ad7bd8c parsl.dataflow.memoization INFO: Task 1532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1532 try 0 launched on executor multi with executor id 1533 parsl.dataflow.dflow INFO: Standard output for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e4c828c1-0390-4889-946b-87679d3c630b_isrForCrosstalkSources_855435_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e4c828c1-0390-4889-946b-87679d3c630b_isrForCrosstalkSources_855435_50.stderr parsl.dataflow.dflow DEBUG: Task 1533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1533 has memoization hash ce7c86eaf34facc13ee2645eb7571919 parsl.dataflow.memoization INFO: Task 1533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1533 try 0 launched on executor multi with executor id 1534 parsl.dataflow.dflow INFO: Standard output for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a1d5cef5-c5a2-46e9-8e2b-b1b1bda67021_isrForCrosstalkSources_855407_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a1d5cef5-c5a2-46e9-8e2b-b1b1bda67021_isrForCrosstalkSources_855407_19.stderr parsl.dataflow.dflow DEBUG: Task 1534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1534 has memoization hash 27377deff25883a9a68b0ac3619b756a parsl.dataflow.memoization INFO: Task 1534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1534 try 0 launched on executor multi with executor id 1535 parsl.dataflow.dflow INFO: Standard output for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f6a9e2e1-6e97-4abc-a67a-1e4999155189_isrForCrosstalkSources_855390_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f6a9e2e1-6e97-4abc-a67a-1e4999155189_isrForCrosstalkSources_855390_36.stderr parsl.dataflow.dflow DEBUG: Task 1535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1535 has memoization hash 7b9a15fd186537bc10e7e37b371a9fcd parsl.dataflow.memoization INFO: Task 1535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1535 try 0 launched on executor multi with executor id 1536 parsl.dataflow.dflow INFO: Standard output for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/140d19c5-8693-4ad5-9786-2fc260335100_isrForCrosstalkSources_855381_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/140d19c5-8693-4ad5-9786-2fc260335100_isrForCrosstalkSources_855381_35.stderr parsl.dataflow.dflow DEBUG: Task 1536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1536 has memoization hash 6ba6224eee612a211a45db6c5ea238ac parsl.dataflow.memoization INFO: Task 1536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1536 try 0 launched on executor multi with executor id 1537 parsl.dataflow.dflow INFO: Standard output for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0ddfcac2-1355-4805-b008-c6855c526783_isrForCrosstalkSources_855429_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0ddfcac2-1355-4805-b008-c6855c526783_isrForCrosstalkSources_855429_57.stderr parsl.dataflow.dflow DEBUG: Task 1537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1537 has memoization hash d5745a9f83e55de64cf0fe1dec672901 parsl.dataflow.memoization INFO: Task 1537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1537 try 0 launched on executor multi with executor id 1538 parsl.dataflow.dflow INFO: Standard output for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/fd993576-4c50-4354-b5a4-271d7f888cd2_isrForCrosstalkSources_855445_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/fd993576-4c50-4354-b5a4-271d7f888cd2_isrForCrosstalkSources_855445_5.stderr parsl.dataflow.dflow DEBUG: Task 1538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1538 has memoization hash 97e54a7f39bf11303dd2b372a8d2ba62 parsl.dataflow.memoization INFO: Task 1538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d253a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1538 try 0 launched on executor multi with executor id 1539 parsl.dataflow.dflow INFO: Standard output for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b247b6e-2cd7-4193-bebe-af66c6fa9b02_isrForCrosstalkSources_855459_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b247b6e-2cd7-4193-bebe-af66c6fa9b02_isrForCrosstalkSources_855459_52.stderr parsl.dataflow.dflow DEBUG: Task 1539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1539 has memoization hash a54c98af35937e42f4b7ed17ce9ab7df parsl.dataflow.memoization INFO: Task 1539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d251c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1539 try 0 launched on executor multi with executor id 1540 parsl.dataflow.dflow INFO: Standard output for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8652b756-8de9-497c-b29e-1ffee48f5bc4_isrForCrosstalkSources_855407_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8652b756-8de9-497c-b29e-1ffee48f5bc4_isrForCrosstalkSources_855407_47.stderr parsl.dataflow.dflow DEBUG: Task 1540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1540 has memoization hash 73257de939d687c1d331108e3cf5e336 parsl.dataflow.memoization INFO: Task 1540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1540 try 0 launched on executor multi with executor id 1541 parsl.dataflow.dflow INFO: Standard output for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/295f32f4-49fa-43d2-bfeb-65cedf898932_isrForCrosstalkSources_855449_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/295f32f4-49fa-43d2-bfeb-65cedf898932_isrForCrosstalkSources_855449_61.stderr parsl.dataflow.dflow DEBUG: Task 1541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1541 has memoization hash e021efea54f450d0305941d42fdbd04d parsl.dataflow.memoization INFO: Task 1541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1541 try 0 launched on executor multi with executor id 1542 parsl.dataflow.dflow INFO: Standard output for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/7b91a2df-d5a7-4dbe-b04e-5f03e620b9c7_isrForCrosstalkSources_855383_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/7b91a2df-d5a7-4dbe-b04e-5f03e620b9c7_isrForCrosstalkSources_855383_39.stderr parsl.dataflow.dflow DEBUG: Task 1542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1542 has memoization hash 18d278424da16c7315987cc3bacbac48 parsl.dataflow.memoization INFO: Task 1542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1542 try 0 launched on executor multi with executor id 1543 parsl.dataflow.dflow INFO: Standard output for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/523baaa9-1b63-4424-aa68-f5d58f1586a8_isrForCrosstalkSources_855390_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/523baaa9-1b63-4424-aa68-f5d58f1586a8_isrForCrosstalkSources_855390_29.stderr parsl.dataflow.dflow DEBUG: Task 1543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1543 has memoization hash 786a954c71848a95ddb361766e4306b5 parsl.dataflow.memoization INFO: Task 1543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1543 try 0 launched on executor multi with executor id 1544 parsl.dataflow.dflow INFO: Standard output for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bd500bbb-ad57-4819-bf82-c41a984166e9_isrForCrosstalkSources_855443_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bd500bbb-ad57-4819-bf82-c41a984166e9_isrForCrosstalkSources_855443_9.stderr parsl.dataflow.dflow DEBUG: Task 1544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1544 has memoization hash f8d1a890f71c9a23871e4036dae8d7ca parsl.dataflow.memoization INFO: Task 1544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1544 try 0 launched on executor multi with executor id 1545 parsl.dataflow.dflow INFO: Standard output for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3273138-d17d-4a52-a926-cf5f6dcfd51a_isrForCrosstalkSources_855414_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3273138-d17d-4a52-a926-cf5f6dcfd51a_isrForCrosstalkSources_855414_12.stderr parsl.dataflow.dflow DEBUG: Task 1545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1545 has memoization hash f75298542a956e416631a6e5eefb7256 parsl.dataflow.memoization INFO: Task 1545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d262a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1545 try 0 launched on executor multi with executor id 1546 parsl.dataflow.dflow INFO: Standard output for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0e5efe77-a5d5-4de1-982e-ae0c232038dd_isrForCrosstalkSources_855387_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0e5efe77-a5d5-4de1-982e-ae0c232038dd_isrForCrosstalkSources_855387_59.stderr parsl.dataflow.dflow DEBUG: Task 1546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1546 has memoization hash a0bfb19761171e6d3ee080abea64f4fa parsl.dataflow.memoization INFO: Task 1546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1546 try 0 launched on executor multi with executor id 1547 parsl.dataflow.dflow INFO: Standard output for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b63a10bf-3dc2-4c4f-a1d7-8a61f6fcc546_isrForCrosstalkSources_855410_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b63a10bf-3dc2-4c4f-a1d7-8a61f6fcc546_isrForCrosstalkSources_855410_37.stderr parsl.dataflow.dflow DEBUG: Task 1547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1547 has memoization hash 9f315c495064cd9fd176822fadc22741 parsl.dataflow.memoization INFO: Task 1547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d263e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1547 try 0 launched on executor multi with executor id 1548 parsl.dataflow.dflow INFO: Standard output for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1346c674-573a-4db2-a222-dc51e3a8ba61_isrForCrosstalkSources_855397_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1346c674-573a-4db2-a222-dc51e3a8ba61_isrForCrosstalkSources_855397_32.stderr parsl.dataflow.dflow DEBUG: Task 1548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1548 has memoization hash cae6a422e78a43f1108211cc5c041cce parsl.dataflow.memoization INFO: Task 1548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1548 try 0 launched on executor multi with executor id 1549 parsl.dataflow.dflow INFO: Standard output for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32b6a40f-2108-4c4e-820a-ac3979e2457c_isrForCrosstalkSources_855448_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32b6a40f-2108-4c4e-820a-ac3979e2457c_isrForCrosstalkSources_855448_40.stderr parsl.dataflow.dflow DEBUG: Task 1549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1549 has memoization hash 638362570183beb962716abf0b279803 parsl.dataflow.memoization INFO: Task 1549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d260c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1549 try 0 launched on executor multi with executor id 1550 parsl.dataflow.dflow INFO: Standard output for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cfc4a8c8-7e36-4880-b364-65222d6ae3c8_isrForCrosstalkSources_855387_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cfc4a8c8-7e36-4880-b364-65222d6ae3c8_isrForCrosstalkSources_855387_49.stderr parsl.dataflow.dflow DEBUG: Task 1550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1550 has memoization hash e415a9f9458942521510570d0476faf9 parsl.dataflow.memoization INFO: Task 1550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1550 try 0 launched on executor multi with executor id 1551 parsl.dataflow.dflow INFO: Standard output for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/11a6c670-b2c1-44e4-97f4-e6d458dd80d9_isrForCrosstalkSources_855429_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/11a6c670-b2c1-44e4-97f4-e6d458dd80d9_isrForCrosstalkSources_855429_60.stderr parsl.dataflow.dflow DEBUG: Task 1551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1551 has memoization hash b22087f2f3e22aad7f1e905fc384c808 parsl.dataflow.memoization INFO: Task 1551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1551 try 0 launched on executor multi with executor id 1552 parsl.dataflow.dflow INFO: Standard output for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7add53af-d084-4cba-a82b-eaf2fa53c21f_isrForCrosstalkSources_855397_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7add53af-d084-4cba-a82b-eaf2fa53c21f_isrForCrosstalkSources_855397_57.stderr parsl.dataflow.dflow DEBUG: Task 1552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1552 has memoization hash fcb4efa6daf7a7c986b55e7e7db21bd5 parsl.dataflow.memoization INFO: Task 1552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1552 try 0 launched on executor multi with executor id 1553 parsl.dataflow.dflow INFO: Standard output for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/7d27de45-3c97-4ac4-a9d1-68cd6b30d1da_isrForCrosstalkSources_855409_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/7d27de45-3c97-4ac4-a9d1-68cd6b30d1da_isrForCrosstalkSources_855409_50.stderr parsl.dataflow.dflow DEBUG: Task 1553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1553 has memoization hash 89233a0dd995a835ff2599042fba0774 parsl.dataflow.memoization INFO: Task 1553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1553 try 0 launched on executor multi with executor id 1554 parsl.dataflow.dflow INFO: Standard output for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a974c91c-e2de-430f-9e74-c6ac9d138f7d_isrForCrosstalkSources_855404_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a974c91c-e2de-430f-9e74-c6ac9d138f7d_isrForCrosstalkSources_855404_36.stderr parsl.dataflow.dflow DEBUG: Task 1554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1554 has memoization hash a014f5ba2d24ed194c5847837be6f67a parsl.dataflow.memoization INFO: Task 1554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1554 try 0 launched on executor multi with executor id 1555 parsl.dataflow.dflow INFO: Standard output for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e58ab48-262d-4186-b7fb-5cca35ba1be0_isrForCrosstalkSources_855423_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e58ab48-262d-4186-b7fb-5cca35ba1be0_isrForCrosstalkSources_855423_21.stderr parsl.dataflow.dflow DEBUG: Task 1555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1555 has memoization hash ca88d58b01f952fcc0fc7c217799372e parsl.dataflow.memoization INFO: Task 1555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1555 try 0 launched on executor multi with executor id 1556 parsl.dataflow.dflow INFO: Standard output for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac135b95-7c72-44cc-ad32-498e9c7e819a_isrForCrosstalkSources_855401_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac135b95-7c72-44cc-ad32-498e9c7e819a_isrForCrosstalkSources_855401_31.stderr parsl.dataflow.dflow DEBUG: Task 1556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1556 has memoization hash f9dad5e5df268b123cc73365b0720a1f parsl.dataflow.memoization INFO: Task 1556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1556 try 0 launched on executor multi with executor id 1557 parsl.dataflow.dflow INFO: Standard output for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8c9a67bf-2ad3-4405-b287-6a1cb19b87da_isrForCrosstalkSources_855446_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8c9a67bf-2ad3-4405-b287-6a1cb19b87da_isrForCrosstalkSources_855446_50.stderr parsl.dataflow.dflow DEBUG: Task 1557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1557 has memoization hash a9f2e455f540bb1ac384d1f2640aa27e parsl.dataflow.memoization INFO: Task 1557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1557 try 0 launched on executor multi with executor id 1558 parsl.dataflow.dflow INFO: Standard output for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/27adbb8c-108e-4a46-85d6-d258d01a55ef_isrForCrosstalkSources_855458_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/27adbb8c-108e-4a46-85d6-d258d01a55ef_isrForCrosstalkSources_855458_35.stderr parsl.dataflow.dflow DEBUG: Task 1558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1558 has memoization hash bceb115929953207d160004731779e89 parsl.dataflow.memoization INFO: Task 1558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1558 try 0 launched on executor multi with executor id 1559 parsl.dataflow.dflow INFO: Standard output for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d77c9c1f-5ba3-4709-82a6-9f739e7a6fb0_isrForCrosstalkSources_855438_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d77c9c1f-5ba3-4709-82a6-9f739e7a6fb0_isrForCrosstalkSources_855438_59.stderr parsl.dataflow.dflow DEBUG: Task 1559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1559 has memoization hash 0154a2b65b2ddddef3553731e7a47b0f parsl.dataflow.memoization INFO: Task 1559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1559 try 0 launched on executor multi with executor id 1560 parsl.dataflow.dflow INFO: Standard output for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a26034b4-efab-4bf5-b3da-a3b43fe3c85c_isrForCrosstalkSources_855452_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a26034b4-efab-4bf5-b3da-a3b43fe3c85c_isrForCrosstalkSources_855452_14.stderr parsl.dataflow.dflow DEBUG: Task 1560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1560 has memoization hash bd0a62c21b33bbade146683da4456bfc parsl.dataflow.memoization INFO: Task 1560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1560 try 0 launched on executor multi with executor id 1561 parsl.dataflow.dflow INFO: Standard output for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c032f2a7-0bfd-4cb0-80bd-e12cf21d427f_isrForCrosstalkSources_855444_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c032f2a7-0bfd-4cb0-80bd-e12cf21d427f_isrForCrosstalkSources_855444_42.stderr parsl.dataflow.dflow DEBUG: Task 1561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1561 has memoization hash b46e2c998dec45562caceca1c76e6681 parsl.dataflow.memoization INFO: Task 1561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d258a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1561 try 0 launched on executor multi with executor id 1562 parsl.dataflow.dflow INFO: Standard output for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/dc9c1d2c-0c8c-420a-8474-0d005a8317a8_isrForCrosstalkSources_855454_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/dc9c1d2c-0c8c-420a-8474-0d005a8317a8_isrForCrosstalkSources_855454_56.stderr parsl.dataflow.dflow DEBUG: Task 1562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1562 has memoization hash b6bb82f7d5ca4a06bf087468b55b8360 parsl.dataflow.memoization INFO: Task 1562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d259e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1562 try 0 launched on executor multi with executor id 1563 parsl.dataflow.dflow INFO: Standard output for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/00527f69-5fef-4857-81b4-04aa03ea6579_isrForCrosstalkSources_855456_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/00527f69-5fef-4857-81b4-04aa03ea6579_isrForCrosstalkSources_855456_29.stderr parsl.dataflow.dflow DEBUG: Task 1563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1563 has memoization hash e8fcd8fd788df7e3b0a836179ee15697 parsl.dataflow.memoization INFO: Task 1563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d256c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1563 try 0 launched on executor multi with executor id 1564 parsl.dataflow.dflow INFO: Standard output for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5146463b-5c32-4b98-ab7e-de511743cb7e_isrForCrosstalkSources_855403_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5146463b-5c32-4b98-ab7e-de511743cb7e_isrForCrosstalkSources_855403_38.stderr parsl.dataflow.dflow DEBUG: Task 1564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1564 has memoization hash 90931785f8558e8ce1482cf863337121 parsl.dataflow.memoization INFO: Task 1564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1564 try 0 launched on executor multi with executor id 1565 parsl.dataflow.dflow INFO: Standard output for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/12602ff5-7e25-4473-8321-889ccb2c04b8_isrForCrosstalkSources_855426_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/12602ff5-7e25-4473-8321-889ccb2c04b8_isrForCrosstalkSources_855426_13.stderr parsl.dataflow.dflow DEBUG: Task 1565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1565 has memoization hash b16f8de9d4d3dcad72751a65ce90b8f6 parsl.dataflow.memoization INFO: Task 1565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1565 try 0 launched on executor multi with executor id 1566 parsl.dataflow.dflow INFO: Standard output for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/69a60aa5-9a45-4a1d-a894-c588ec595756_isrForCrosstalkSources_855408_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/69a60aa5-9a45-4a1d-a894-c588ec595756_isrForCrosstalkSources_855408_14.stderr parsl.dataflow.dflow DEBUG: Task 1566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1566 has memoization hash 97167f787edb915366eff35225a9a997 parsl.dataflow.memoization INFO: Task 1566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1566 try 0 launched on executor multi with executor id 1567 parsl.dataflow.dflow INFO: Standard output for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b7e404-85ae-4ef5-9e9b-1650f1cf3096_isrForCrosstalkSources_855416_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b7e404-85ae-4ef5-9e9b-1650f1cf3096_isrForCrosstalkSources_855416_15.stderr parsl.dataflow.dflow DEBUG: Task 1567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1567 has memoization hash 8d2cabfcf53e5d51a9c2e63a42fed09f parsl.dataflow.memoization INFO: Task 1567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d254e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1567 try 0 launched on executor multi with executor id 1568 parsl.dataflow.dflow INFO: Standard output for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/228d49ff-d8d0-4a5b-a371-d785090bd582_isrForCrosstalkSources_855390_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/228d49ff-d8d0-4a5b-a371-d785090bd582_isrForCrosstalkSources_855390_34.stderr parsl.dataflow.dflow DEBUG: Task 1568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1568 has memoization hash 76501f48591cb1de82d4ee150f6c8f38 parsl.dataflow.memoization INFO: Task 1568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1568 try 0 launched on executor multi with executor id 1569 parsl.dataflow.dflow INFO: Standard output for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/22bd8083-e906-45bc-aa23-c4396809fa7a_isrForCrosstalkSources_855406_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/22bd8083-e906-45bc-aa23-c4396809fa7a_isrForCrosstalkSources_855406_61.stderr parsl.dataflow.dflow DEBUG: Task 1569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1569 has memoization hash d3156529980239f144d5acd3b2210509 parsl.dataflow.memoization INFO: Task 1569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1569 try 0 launched on executor multi with executor id 1570 parsl.dataflow.dflow INFO: Standard output for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/819fb5a4-9421-4f17-8c69-e54f9341af8d_isrForCrosstalkSources_855400_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/819fb5a4-9421-4f17-8c69-e54f9341af8d_isrForCrosstalkSources_855400_51.stderr parsl.dataflow.dflow DEBUG: Task 1570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1570 has memoization hash 12edb39be866f07c31ae9f8d533b65cd parsl.dataflow.memoization INFO: Task 1570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d25580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1570 try 0 launched on executor multi with executor id 1571 parsl.dataflow.dflow INFO: Standard output for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64c49390-54da-4c7f-af02-a62a37584451_isrForCrosstalkSources_855424_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64c49390-54da-4c7f-af02-a62a37584451_isrForCrosstalkSources_855424_22.stderr parsl.dataflow.dflow DEBUG: Task 1571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1571 has memoization hash 6083aaaf41a9180f2f7c922d52e5532d parsl.dataflow.memoization INFO: Task 1571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1571 try 0 launched on executor multi with executor id 1572 parsl.dataflow.dflow INFO: Standard output for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/581eff3a-2f47-4131-875f-c7ff454d1ec8_isrForCrosstalkSources_855422_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/581eff3a-2f47-4131-875f-c7ff454d1ec8_isrForCrosstalkSources_855422_35.stderr parsl.dataflow.dflow DEBUG: Task 1572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1572 has memoization hash a11cb6638f9a958aa7d51597740b2031 parsl.dataflow.memoization INFO: Task 1572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d268e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964071 proc_lsst.multi INFO: [multi] job_id 16964071 2024-03-09 14:50:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 14:50:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964071 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964071 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16964071 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-09 14:50:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1572 try 0 launched on executor multi with executor id 1573 parsl.dataflow.dflow INFO: Standard output for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/ebace9a5-72ad-4969-88f0-94d28d56dcf9_isrForCrosstalkSources_855456_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/ebace9a5-72ad-4969-88f0-94d28d56dcf9_isrForCrosstalkSources_855456_7.stderr parsl.dataflow.dflow DEBUG: Task 1573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1573 has memoization hash 4dd907a6fc9a993e8ad91093dbf93cb6 parsl.dataflow.memoization INFO: Task 1573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1573 try 0 launched on executor multi with executor id 1574 parsl.dataflow.dflow INFO: Standard output for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f7f1fa85-1c2f-4729-93c0-6e743eeeeefb_isrForCrosstalkSources_855416_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f7f1fa85-1c2f-4729-93c0-6e743eeeeefb_isrForCrosstalkSources_855416_3.stderr parsl.dataflow.dflow DEBUG: Task 1574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1574 has memoization hash 5d0a015e960051a6fa936f19ac172772 parsl.dataflow.memoization INFO: Task 1574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d265c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1574 try 0 launched on executor multi with executor id 1575 parsl.dataflow.dflow INFO: Standard output for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c5945b7b-f560-4a90-9056-dfe7d39dd3d3_isrForCrosstalkSources_855389_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c5945b7b-f560-4a90-9056-dfe7d39dd3d3_isrForCrosstalkSources_855389_44.stderr parsl.dataflow.dflow DEBUG: Task 1575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1575 has memoization hash 62f9a580e0b49d739630be3b633a2cb9 parsl.dataflow.memoization INFO: Task 1575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7cc7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1575 try 0 launched on executor multi with executor id 1576 parsl.dataflow.dflow INFO: Standard output for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e11eadc8-aeee-44b7-a838-4f3ec5ef0296_isrForCrosstalkSources_855391_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e11eadc8-aeee-44b7-a838-4f3ec5ef0296_isrForCrosstalkSources_855391_24.stderr parsl.dataflow.dflow DEBUG: Task 1576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1576 has memoization hash 47f397f0cf9e5254e3e45bd39968c91b parsl.dataflow.memoization INFO: Task 1576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1576 try 0 launched on executor multi with executor id 1577 parsl.dataflow.dflow INFO: Standard output for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2c2cb593-6e83-4da7-9af8-f9afb2683b4b_isrForCrosstalkSources_855457_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2c2cb593-6e83-4da7-9af8-f9afb2683b4b_isrForCrosstalkSources_855457_46.stderr parsl.dataflow.dflow DEBUG: Task 1577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1577 has memoization hash ba036bc3be26651023989afd049a4cdc parsl.dataflow.memoization INFO: Task 1577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1577 try 0 launched on executor multi with executor id 1578 parsl.dataflow.dflow INFO: Standard output for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8c4d424d-ad91-4c1a-8b59-c5fe2e8fc7a0_isrForCrosstalkSources_855395_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8c4d424d-ad91-4c1a-8b59-c5fe2e8fc7a0_isrForCrosstalkSources_855395_41.stderr parsl.dataflow.dflow DEBUG: Task 1578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1578 has memoization hash 470610a527d22bdf8ceaa580ab65a3ab parsl.dataflow.memoization INFO: Task 1578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1578 try 0 launched on executor multi with executor id 1579 parsl.dataflow.dflow INFO: Standard output for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1712e0d2-f64d-4f6e-ab17-3baa9ac02d1e_isrForCrosstalkSources_855428_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1712e0d2-f64d-4f6e-ab17-3baa9ac02d1e_isrForCrosstalkSources_855428_30.stderr parsl.dataflow.dflow DEBUG: Task 1579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1579 has memoization hash 3d1bb162337f0d856cc4701c1c0456a3 parsl.dataflow.memoization INFO: Task 1579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1579 try 0 launched on executor multi with executor id 1580 parsl.dataflow.dflow INFO: Standard output for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aacf0d71-6ef0-4724-ba20-51d06b38ef55_isrForCrosstalkSources_855426_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aacf0d71-6ef0-4724-ba20-51d06b38ef55_isrForCrosstalkSources_855426_27.stderr parsl.dataflow.dflow DEBUG: Task 1580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1580 has memoization hash 06b3d240173982cb9801ef788f3ebc39 parsl.dataflow.memoization INFO: Task 1580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1580 try 0 launched on executor multi with executor id 1581 parsl.dataflow.dflow INFO: Standard output for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9879b347-14db-4226-aaaf-1f87cf329755_isrForCrosstalkSources_855389_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9879b347-14db-4226-aaaf-1f87cf329755_isrForCrosstalkSources_855389_35.stderr parsl.dataflow.dflow DEBUG: Task 1581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1581 has memoization hash 499fe39ceda4f7900167073fc7554ca9 parsl.dataflow.memoization INFO: Task 1581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d277e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1581 try 0 launched on executor multi with executor id 1582 parsl.dataflow.dflow INFO: Standard output for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a0e23a70-9b12-4736-920c-34b62b31eed8_isrForCrosstalkSources_855395_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a0e23a70-9b12-4736-920c-34b62b31eed8_isrForCrosstalkSources_855395_8.stderr parsl.dataflow.dflow DEBUG: Task 1582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1582 has memoization hash 3955c6cc7bf2411edbc6fc106c2f74e8 parsl.dataflow.memoization INFO: Task 1582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1582 try 0 launched on executor multi with executor id 1583 parsl.dataflow.dflow INFO: Standard output for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0c40509-c6af-4c50-a9f0-78f7aa174604_isrForCrosstalkSources_855408_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0c40509-c6af-4c50-a9f0-78f7aa174604_isrForCrosstalkSources_855408_56.stderr parsl.dataflow.dflow DEBUG: Task 1583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1583 has memoization hash 0d19f752df8d6eb5296357a04e1a11a5 parsl.dataflow.memoization INFO: Task 1583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d279c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1583 try 0 launched on executor multi with executor id 1584 parsl.dataflow.dflow INFO: Standard output for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7446fc0b-af74-40b4-830d-0d1fb69122f6_isrForCrosstalkSources_855393_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7446fc0b-af74-40b4-830d-0d1fb69122f6_isrForCrosstalkSources_855393_6.stderr parsl.dataflow.dflow DEBUG: Task 1584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1584 has memoization hash dcce67e71ad6617960fb1f72969a1370 parsl.dataflow.memoization INFO: Task 1584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d276a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1584 try 0 launched on executor multi with executor id 1585 parsl.dataflow.dflow INFO: Standard output for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fefe9b90-1c9c-4809-8f28-10dd7ff7682e_isrForCrosstalkSources_855385_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fefe9b90-1c9c-4809-8f28-10dd7ff7682e_isrForCrosstalkSources_855385_22.stderr parsl.dataflow.dflow DEBUG: Task 1585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1585 has memoization hash bc78d9b637eaa9a874d5601914037f59 parsl.dataflow.memoization INFO: Task 1585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1585 try 0 launched on executor multi with executor id 1586 parsl.dataflow.dflow INFO: Standard output for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4b09ecc4-cd08-4644-b707-aad2c1617683_isrForCrosstalkSources_855389_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4b09ecc4-cd08-4644-b707-aad2c1617683_isrForCrosstalkSources_855389_24.stderr parsl.dataflow.dflow DEBUG: Task 1586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1586 has memoization hash 3b2b5dab7733f5cb18c4e37c4921ccf4 parsl.dataflow.memoization INFO: Task 1586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1586 try 0 launched on executor multi with executor id 1587 parsl.dataflow.dflow INFO: Standard output for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8d20eb3c-feb3-4e28-846b-f4d15a1a3756_isrForCrosstalkSources_855408_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8d20eb3c-feb3-4e28-846b-f4d15a1a3756_isrForCrosstalkSources_855408_12.stderr parsl.dataflow.dflow DEBUG: Task 1587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1587 has memoization hash ce3c3ca9b630b741a5f2303e01bb6aa9 parsl.dataflow.memoization INFO: Task 1587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d274c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1587 try 0 launched on executor multi with executor id 1588 parsl.dataflow.dflow INFO: Standard output for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ea32baf6-d850-4c13-af69-631a1cb647e8_isrForCrosstalkSources_855396_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ea32baf6-d850-4c13-af69-631a1cb647e8_isrForCrosstalkSources_855396_33.stderr parsl.dataflow.dflow DEBUG: Task 1588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1588 has memoization hash 9cc7dbbaf6206c6110b045554c246b3f parsl.dataflow.memoization INFO: Task 1588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1588 try 0 launched on executor multi with executor id 1589 parsl.dataflow.dflow INFO: Standard output for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fa28ce68-162a-4cc9-b0db-0dc0b7cec237_isrForCrosstalkSources_855420_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fa28ce68-162a-4cc9-b0db-0dc0b7cec237_isrForCrosstalkSources_855420_11.stderr parsl.dataflow.dflow DEBUG: Task 1589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1589 has memoization hash f92f14ad325f0ff397636e713b3c5549 parsl.dataflow.memoization INFO: Task 1589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1589 try 0 launched on executor multi with executor id 1590 parsl.dataflow.dflow INFO: Standard output for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/08f36281-6dcb-46f3-a61c-96df9703f129_isrForCrosstalkSources_855455_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/08f36281-6dcb-46f3-a61c-96df9703f129_isrForCrosstalkSources_855455_50.stderr parsl.dataflow.dflow DEBUG: Task 1590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1590 has memoization hash 78e5b8febc9e0b6e175bd719077afdab parsl.dataflow.memoization INFO: Task 1590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1590 try 0 launched on executor multi with executor id 1591 parsl.dataflow.dflow INFO: Standard output for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/04c31364-febb-4154-ae4c-a9858a969e23_isrForCrosstalkSources_855426_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/04c31364-febb-4154-ae4c-a9858a969e23_isrForCrosstalkSources_855426_1.stderr parsl.dataflow.dflow DEBUG: Task 1591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1591 has memoization hash 9172a15df8b7b5aa0c2ddd063417144e parsl.dataflow.memoization INFO: Task 1591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d272e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1591 try 0 launched on executor multi with executor id 1592 parsl.dataflow.dflow INFO: Standard output for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3ec14f21-a786-4e20-b31b-0f996df837b8_isrForCrosstalkSources_855386_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3ec14f21-a786-4e20-b31b-0f996df837b8_isrForCrosstalkSources_855386_57.stderr parsl.dataflow.dflow DEBUG: Task 1592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1592 has memoization hash 306a2286a0c6fa3fa5032e81eb18630a parsl.dataflow.memoization INFO: Task 1592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1592 try 0 launched on executor multi with executor id 1593 parsl.dataflow.dflow INFO: Standard output for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a72d6aec-28b3-4cfb-a3dc-e1442b9ac0d5_isrForCrosstalkSources_855431_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a72d6aec-28b3-4cfb-a3dc-e1442b9ac0d5_isrForCrosstalkSources_855431_5.stderr parsl.dataflow.dflow DEBUG: Task 1593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1593 has memoization hash d9175d05730cb29cbbeae005f4e79ad8 parsl.dataflow.memoization INFO: Task 1593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1593 try 0 launched on executor multi with executor id 1594 parsl.dataflow.dflow INFO: Standard output for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d75f4b4c-5ca7-4f5e-8eed-c6aff9532db2_isrForCrosstalkSources_855417_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d75f4b4c-5ca7-4f5e-8eed-c6aff9532db2_isrForCrosstalkSources_855417_26.stderr parsl.dataflow.dflow DEBUG: Task 1594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1594 has memoization hash 8f021d27ed49270e6ead965611bc0b60 parsl.dataflow.memoization INFO: Task 1594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1594 try 0 launched on executor multi with executor id 1595 parsl.dataflow.dflow INFO: Standard output for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/079a7deb-eed3-4598-90a7-68dd8c317c28_isrForCrosstalkSources_855453_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/079a7deb-eed3-4598-90a7-68dd8c317c28_isrForCrosstalkSources_855453_27.stderr parsl.dataflow.dflow DEBUG: Task 1595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1595 has memoization hash 8e3c21a5ef29efb789234eb38bd0f262 parsl.dataflow.memoization INFO: Task 1595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1595 try 0 launched on executor multi with executor id 1596 parsl.dataflow.dflow INFO: Standard output for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/753c04a6-5a41-4167-8267-508234a21d98_isrForCrosstalkSources_855406_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/753c04a6-5a41-4167-8267-508234a21d98_isrForCrosstalkSources_855406_21.stderr parsl.dataflow.dflow DEBUG: Task 1596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1596 has memoization hash d7dc0d03d8409f054c7b0db9eb9e589f parsl.dataflow.memoization INFO: Task 1596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d271a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1596 try 0 launched on executor multi with executor id 1597 parsl.dataflow.dflow INFO: Standard output for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/baf6598c-f3a2-4179-9284-53600cd31cfb_isrForCrosstalkSources_855427_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/baf6598c-f3a2-4179-9284-53600cd31cfb_isrForCrosstalkSources_855427_16.stderr parsl.dataflow.dflow DEBUG: Task 1597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1597 has memoization hash 7efcb5ba0fc698ad94497dd21b35ee45 parsl.dataflow.memoization INFO: Task 1597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1597 try 0 launched on executor multi with executor id 1598 parsl.dataflow.dflow INFO: Standard output for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2d5d00c9-1286-4520-b7f5-7d5eb47cffeb_isrForCrosstalkSources_855396_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2d5d00c9-1286-4520-b7f5-7d5eb47cffeb_isrForCrosstalkSources_855396_61.stderr parsl.dataflow.dflow DEBUG: Task 1598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1598 has memoization hash 40c8b85195d2417f0def0225c7da762d parsl.dataflow.memoization INFO: Task 1598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1598 try 0 launched on executor multi with executor id 1599 parsl.dataflow.dflow INFO: Standard output for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/5001c8b2-d62c-4d25-a0aa-83f34a6f3fbe_isrForCrosstalkSources_855447_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/5001c8b2-d62c-4d25-a0aa-83f34a6f3fbe_isrForCrosstalkSources_855447_31.stderr parsl.dataflow.dflow DEBUG: Task 1599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1599 has memoization hash dbbd8c23a4d0818a0788348b0fc41294 parsl.dataflow.memoization INFO: Task 1599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1599 try 0 launched on executor multi with executor id 1600 parsl.dataflow.dflow INFO: Standard output for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b289ea68-2f27-4129-998d-152530921f86_isrForCrosstalkSources_855407_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b289ea68-2f27-4129-998d-152530921f86_isrForCrosstalkSources_855407_37.stderr parsl.dataflow.dflow DEBUG: Task 1600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1600 has memoization hash e6254911fee447ee959db5b6f16c7844 parsl.dataflow.memoization INFO: Task 1600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1600 try 0 launched on executor multi with executor id 1601 parsl.dataflow.dflow INFO: Standard output for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2363e20a-2c30-4b9b-ab13-c81fa46dcc1d_isrForCrosstalkSources_855418_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2363e20a-2c30-4b9b-ab13-c81fa46dcc1d_isrForCrosstalkSources_855418_47.stderr parsl.dataflow.dflow DEBUG: Task 1601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1601 has memoization hash ab79107f6e5b6573383c4b0ef38f0f1c parsl.dataflow.memoization INFO: Task 1601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1601 try 0 launched on executor multi with executor id 1602 parsl.dataflow.dflow INFO: Standard output for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/7a1250a2-8a27-4f85-a5fe-41e175f7acc2_isrForCrosstalkSources_855416_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/7a1250a2-8a27-4f85-a5fe-41e175f7acc2_isrForCrosstalkSources_855416_34.stderr parsl.dataflow.dflow DEBUG: Task 1602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1602 has memoization hash 39113511d016b77a78a68f4011ad7524 parsl.dataflow.memoization INFO: Task 1602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1602 try 0 launched on executor multi with executor id 1603 parsl.dataflow.dflow INFO: Standard output for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9e005463-3055-40e6-ad92-d0b3188fa1b2_isrForCrosstalkSources_855411_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9e005463-3055-40e6-ad92-d0b3188fa1b2_isrForCrosstalkSources_855411_61.stderr parsl.dataflow.dflow DEBUG: Task 1603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1603 has memoization hash 181eece884b121a23da13fc836e93894 parsl.dataflow.memoization INFO: Task 1603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1603 try 0 launched on executor multi with executor id 1604 parsl.dataflow.dflow INFO: Standard output for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ca438309-7033-40e3-b0de-27bc6e89a1c3_isrForCrosstalkSources_855450_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ca438309-7033-40e3-b0de-27bc6e89a1c3_isrForCrosstalkSources_855450_41.stderr parsl.dataflow.dflow DEBUG: Task 1604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1604 has memoization hash 7728dd5ef25f35df829373efca0a38ce parsl.dataflow.memoization INFO: Task 1604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1604 try 0 launched on executor multi with executor id 1605 parsl.dataflow.dflow INFO: Standard output for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00c4acbe-f676-4409-809b-b398d7c108c7_isrForCrosstalkSources_855390_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00c4acbe-f676-4409-809b-b398d7c108c7_isrForCrosstalkSources_855390_18.stderr parsl.dataflow.dflow DEBUG: Task 1605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1605 has memoization hash 9cff6bf1b03bfbd051221ade339de6f1 parsl.dataflow.memoization INFO: Task 1605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1605 try 0 launched on executor multi with executor id 1606 parsl.dataflow.dflow INFO: Standard output for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bf18d034-020c-44c5-9b60-a6f738d602b9_isrForCrosstalkSources_855412_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bf18d034-020c-44c5-9b60-a6f738d602b9_isrForCrosstalkSources_855412_6.stderr parsl.dataflow.dflow DEBUG: Task 1606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1606 has memoization hash bcc917e8664c0ef200227efe6d89e42b parsl.dataflow.memoization INFO: Task 1606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1606 try 0 launched on executor multi with executor id 1607 parsl.dataflow.dflow INFO: Standard output for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/ef3fea41-a052-438b-b957-43ffe6d30c7f_isrForCrosstalkSources_855387_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/ef3fea41-a052-438b-b957-43ffe6d30c7f_isrForCrosstalkSources_855387_43.stderr parsl.dataflow.dflow DEBUG: Task 1607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1607 has memoization hash 125276193578c31e609d0cfa18b5ddaa parsl.dataflow.memoization INFO: Task 1607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1607 try 0 launched on executor multi with executor id 1608 parsl.dataflow.dflow INFO: Standard output for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/d6741511-7c34-4619-876a-57d572f9f818_isrForCrosstalkSources_855393_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/d6741511-7c34-4619-876a-57d572f9f818_isrForCrosstalkSources_855393_7.stderr parsl.dataflow.dflow DEBUG: Task 1608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1608 has memoization hash 55a7623ac08a53dba0c064ec8fe2076c parsl.dataflow.memoization INFO: Task 1608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1608 try 0 launched on executor multi with executor id 1609 parsl.dataflow.dflow INFO: Standard output for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6d491b2-0c09-4f29-8d81-bb0f81ee2f91_isrForCrosstalkSources_855380_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6d491b2-0c09-4f29-8d81-bb0f81ee2f91_isrForCrosstalkSources_855380_3.stderr parsl.dataflow.dflow DEBUG: Task 1609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1609 has memoization hash e71e94bbf39d9905a9a5305d9d07b2e1 parsl.dataflow.memoization INFO: Task 1609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1609 try 0 launched on executor multi with executor id 1610 parsl.dataflow.dflow INFO: Standard output for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/51964eed-b3f6-4fc3-9f5c-7b2563ceaaeb_isrForCrosstalkSources_855459_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/51964eed-b3f6-4fc3-9f5c-7b2563ceaaeb_isrForCrosstalkSources_855459_26.stderr parsl.dataflow.dflow DEBUG: Task 1610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1610 has memoization hash 24686cf8b25c1e92f073e81539b79303 parsl.dataflow.memoization INFO: Task 1610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1610 try 0 launched on executor multi with executor id 1611 parsl.dataflow.dflow INFO: Standard output for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/22e6e697-ad3d-4767-b88d-eab0b3a8ede7_isrForCrosstalkSources_855423_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/22e6e697-ad3d-4767-b88d-eab0b3a8ede7_isrForCrosstalkSources_855423_45.stderr parsl.dataflow.dflow DEBUG: Task 1611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1611 has memoization hash 7cb1ab5a506e8d7fd244d634b19d22e8 parsl.dataflow.memoization INFO: Task 1611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964072 proc_lsst.multi INFO: [multi] job_id 16964072 2024-03-09 14:50:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 14:50:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964072 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964072 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16964072 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-09 14:50:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 14:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1611 try 0 launched on executor multi with executor id 1612 parsl.dataflow.dflow INFO: Standard output for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/52ae6f11-cbf7-4ac3-afea-6f473e8f530a_isrForCrosstalkSources_855392_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/52ae6f11-cbf7-4ac3-afea-6f473e8f530a_isrForCrosstalkSources_855392_61.stderr parsl.dataflow.dflow DEBUG: Task 1612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1612 has memoization hash 38b9545ad188e1ec1b9f7a7fca46416a parsl.dataflow.memoization INFO: Task 1612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1612 try 0 launched on executor multi with executor id 1613 parsl.dataflow.dflow INFO: Standard output for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9d50c72-a5d8-4057-8967-e067c11fa666_isrForCrosstalkSources_855458_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9d50c72-a5d8-4057-8967-e067c11fa666_isrForCrosstalkSources_855458_25.stderr parsl.dataflow.dflow DEBUG: Task 1613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1613 has memoization hash 746448d1a60ca7f5d0a2a6dd82d58f1a parsl.dataflow.memoization INFO: Task 1613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d24900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1613 try 0 launched on executor multi with executor id 1614 parsl.dataflow.dflow INFO: Standard output for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/3d424ffd-669c-4ac3-b4a5-d1c01f63d32a_isrForCrosstalkSources_855421_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/3d424ffd-669c-4ac3-b4a5-d1c01f63d32a_isrForCrosstalkSources_855421_25.stderr parsl.dataflow.dflow DEBUG: Task 1614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1614 has memoization hash 6076c6d845335e86f4e9f8fe8f715eca parsl.dataflow.memoization INFO: Task 1614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d26a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1614 try 0 launched on executor multi with executor id 1615 parsl.dataflow.dflow INFO: Standard output for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1706d8d8-6601-4009-af19-7900ec9aa905_isrForCrosstalkSources_855424_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1706d8d8-6601-4009-af19-7900ec9aa905_isrForCrosstalkSources_855424_38.stderr parsl.dataflow.dflow DEBUG: Task 1615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1615 has memoization hash 045206164c7f1614396296181af95800 parsl.dataflow.memoization INFO: Task 1615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1615 try 0 launched on executor multi with executor id 1616 parsl.dataflow.dflow INFO: Standard output for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cce2dcd1-cd22-40f4-83e1-36b1fb2ca186_isrForCrosstalkSources_855415_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cce2dcd1-cd22-40f4-83e1-36b1fb2ca186_isrForCrosstalkSources_855415_47.stderr parsl.dataflow.dflow DEBUG: Task 1616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1616 has memoization hash d2e85df481ada8712a37447f99e5a080 parsl.dataflow.memoization INFO: Task 1616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1616 try 0 launched on executor multi with executor id 1617 parsl.dataflow.dflow INFO: Standard output for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/980111e6-f4d2-4283-aa20-5ad4687b951c_isrForCrosstalkSources_855427_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/980111e6-f4d2-4283-aa20-5ad4687b951c_isrForCrosstalkSources_855427_24.stderr parsl.dataflow.dflow DEBUG: Task 1617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1617 has memoization hash 682f40a48dfb2801f5fec575b068af40 parsl.dataflow.memoization INFO: Task 1617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1617 try 0 launched on executor multi with executor id 1618 parsl.dataflow.dflow INFO: Standard output for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/74d4a24e-3be4-4afc-a19b-abc079f1d168_isrForCrosstalkSources_855392_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/74d4a24e-3be4-4afc-a19b-abc079f1d168_isrForCrosstalkSources_855392_9.stderr parsl.dataflow.dflow DEBUG: Task 1618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1618 has memoization hash fd8167cce75ebaec99a9f1b712ec1a61 parsl.dataflow.memoization INFO: Task 1618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1618 try 0 launched on executor multi with executor id 1619 parsl.dataflow.dflow INFO: Standard output for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4013b9ad-8f28-40c8-9821-82d4176da316_isrForCrosstalkSources_855385_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4013b9ad-8f28-40c8-9821-82d4176da316_isrForCrosstalkSources_855385_14.stderr parsl.dataflow.dflow DEBUG: Task 1619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1619 has memoization hash d28b9f6a00d63292cee0eb8c1d8155c6 parsl.dataflow.memoization INFO: Task 1619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1619 try 0 launched on executor multi with executor id 1620 parsl.dataflow.dflow INFO: Standard output for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/8a8e9522-0e74-48fe-ab99-eff873ce6152_isrForCrosstalkSources_855380_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/8a8e9522-0e74-48fe-ab99-eff873ce6152_isrForCrosstalkSources_855380_8.stderr parsl.dataflow.dflow DEBUG: Task 1620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1620 has memoization hash 2e1715cafae690400178f13021a900c2 parsl.dataflow.memoization INFO: Task 1620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1620 try 0 launched on executor multi with executor id 1621 parsl.dataflow.dflow INFO: Standard output for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3900f274-fe0d-4d26-ac62-cdaabc48c02a_isrForCrosstalkSources_855416_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3900f274-fe0d-4d26-ac62-cdaabc48c02a_isrForCrosstalkSources_855416_17.stderr parsl.dataflow.dflow DEBUG: Task 1621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1621 has memoization hash a611c1444c2e931a555809ec28a73d99 parsl.dataflow.memoization INFO: Task 1621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19034cc880f5b1c8da06eb82e7068765 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Parsl task 1621 try 0 launched on executor multi with executor id 1622 parsl.dataflow.dflow INFO: Standard output for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/14e9cb5d-7819-4cb9-b64c-5e92b38ec39a_isrForCrosstalkSources_855400_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/14e9cb5d-7819-4cb9-b64c-5e92b38ec39a_isrForCrosstalkSources_855400_59.stderr parsl.dataflow.dflow DEBUG: Task 1622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1622 has memoization hash 699e1c207beb117adf330fda58ea8a8d parsl.dataflow.memoization INFO: Task 1622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d783b8e7-8d12-4ef3-a53e-458f878d0843_isrForCrosstalkSources_855457_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d783b8e7-8d12-4ef3-a53e-458f878d0843_isrForCrosstalkSources_855457_41.stderr parsl.dataflow.dflow INFO: Parsl task 1622 try 0 launched on executor multi with executor id 1623 parsl.dataflow.dflow INFO: Standard output for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8e9866a2-0623-47ac-9baa-6a8f3bacebe7_isrForCrosstalkSources_855428_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8e9866a2-0623-47ac-9baa-6a8f3bacebe7_isrForCrosstalkSources_855428_37.stderr parsl.dataflow.dflow DEBUG: Task 1623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1623 has memoization hash a1017c720456a9bbaff0639b16a9315d parsl.dataflow.memoization INFO: Task 1623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1623 try 0 launched on executor multi with executor id 1624 parsl.dataflow.dflow INFO: Standard output for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/08e7fcaa-fc38-4abb-a540-012768f2d398_isrForCrosstalkSources_855434_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/08e7fcaa-fc38-4abb-a540-012768f2d398_isrForCrosstalkSources_855434_10.stderr parsl.dataflow.dflow DEBUG: Task 1624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1624 has memoization hash 133e6b3279cc5b6b43029432fb8cade6 parsl.dataflow.memoization INFO: Task 1624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1624 try 0 launched on executor multi with executor id 1625 parsl.dataflow.dflow INFO: Standard output for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ca223f9d-0a26-4031-8081-7a1b8ce227b9_isrForCrosstalkSources_855433_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ca223f9d-0a26-4031-8081-7a1b8ce227b9_isrForCrosstalkSources_855433_5.stderr parsl.dataflow.dflow DEBUG: Task 1625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1625 has memoization hash 6b72693a00d237cad5d4bdd266acbff9 parsl.dataflow.memoization INFO: Task 1625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1625 try 0 launched on executor multi with executor id 1626 parsl.dataflow.dflow INFO: Standard output for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9a81517f-4db0-4b04-9402-43a3f13cd132_isrForCrosstalkSources_855448_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9a81517f-4db0-4b04-9402-43a3f13cd132_isrForCrosstalkSources_855448_47.stderr parsl.dataflow.dflow DEBUG: Task 1626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1626 has memoization hash d68f04d36abd990b7fb534702c51d429 parsl.dataflow.memoization INFO: Task 1626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1626 try 0 launched on executor multi with executor id 1627 parsl.dataflow.dflow INFO: Standard output for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3f55fb9b-4a1a-4748-869c-4151b6366f2c_isrForCrosstalkSources_855405_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3f55fb9b-4a1a-4748-869c-4151b6366f2c_isrForCrosstalkSources_855405_7.stderr parsl.dataflow.dflow DEBUG: Task 1627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1627 has memoization hash 41b5956780873dc98fcc7aab3d68e6d9 parsl.dataflow.memoization INFO: Task 1627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1627 try 0 launched on executor multi with executor id 1628 parsl.dataflow.dflow INFO: Standard output for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8f5c2d7a-436e-4f50-b874-a289c1b4aee1_isrForCrosstalkSources_855402_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8f5c2d7a-436e-4f50-b874-a289c1b4aee1_isrForCrosstalkSources_855402_21.stderr parsl.dataflow.dflow DEBUG: Task 1628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1628 has memoization hash 8cd37757ecb23338868f7068673ddb54 parsl.dataflow.memoization INFO: Task 1628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1628 try 0 launched on executor multi with executor id 1629 parsl.dataflow.dflow INFO: Standard output for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5e330391-98f5-4b5d-b611-b8a746104e88_isrForCrosstalkSources_855444_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5e330391-98f5-4b5d-b611-b8a746104e88_isrForCrosstalkSources_855444_13.stderr parsl.dataflow.dflow DEBUG: Task 1629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1629 has memoization hash 59d38d3b9d1852c6d172427d45d99eb5 parsl.dataflow.memoization INFO: Task 1629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1629 try 0 launched on executor multi with executor id 1630 parsl.dataflow.dflow INFO: Standard output for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ff3e6171-1a96-4362-b360-0f52cb130e48_isrForCrosstalkSources_855394_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ff3e6171-1a96-4362-b360-0f52cb130e48_isrForCrosstalkSources_855394_22.stderr parsl.dataflow.dflow DEBUG: Task 1630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1630 has memoization hash c71e3fb0e7c51da84152db2db1aa7c96 parsl.dataflow.memoization INFO: Task 1630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1630 try 0 launched on executor multi with executor id 1631 parsl.dataflow.dflow INFO: Standard output for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ddd60077-ad04-449d-90ac-bd3128698487_isrForCrosstalkSources_855383_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ddd60077-ad04-449d-90ac-bd3128698487_isrForCrosstalkSources_855383_46.stderr parsl.dataflow.dflow DEBUG: Task 1631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1631 has memoization hash 1a4b5aabe8002cdf5176dfcdeb6ef11e parsl.dataflow.memoization INFO: Task 1631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1631 try 0 launched on executor multi with executor id 1632 parsl.dataflow.dflow INFO: Standard output for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/80a769f8-b156-4a4f-b180-0addc208d711_isrForCrosstalkSources_855430_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/80a769f8-b156-4a4f-b180-0addc208d711_isrForCrosstalkSources_855430_53.stderr parsl.dataflow.dflow DEBUG: Task 1632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1632 has memoization hash d3d1fd025e740ecd0e92bb127c6dd49f parsl.dataflow.memoization INFO: Task 1632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1632 try 0 launched on executor multi with executor id 1633 parsl.dataflow.dflow INFO: Standard output for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dfcf2489-8a1e-413a-a15e-333a4c179fa9_isrForCrosstalkSources_855441_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dfcf2489-8a1e-413a-a15e-333a4c179fa9_isrForCrosstalkSources_855441_4.stderr parsl.dataflow.dflow DEBUG: Task 1633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1633 has memoization hash d785cde519b995c0dc8ada92d4930eff parsl.dataflow.memoization INFO: Task 1633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1633 try 0 launched on executor multi with executor id 1634 parsl.dataflow.dflow INFO: Standard output for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aca0594-ec99-499d-86bb-d837e36cdcb2_isrForCrosstalkSources_855407_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aca0594-ec99-499d-86bb-d837e36cdcb2_isrForCrosstalkSources_855407_39.stderr parsl.dataflow.dflow DEBUG: Task 1634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1634 has memoization hash 5e86f54f8ec532764d80568e7cada1ac parsl.dataflow.memoization INFO: Task 1634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1634 try 0 launched on executor multi with executor id 1635 parsl.dataflow.dflow INFO: Standard output for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0653f3bd-4348-480a-9f9c-af0aae846971_isrForCrosstalkSources_855409_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0653f3bd-4348-480a-9f9c-af0aae846971_isrForCrosstalkSources_855409_22.stderr parsl.dataflow.dflow DEBUG: Task 1635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1635 has memoization hash 6871912212fb4684d4bc1905acb1aab1 parsl.dataflow.memoization INFO: Task 1635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1635 try 0 launched on executor multi with executor id 1636 parsl.dataflow.dflow INFO: Standard output for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/fb402c24-68c3-4efd-8f69-68a75c58b1ad_isrForCrosstalkSources_855408_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/fb402c24-68c3-4efd-8f69-68a75c58b1ad_isrForCrosstalkSources_855408_35.stderr parsl.dataflow.dflow DEBUG: Task 1636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1636 has memoization hash ba35be1cd9611458a45a4f641381181f parsl.dataflow.memoization INFO: Task 1636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1636 try 0 launched on executor multi with executor id 1637 parsl.dataflow.dflow INFO: Standard output for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/94eb1664-85a8-4c71-8296-6c8c8810462b_isrForCrosstalkSources_855447_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/94eb1664-85a8-4c71-8296-6c8c8810462b_isrForCrosstalkSources_855447_10.stderr parsl.dataflow.dflow DEBUG: Task 1637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1637 has memoization hash d8a69494bc69360d6c404c668792fe95 parsl.dataflow.memoization INFO: Task 1637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:20 proc_lsst.multi:170 [INFO] [multi] job_id 16964073 proc_lsst.multi INFO: [multi] job_id 16964073 2024-03-09 14:50:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 14:50:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964073 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964073 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16964073 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-09 14:50:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1637 try 0 launched on executor multi with executor id 1638 parsl.dataflow.dflow INFO: Standard output for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/a406a6e0-1b79-47fa-b4f6-62da6c5269e9_isrForCrosstalkSources_855458_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/a406a6e0-1b79-47fa-b4f6-62da6c5269e9_isrForCrosstalkSources_855458_51.stderr parsl.dataflow.dflow DEBUG: Task 1638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1638 has memoization hash 47c3adb4278557cfdc5af83e3f48e45d parsl.dataflow.memoization INFO: Task 1638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1638 try 0 launched on executor multi with executor id 1639 parsl.dataflow.dflow INFO: Standard output for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a6abdd33-c75f-48ca-8250-f49e9bfde36b_isrForCrosstalkSources_855394_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a6abdd33-c75f-48ca-8250-f49e9bfde36b_isrForCrosstalkSources_855394_32.stderr parsl.dataflow.dflow DEBUG: Task 1639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1639 has memoization hash 790c28780344d871d94782e0fdb26cb4 parsl.dataflow.memoization INFO: Task 1639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1639 try 0 launched on executor multi with executor id 1640 parsl.dataflow.dflow INFO: Standard output for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3344b202-6125-47fb-a384-2ded742fe9ff_isrForCrosstalkSources_855425_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3344b202-6125-47fb-a384-2ded742fe9ff_isrForCrosstalkSources_855425_21.stderr parsl.dataflow.dflow DEBUG: Task 1640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1640 has memoization hash 336532fe9a5a6d970e13011a2fce73a1 parsl.dataflow.memoization INFO: Task 1640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1640 try 0 launched on executor multi with executor id 1641 parsl.dataflow.dflow INFO: Standard output for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/08eec2df-d9be-412a-afee-62d21ae5209d_isrForCrosstalkSources_855390_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/08eec2df-d9be-412a-afee-62d21ae5209d_isrForCrosstalkSources_855390_37.stderr parsl.dataflow.dflow DEBUG: Task 1641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1641 has memoization hash 03e18faa93135feef52957a5bc02dcba parsl.dataflow.memoization INFO: Task 1641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1641 try 0 launched on executor multi with executor id 1642 parsl.dataflow.dflow INFO: Standard output for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/55930dcc-fb5c-433f-abda-472e2a530fe3_isrForCrosstalkSources_855427_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/55930dcc-fb5c-433f-abda-472e2a530fe3_isrForCrosstalkSources_855427_52.stderr parsl.dataflow.dflow DEBUG: Task 1642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1642 has memoization hash 691a2f69195c72a12841a5082459387b parsl.dataflow.memoization INFO: Task 1642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1642 try 0 launched on executor multi with executor id 1643 parsl.dataflow.dflow INFO: Standard output for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ba50978-7644-44ef-9e68-5ac6bc3d8268_isrForCrosstalkSources_855383_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ba50978-7644-44ef-9e68-5ac6bc3d8268_isrForCrosstalkSources_855383_43.stderr parsl.dataflow.dflow DEBUG: Task 1643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1643 has memoization hash bb85e22c3eb01ac253f6c6fb093b9b19 parsl.dataflow.memoization INFO: Task 1643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1643 try 0 launched on executor multi with executor id 1644 parsl.dataflow.dflow INFO: Standard output for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/36542a30-5cd0-4f95-b8c6-9ff5f165ad42_isrForCrosstalkSources_855448_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/36542a30-5cd0-4f95-b8c6-9ff5f165ad42_isrForCrosstalkSources_855448_23.stderr parsl.dataflow.dflow DEBUG: Task 1644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1644 has memoization hash 24b4f1a464a4ac241212b3486b9b6994 parsl.dataflow.memoization INFO: Task 1644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1644 try 0 launched on executor multi with executor id 1645 parsl.dataflow.dflow INFO: Standard output for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/54eb0520-4874-43b9-b817-04b9c0100c21_isrForCrosstalkSources_855398_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/54eb0520-4874-43b9-b817-04b9c0100c21_isrForCrosstalkSources_855398_23.stderr parsl.dataflow.dflow DEBUG: Task 1645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1645 has memoization hash 9d0ea6ad4adc395a158bab568a1f302d parsl.dataflow.memoization INFO: Task 1645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1645 try 0 launched on executor multi with executor id 1646 parsl.dataflow.dflow INFO: Standard output for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2ca336ff-b99b-4b1e-b838-477abc0dd105_isrForCrosstalkSources_855395_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2ca336ff-b99b-4b1e-b838-477abc0dd105_isrForCrosstalkSources_855395_34.stderr parsl.dataflow.dflow DEBUG: Task 1646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1646 has memoization hash ca86b6d51756d1200f931ff32deb77d9 parsl.dataflow.memoization INFO: Task 1646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1646 try 0 launched on executor multi with executor id 1647 parsl.dataflow.dflow INFO: Standard output for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/2f9eba95-a0d0-4c93-9c88-1fde030b456d_isrForCrosstalkSources_855433_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/2f9eba95-a0d0-4c93-9c88-1fde030b456d_isrForCrosstalkSources_855433_15.stderr parsl.dataflow.dflow DEBUG: Task 1647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1647 has memoization hash a1a7bf91839cb9dfe6d24e718955be65 parsl.dataflow.memoization INFO: Task 1647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1647 try 0 launched on executor multi with executor id 1648 parsl.dataflow.dflow INFO: Standard output for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/daaf818e-e95b-4817-9781-861726646926_isrForCrosstalkSources_855383_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/daaf818e-e95b-4817-9781-861726646926_isrForCrosstalkSources_855383_55.stderr parsl.dataflow.dflow DEBUG: Task 1648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1648 has memoization hash 868ada754ff1d77f9979550454e28836 parsl.dataflow.memoization INFO: Task 1648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1648 try 0 launched on executor multi with executor id 1649 parsl.dataflow.dflow INFO: Standard output for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/69e7fffa-7222-442f-9318-a11d09380887_isrForCrosstalkSources_855434_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/69e7fffa-7222-442f-9318-a11d09380887_isrForCrosstalkSources_855434_9.stderr parsl.dataflow.dflow DEBUG: Task 1649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1649 has memoization hash 84e3c521e28be6b37e3de35f4cb56178 parsl.dataflow.memoization INFO: Task 1649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1649 try 0 launched on executor multi with executor id 1650 parsl.dataflow.dflow INFO: Standard output for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/0b74d105-a385-4138-87c1-3bbf8b5f205f_isrForCrosstalkSources_855416_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/0b74d105-a385-4138-87c1-3bbf8b5f205f_isrForCrosstalkSources_855416_12.stderr parsl.dataflow.dflow DEBUG: Task 1650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1650 has memoization hash a75a15f94146d462379e7d9f6e84de92 parsl.dataflow.memoization INFO: Task 1650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1650 try 0 launched on executor multi with executor id 1651 parsl.dataflow.dflow INFO: Standard output for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a0d27a91-b4c8-436c-bc21-d1b1e8120e9a_isrForCrosstalkSources_855405_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a0d27a91-b4c8-436c-bc21-d1b1e8120e9a_isrForCrosstalkSources_855405_26.stderr parsl.dataflow.dflow DEBUG: Task 1651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1651 has memoization hash 8444ab750d992e43f0f8f611c858d58f parsl.dataflow.memoization INFO: Task 1651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1651 try 0 launched on executor multi with executor id 1652 parsl.dataflow.dflow INFO: Standard output for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/1c8ccb28-ea3d-4e51-8507-725c3521b795_isrForCrosstalkSources_855430_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/1c8ccb28-ea3d-4e51-8507-725c3521b795_isrForCrosstalkSources_855430_23.stderr parsl.dataflow.dflow DEBUG: Task 1652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1652 has memoization hash 010ed17802c01a42dbb26e5ae3f14b9e parsl.dataflow.memoization INFO: Task 1652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1652 try 0 launched on executor multi with executor id 1653 parsl.dataflow.dflow INFO: Standard output for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/a462c518-7ef4-4df0-8a66-cb19175cbaa3_isrForCrosstalkSources_855413_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/a462c518-7ef4-4df0-8a66-cb19175cbaa3_isrForCrosstalkSources_855413_49.stderr parsl.dataflow.dflow DEBUG: Task 1653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1653 has memoization hash 683d38f7a4ccfe205313b8deb1c9fd6d parsl.dataflow.memoization INFO: Task 1653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1653 try 0 launched on executor multi with executor id 1654 parsl.dataflow.dflow INFO: Standard output for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b406c6c-c3e9-437b-9350-1d3bb9be8e79_isrForCrosstalkSources_855447_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b406c6c-c3e9-437b-9350-1d3bb9be8e79_isrForCrosstalkSources_855447_12.stderr parsl.dataflow.dflow DEBUG: Task 1654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1654 has memoization hash 6b3e8aa8dcb143f82ab3ae24dccf8027 parsl.dataflow.memoization INFO: Task 1654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1654 try 0 launched on executor multi with executor id 1655 parsl.dataflow.dflow INFO: Standard output for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b5ae10fb-ea71-4367-80bf-5994b1ca8efb_isrForCrosstalkSources_855438_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b5ae10fb-ea71-4367-80bf-5994b1ca8efb_isrForCrosstalkSources_855438_18.stderr parsl.dataflow.dflow DEBUG: Task 1655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1655 has memoization hash d2c8e0b1a252ac1377c2dd1271cdf909 parsl.dataflow.memoization INFO: Task 1655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1655 try 0 launched on executor multi with executor id 1656 parsl.dataflow.dflow INFO: Standard output for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/fabfb599-02c2-48de-9349-0b11231ce336_isrForCrosstalkSources_855437_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/fabfb599-02c2-48de-9349-0b11231ce336_isrForCrosstalkSources_855437_57.stderr parsl.dataflow.dflow DEBUG: Task 1656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1656 has memoization hash 243446f3b17073a854f7ca57af486181 parsl.dataflow.memoization INFO: Task 1656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1656 try 0 launched on executor multi with executor id 1657 parsl.dataflow.dflow INFO: Standard output for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6fb1cf1f-3939-4c53-a3d9-a4aae52ec7b1_isrForCrosstalkSources_855420_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6fb1cf1f-3939-4c53-a3d9-a4aae52ec7b1_isrForCrosstalkSources_855420_41.stderr parsl.dataflow.dflow DEBUG: Task 1657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1657 has memoization hash 56e174125b46b5b47fa6a1ff4ad474ae parsl.dataflow.memoization INFO: Task 1657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1657 try 0 launched on executor multi with executor id 1658 parsl.dataflow.dflow INFO: Standard output for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/00a2d0b3-f3de-4734-9d8b-4662b422f0cd_isrForCrosstalkSources_855415_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/00a2d0b3-f3de-4734-9d8b-4662b422f0cd_isrForCrosstalkSources_855415_44.stderr parsl.dataflow.dflow DEBUG: Task 1658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1658 has memoization hash 6ddbcb1e3410b9c7271fe88fff13f7ae parsl.dataflow.memoization INFO: Task 1658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1658 try 0 launched on executor multi with executor id 1659 parsl.dataflow.dflow INFO: Standard output for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e81895a8-47a9-41ce-8299-6a44036c1bc6_isrForCrosstalkSources_855408_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e81895a8-47a9-41ce-8299-6a44036c1bc6_isrForCrosstalkSources_855408_42.stderr parsl.dataflow.dflow DEBUG: Task 1659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1659 has memoization hash fd2780389fe8526003c273e9fdbd6ba6 parsl.dataflow.memoization INFO: Task 1659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1659 try 0 launched on executor multi with executor id 1660 parsl.dataflow.dflow INFO: Standard output for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/838d8c99-052b-4cc9-b70c-dd3328f2c41a_isrForCrosstalkSources_855426_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/838d8c99-052b-4cc9-b70c-dd3328f2c41a_isrForCrosstalkSources_855426_47.stderr parsl.dataflow.dflow DEBUG: Task 1660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1660 has memoization hash 0072a164b48d6088a0c0237b07b4c9ff parsl.dataflow.memoization INFO: Task 1660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1660 try 0 launched on executor multi with executor id 1661 parsl.dataflow.dflow INFO: Standard output for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a68a9cf5-c0ab-4588-a2f4-d72b0b72fe20_isrForCrosstalkSources_855381_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a68a9cf5-c0ab-4588-a2f4-d72b0b72fe20_isrForCrosstalkSources_855381_45.stderr parsl.dataflow.dflow DEBUG: Task 1661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1661 has memoization hash ded388f54e713e9e61367226086ffe37 parsl.dataflow.memoization INFO: Task 1661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1661 try 0 launched on executor multi with executor id 1662 parsl.dataflow.dflow INFO: Standard output for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/34fbdaf4-e501-4b0f-92e7-9318e3d9d208_isrForCrosstalkSources_855384_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/34fbdaf4-e501-4b0f-92e7-9318e3d9d208_isrForCrosstalkSources_855384_20.stderr parsl.dataflow.dflow DEBUG: Task 1662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1662 has memoization hash 2f4f7dedcd727c39a92778844ea724d8 parsl.dataflow.memoization INFO: Task 1662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1662 try 0 launched on executor multi with executor id 1663 parsl.dataflow.dflow INFO: Standard output for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/21df260b-1aa7-4c72-a022-bcc1a87146f6_isrForCrosstalkSources_855417_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/21df260b-1aa7-4c72-a022-bcc1a87146f6_isrForCrosstalkSources_855417_27.stderr parsl.dataflow.dflow DEBUG: Task 1663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1663 has memoization hash bcef23ef19c86e81db4129f86bd6fb4e parsl.dataflow.memoization INFO: Task 1663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1663 try 0 launched on executor multi with executor id 1664 parsl.dataflow.dflow INFO: Standard output for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/cc477949-2725-42ca-91a3-37f9427a9410_isrForCrosstalkSources_855434_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/cc477949-2725-42ca-91a3-37f9427a9410_isrForCrosstalkSources_855434_11.stderr parsl.dataflow.dflow DEBUG: Task 1664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1664 has memoization hash f58ba86b384008f1534c89b038a1abc7 parsl.dataflow.memoization INFO: Task 1664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1664 try 0 launched on executor multi with executor id 1665 parsl.dataflow.dflow INFO: Standard output for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/048e3640-8036-4488-a7e3-ddeec8500658_isrForCrosstalkSources_855428_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/048e3640-8036-4488-a7e3-ddeec8500658_isrForCrosstalkSources_855428_45.stderr parsl.dataflow.dflow DEBUG: Task 1665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1665 has memoization hash c5ef8718085f42bdc8c7ae9a387ab697 parsl.dataflow.memoization INFO: Task 1665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1665 try 0 launched on executor multi with executor id 1666 parsl.dataflow.dflow INFO: Standard output for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/92e1c4c2-552e-49dc-a192-2d5d8bce5b99_isrForCrosstalkSources_855405_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/92e1c4c2-552e-49dc-a192-2d5d8bce5b99_isrForCrosstalkSources_855405_29.stderr parsl.dataflow.dflow DEBUG: Task 1666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1666 has memoization hash 14292994f2c75c4e3c804e606687b7fb parsl.dataflow.memoization INFO: Task 1666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1666 try 0 launched on executor multi with executor id 1667 parsl.dataflow.dflow INFO: Standard output for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/902df79e-96e4-4ca5-bd17-62ed5a71455f_isrForCrosstalkSources_855422_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/902df79e-96e4-4ca5-bd17-62ed5a71455f_isrForCrosstalkSources_855422_61.stderr parsl.dataflow.dflow DEBUG: Task 1667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1667 has memoization hash 78391d9319647ee251b4b8cb5679c3d4 parsl.dataflow.memoization INFO: Task 1667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1667 try 0 launched on executor multi with executor id 1668 parsl.dataflow.dflow INFO: Standard output for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2ee79f94-2f7c-4ed4-8f27-624429e83418_isrForCrosstalkSources_855422_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2ee79f94-2f7c-4ed4-8f27-624429e83418_isrForCrosstalkSources_855422_20.stderr parsl.dataflow.dflow DEBUG: Task 1668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1668 has memoization hash 29c2184fb807a6770cb47f138d0d866b parsl.dataflow.memoization INFO: Task 1668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1668 try 0 launched on executor multi with executor id 1669 parsl.dataflow.dflow INFO: Standard output for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1b99ef52-8e5a-492d-b7da-35dbb5156158_isrForCrosstalkSources_855437_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1b99ef52-8e5a-492d-b7da-35dbb5156158_isrForCrosstalkSources_855437_28.stderr parsl.dataflow.dflow DEBUG: Task 1669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1669 has memoization hash 36763112862df3822d9eeed2dccf21f4 parsl.dataflow.memoization INFO: Task 1669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:20 proc_lsst.multi:170 [INFO] [multi] job_id 16964074 proc_lsst.multi INFO: [multi] job_id 16964074 2024-03-09 14:50:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 14:50:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964074 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964074 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16964074 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-09 14:50:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 14:50:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1669 try 0 launched on executor multi with executor id 1670 parsl.dataflow.dflow INFO: Standard output for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/6b8eac9c-f6d6-4406-a1ae-71151b1a8a82_isrForCrosstalkSources_855384_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/6b8eac9c-f6d6-4406-a1ae-71151b1a8a82_isrForCrosstalkSources_855384_3.stderr parsl.dataflow.dflow DEBUG: Task 1670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1670 has memoization hash e901df9079aceea9f4452477271b61a8 parsl.dataflow.memoization INFO: Task 1670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1670 try 0 launched on executor multi with executor id 1671 parsl.dataflow.dflow INFO: Standard output for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/afc89b97-a142-419a-a6cb-9f1416f21deb_isrForCrosstalkSources_855440_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/afc89b97-a142-419a-a6cb-9f1416f21deb_isrForCrosstalkSources_855440_43.stderr parsl.dataflow.dflow DEBUG: Task 1671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1671 has memoization hash cf8046593f43931ba5dc3b008ccb868f parsl.dataflow.memoization INFO: Task 1671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1671 try 0 launched on executor multi with executor id 1672 parsl.dataflow.dflow INFO: Standard output for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/298add35-747c-4eb8-94c5-55a825ca3528_isrForCrosstalkSources_855407_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/298add35-747c-4eb8-94c5-55a825ca3528_isrForCrosstalkSources_855407_44.stderr parsl.dataflow.dflow DEBUG: Task 1672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1672 has memoization hash 08acfb4f762812106b78426d2bb647f2 parsl.dataflow.memoization INFO: Task 1672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1672 try 0 launched on executor multi with executor id 1673 parsl.dataflow.dflow INFO: Standard output for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/affa07b4-ea68-4281-aea1-cf3f33ad16c5_isrForCrosstalkSources_855402_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/affa07b4-ea68-4281-aea1-cf3f33ad16c5_isrForCrosstalkSources_855402_39.stderr parsl.dataflow.dflow DEBUG: Task 1673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1673 has memoization hash 0051b1e470c45ce5e41227d9c6d0b7d8 parsl.dataflow.memoization INFO: Task 1673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a7d27f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1673 try 0 launched on executor multi with executor id 1674 parsl.dataflow.dflow INFO: Standard output for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/a1579e62-59cc-472b-9a9a-e6abd0c57181_isrForCrosstalkSources_855447_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/a1579e62-59cc-472b-9a9a-e6abd0c57181_isrForCrosstalkSources_855447_28.stderr parsl.dataflow.dflow DEBUG: Task 1674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1674 has memoization hash 138e96d8a314fef852576eed091ef104 parsl.dataflow.memoization INFO: Task 1674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1674 try 0 launched on executor multi with executor id 1675 parsl.dataflow.dflow INFO: Standard output for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b29e330-ce35-4366-aa3f-211e453e8fc0_isrForCrosstalkSources_855398_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b29e330-ce35-4366-aa3f-211e453e8fc0_isrForCrosstalkSources_855398_29.stderr parsl.dataflow.dflow DEBUG: Task 1675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1675 has memoization hash be72964ec072b6b23d33cd5a2ecd0416 parsl.dataflow.memoization INFO: Task 1675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1675 try 0 launched on executor multi with executor id 1676 parsl.dataflow.dflow INFO: Standard output for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d579c26-4040-4dad-8d61-59825f8d3d9a_isrForCrosstalkSources_855444_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d579c26-4040-4dad-8d61-59825f8d3d9a_isrForCrosstalkSources_855444_4.stderr parsl.dataflow.dflow DEBUG: Task 1676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1676 has memoization hash 67f0b8f217cac0cbd96dea832222b3d3 parsl.dataflow.memoization INFO: Task 1676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1676 try 0 launched on executor multi with executor id 1677 parsl.dataflow.dflow INFO: Standard output for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8cf1c856-b7c3-4a64-b83b-1b957224d7e4_isrForCrosstalkSources_855453_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8cf1c856-b7c3-4a64-b83b-1b957224d7e4_isrForCrosstalkSources_855453_25.stderr parsl.dataflow.dflow DEBUG: Task 1677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1677 has memoization hash ca6175eec1121853acb00edae12c4e82 parsl.dataflow.memoization INFO: Task 1677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1677 try 0 launched on executor multi with executor id 1678 parsl.dataflow.dflow INFO: Standard output for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fcbd7c1e-66df-468b-b11a-19e14e50f693_isrForCrosstalkSources_855409_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fcbd7c1e-66df-468b-b11a-19e14e50f693_isrForCrosstalkSources_855409_40.stderr parsl.dataflow.dflow DEBUG: Task 1678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1678 has memoization hash a5bed85fd461510746ae64ef0250a9bb parsl.dataflow.memoization INFO: Task 1678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1678 try 0 launched on executor multi with executor id 1679 parsl.dataflow.dflow INFO: Standard output for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c5fd62da-d3b3-4c15-9237-b791e7dfca77_isrForCrosstalkSources_855397_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c5fd62da-d3b3-4c15-9237-b791e7dfca77_isrForCrosstalkSources_855397_40.stderr parsl.dataflow.dflow DEBUG: Task 1679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1679 has memoization hash d360bb1eb1732108eebbae17a2fbbd1c parsl.dataflow.memoization INFO: Task 1679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1679 try 0 launched on executor multi with executor id 1680 parsl.dataflow.dflow INFO: Standard output for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/cffaf12a-d6c3-4ca5-818f-f9bcb2154602_isrForCrosstalkSources_855437_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/cffaf12a-d6c3-4ca5-818f-f9bcb2154602_isrForCrosstalkSources_855437_62.stderr parsl.dataflow.dflow DEBUG: Task 1680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1680 has memoization hash 7f2fc3509a1167e206a0c521157c46b4 parsl.dataflow.memoization INFO: Task 1680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1680 try 0 launched on executor multi with executor id 1681 parsl.dataflow.dflow INFO: Standard output for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f79a7d1a-8cde-4b3a-ae80-efd4ff935151_isrForCrosstalkSources_855453_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f79a7d1a-8cde-4b3a-ae80-efd4ff935151_isrForCrosstalkSources_855453_46.stderr parsl.dataflow.dflow DEBUG: Task 1681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1681 has memoization hash 62ab0fc04b046cd08763ed793d52af19 parsl.dataflow.memoization INFO: Task 1681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1681 try 0 launched on executor multi with executor id 1682 parsl.dataflow.dflow INFO: Standard output for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e7a0546c-9acf-4f98-8d9c-12a36781ae6d_isrForCrosstalkSources_855411_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e7a0546c-9acf-4f98-8d9c-12a36781ae6d_isrForCrosstalkSources_855411_3.stderr parsl.dataflow.dflow DEBUG: Task 1682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1682 has memoization hash 46060b239b62749d8868b0c521ef4279 parsl.dataflow.memoization INFO: Task 1682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1682 try 0 launched on executor multi with executor id 1683 parsl.dataflow.dflow INFO: Standard output for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0b4df0c5-2e06-4970-9812-b2b4f513dfe6_isrForCrosstalkSources_855424_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0b4df0c5-2e06-4970-9812-b2b4f513dfe6_isrForCrosstalkSources_855424_36.stderr parsl.dataflow.dflow DEBUG: Task 1683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1683 has memoization hash 772f9ec448263b3ade5fe92ed430e036 parsl.dataflow.memoization INFO: Task 1683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1683 try 0 launched on executor multi with executor id 1684 parsl.dataflow.dflow INFO: Standard output for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/cd28c47b-24c8-4b37-b204-11dbc786d18b_isrForCrosstalkSources_855457_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/cd28c47b-24c8-4b37-b204-11dbc786d18b_isrForCrosstalkSources_855457_56.stderr parsl.dataflow.dflow DEBUG: Task 1684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1684 has memoization hash b3387595304fa98aac585a57d4e0a7ca parsl.dataflow.memoization INFO: Task 1684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1684 try 0 launched on executor multi with executor id 1685 parsl.dataflow.dflow INFO: Standard output for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/293f3222-bede-42f6-b000-fd6d3fac93c1_isrForCrosstalkSources_855446_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/293f3222-bede-42f6-b000-fd6d3fac93c1_isrForCrosstalkSources_855446_1.stderr parsl.dataflow.dflow DEBUG: Task 1685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1685 has memoization hash 03d9e88925022e41cf4a399cd941fe4f parsl.dataflow.memoization INFO: Task 1685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1685 try 0 launched on executor multi with executor id 1686 parsl.dataflow.dflow INFO: Standard output for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb0df5db-f9d0-4dfc-8d3f-57cd72ed3b61_isrForCrosstalkSources_855386_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb0df5db-f9d0-4dfc-8d3f-57cd72ed3b61_isrForCrosstalkSources_855386_50.stderr parsl.dataflow.dflow DEBUG: Task 1686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1686 has memoization hash a0f1b6030b702a9f90b846ab3c33ba50 parsl.dataflow.memoization INFO: Task 1686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1686 try 0 launched on executor multi with executor id 1687 parsl.dataflow.dflow INFO: Standard output for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/86e3efa3-4153-4554-88d1-622cdbd400be_isrForCrosstalkSources_855423_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/86e3efa3-4153-4554-88d1-622cdbd400be_isrForCrosstalkSources_855423_28.stderr parsl.dataflow.dflow DEBUG: Task 1687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1687 has memoization hash 6e29d6232201082bcf5f55833fbbcb89 parsl.dataflow.memoization INFO: Task 1687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1687 try 0 launched on executor multi with executor id 1688 parsl.dataflow.dflow INFO: Standard output for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/b38ce4ee-a64e-4d86-a151-a6418ecebf25_isrForCrosstalkSources_855383_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/b38ce4ee-a64e-4d86-a151-a6418ecebf25_isrForCrosstalkSources_855383_58.stderr parsl.dataflow.dflow DEBUG: Task 1688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1688 has memoization hash bb64e95a183574624ea84cd00a7605da parsl.dataflow.memoization INFO: Task 1688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1688 try 0 launched on executor multi with executor id 1689 parsl.dataflow.dflow INFO: Standard output for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/265739eb-af6e-4fb2-872b-9d67cda32dea_isrForCrosstalkSources_855408_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/265739eb-af6e-4fb2-872b-9d67cda32dea_isrForCrosstalkSources_855408_41.stderr parsl.dataflow.dflow DEBUG: Task 1689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1689 has memoization hash e141845aea96816f475cb07a52b7252c parsl.dataflow.memoization INFO: Task 1689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1689 try 0 launched on executor multi with executor id 1690 parsl.dataflow.dflow INFO: Standard output for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/80a0c7d7-f58f-4733-9b56-cd3709f371e9_isrForCrosstalkSources_855442_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/80a0c7d7-f58f-4733-9b56-cd3709f371e9_isrForCrosstalkSources_855442_40.stderr parsl.dataflow.dflow DEBUG: Task 1690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1690 has memoization hash 5a759bc2a9bd432ac9396b93375033c8 parsl.dataflow.memoization INFO: Task 1690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1690 try 0 launched on executor multi with executor id 1691 parsl.dataflow.dflow INFO: Standard output for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/61cb4592-809b-4006-8849-49ace21aab5c_isrForCrosstalkSources_855426_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/61cb4592-809b-4006-8849-49ace21aab5c_isrForCrosstalkSources_855426_33.stderr parsl.dataflow.dflow DEBUG: Task 1691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1691 has memoization hash 34de3851bd2d8633511dbda65e78455c parsl.dataflow.memoization INFO: Task 1691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1691 try 0 launched on executor multi with executor id 1692 parsl.dataflow.dflow INFO: Standard output for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8ef54967-6965-4c47-bde9-28779d95c952_isrForCrosstalkSources_855441_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8ef54967-6965-4c47-bde9-28779d95c952_isrForCrosstalkSources_855441_13.stderr parsl.dataflow.dflow DEBUG: Task 1692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1692 has memoization hash 9e09a710cd660f53ffa60bee717608b0 parsl.dataflow.memoization INFO: Task 1692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1692 try 0 launched on executor multi with executor id 1693 parsl.dataflow.dflow INFO: Standard output for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/82248b50-c601-44bc-bb2e-8d42105505ae_isrForCrosstalkSources_855387_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/82248b50-c601-44bc-bb2e-8d42105505ae_isrForCrosstalkSources_855387_2.stderr parsl.dataflow.dflow DEBUG: Task 1693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1693 has memoization hash 27d0bc890b784870e6f2a8eaf19de7fb parsl.dataflow.memoization INFO: Task 1693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1693 try 0 launched on executor multi with executor id 1694 parsl.dataflow.dflow INFO: Standard output for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/eacf4fcf-a147-49db-b5cb-2c0e26b3dc9d_isrForCrosstalkSources_855425_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/eacf4fcf-a147-49db-b5cb-2c0e26b3dc9d_isrForCrosstalkSources_855425_27.stderr parsl.dataflow.dflow DEBUG: Task 1694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1694 has memoization hash c0b99ed85c534e27b733c1b843351350 parsl.dataflow.memoization INFO: Task 1694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1694 try 0 launched on executor multi with executor id 1695 parsl.dataflow.dflow INFO: Standard output for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0269c3bf-099a-42cf-add3-5a1de4281538_isrForCrosstalkSources_855430_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0269c3bf-099a-42cf-add3-5a1de4281538_isrForCrosstalkSources_855430_33.stderr parsl.dataflow.dflow DEBUG: Task 1695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1695 has memoization hash 32d3e341da6ca2e29e07b132870a35d7 parsl.dataflow.memoization INFO: Task 1695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1695 try 0 launched on executor multi with executor id 1696 parsl.dataflow.dflow INFO: Standard output for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8c7a55d3-908b-4752-9c62-e8b2547970ff_isrForCrosstalkSources_855449_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8c7a55d3-908b-4752-9c62-e8b2547970ff_isrForCrosstalkSources_855449_24.stderr parsl.dataflow.dflow DEBUG: Task 1696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1696 has memoization hash 515b6a1ede694e12e978da8599416098 parsl.dataflow.memoization INFO: Task 1696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1696 try 0 launched on executor multi with executor id 1697 parsl.dataflow.dflow INFO: Standard output for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8a7688b9-5f10-446c-b223-bb2873598832_isrForCrosstalkSources_855447_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8a7688b9-5f10-446c-b223-bb2873598832_isrForCrosstalkSources_855447_8.stderr parsl.dataflow.dflow DEBUG: Task 1697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1697 has memoization hash d14db5cda72c09288538f6eafa11bd4d parsl.dataflow.memoization INFO: Task 1697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1697 try 0 launched on executor multi with executor id 1698 parsl.dataflow.dflow INFO: Standard output for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60ec4336-a9b6-4d17-84f9-6835540f42e6_isrForCrosstalkSources_855433_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60ec4336-a9b6-4d17-84f9-6835540f42e6_isrForCrosstalkSources_855433_12.stderr parsl.dataflow.dflow DEBUG: Task 1698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1698 has memoization hash aca05f80f1fbacd050454588a1ae88ce parsl.dataflow.memoization INFO: Task 1698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1698 try 0 launched on executor multi with executor id 1699 parsl.dataflow.dflow INFO: Standard output for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b98d7050-aab7-48e5-8dde-8b24f9c56465_isrForCrosstalkSources_855407_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b98d7050-aab7-48e5-8dde-8b24f9c56465_isrForCrosstalkSources_855407_8.stderr parsl.dataflow.dflow DEBUG: Task 1699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1699 has memoization hash 26ab668fa29beda244be42443175ae24 parsl.dataflow.memoization INFO: Task 1699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1699 try 0 launched on executor multi with executor id 1700 parsl.dataflow.dflow INFO: Standard output for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c420ed87-26e8-4edb-b154-f3b4ec9134ba_isrForCrosstalkSources_855417_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c420ed87-26e8-4edb-b154-f3b4ec9134ba_isrForCrosstalkSources_855417_20.stderr parsl.dataflow.dflow DEBUG: Task 1700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1700 has memoization hash 7ea737f3bf23aa7a671de75848852848 parsl.dataflow.memoization INFO: Task 1700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1700 try 0 launched on executor multi with executor id 1701 parsl.dataflow.dflow INFO: Standard output for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/dc66eca6-a5eb-4827-9d5d-6716935aea59_isrForCrosstalkSources_855390_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/dc66eca6-a5eb-4827-9d5d-6716935aea59_isrForCrosstalkSources_855390_26.stderr parsl.dataflow.dflow DEBUG: Task 1701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1701 has memoization hash e8420b7730cfb58e9eb013717394a876 parsl.dataflow.memoization INFO: Task 1701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1701 try 0 launched on executor multi with executor id 1702 parsl.dataflow.dflow INFO: Standard output for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/db8e3bab-d755-40a8-aa83-6d435e7ea652_isrForCrosstalkSources_855446_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/db8e3bab-d755-40a8-aa83-6d435e7ea652_isrForCrosstalkSources_855446_22.stderr parsl.dataflow.dflow DEBUG: Task 1702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1702 has memoization hash 054a96ca816e53de5aa5b551b105ff0f parsl.dataflow.memoization INFO: Task 1702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1702 try 0 launched on executor multi with executor id 1703 parsl.dataflow.dflow INFO: Standard output for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1fc488b8-5907-4556-9ccd-4b1979ed811a_isrForCrosstalkSources_855407_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1fc488b8-5907-4556-9ccd-4b1979ed811a_isrForCrosstalkSources_855407_5.stderr parsl.dataflow.dflow DEBUG: Task 1703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1703 has memoization hash 3b38fc8564263f4a89b7fc3554ec7002 parsl.dataflow.memoization INFO: Task 1703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1703 try 0 launched on executor multi with executor id 1704 parsl.dataflow.dflow INFO: Standard output for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/bcf45702-47a2-4831-b3e2-8b575eb36a23_isrForCrosstalkSources_855399_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/bcf45702-47a2-4831-b3e2-8b575eb36a23_isrForCrosstalkSources_855399_46.stderr parsl.dataflow.dflow DEBUG: Task 1704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1704 has memoization hash 494315d8181ae9b1b858a9d188c14fcf parsl.dataflow.memoization INFO: Task 1704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1704 try 0 launched on executor multi with executor id 1705 parsl.dataflow.dflow INFO: Standard output for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/08a2fead-a820-4ede-86cc-e01b38ed34ef_isrForCrosstalkSources_855385_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/08a2fead-a820-4ede-86cc-e01b38ed34ef_isrForCrosstalkSources_855385_21.stderr parsl.dataflow.dflow DEBUG: Task 1705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1705 has memoization hash 377257d29c35d71a3406eb05573b840d parsl.dataflow.memoization INFO: Task 1705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:21 proc_lsst.multi:170 [INFO] [multi] job_id 16964075 proc_lsst.multi INFO: [multi] job_id 16964075 2024-03-09 14:50:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 14:50:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964075 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964075 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16964075 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-09 14:50:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1705 try 0 launched on executor multi with executor id 1706 parsl.dataflow.dflow INFO: Standard output for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/35371575-5b04-4187-bd7a-416c0cd69f9f_isrForCrosstalkSources_855458_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/35371575-5b04-4187-bd7a-416c0cd69f9f_isrForCrosstalkSources_855458_42.stderr parsl.dataflow.dflow DEBUG: Task 1706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1706 has memoization hash a36f16f0316fa072410aacbbcf52f304 parsl.dataflow.memoization INFO: Task 1706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1706 try 0 launched on executor multi with executor id 1707 parsl.dataflow.dflow INFO: Standard output for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1825a882-058b-4b14-83ce-928cff89d347_isrForCrosstalkSources_855401_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1825a882-058b-4b14-83ce-928cff89d347_isrForCrosstalkSources_855401_24.stderr parsl.dataflow.dflow DEBUG: Task 1707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1707 has memoization hash 57492ed1336c851d1acb0009724f54dc parsl.dataflow.memoization INFO: Task 1707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1707 try 0 launched on executor multi with executor id 1708 parsl.dataflow.dflow INFO: Standard output for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a82eee6b-359e-409c-8c0a-60fb6de5a6e8_isrForCrosstalkSources_855451_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a82eee6b-359e-409c-8c0a-60fb6de5a6e8_isrForCrosstalkSources_855451_14.stderr parsl.dataflow.dflow DEBUG: Task 1708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1708 has memoization hash cef9219459a419cf4d041a0127923fbb parsl.dataflow.memoization INFO: Task 1708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1708 try 0 launched on executor multi with executor id 1709 parsl.dataflow.dflow INFO: Standard output for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d5366984-807a-46ad-8917-dca4162df657_isrForCrosstalkSources_855416_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d5366984-807a-46ad-8917-dca4162df657_isrForCrosstalkSources_855416_59.stderr parsl.dataflow.dflow DEBUG: Task 1709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1709 has memoization hash 1345af2422f943e4fe667bfbc5aaea73 parsl.dataflow.memoization INFO: Task 1709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1709 try 0 launched on executor multi with executor id 1710 parsl.dataflow.dflow INFO: Standard output for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a8aa44a1-8191-4665-b6b9-48f11cddf8a4_isrForCrosstalkSources_855432_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a8aa44a1-8191-4665-b6b9-48f11cddf8a4_isrForCrosstalkSources_855432_48.stderr parsl.dataflow.dflow DEBUG: Task 1710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1710 has memoization hash 22de2e5db8a319d74ed1a0d4fb473474 parsl.dataflow.memoization INFO: Task 1710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1710 try 0 launched on executor multi with executor id 1711 parsl.dataflow.dflow INFO: Standard output for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ad4b6995-d4b4-428d-9dd6-c55dc2df1765_isrForCrosstalkSources_855413_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ad4b6995-d4b4-428d-9dd6-c55dc2df1765_isrForCrosstalkSources_855413_3.stderr parsl.dataflow.dflow DEBUG: Task 1711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1711 has memoization hash aa53e2e584d21963e010301d01e03110 parsl.dataflow.memoization INFO: Task 1711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1711 try 0 launched on executor multi with executor id 1712 parsl.dataflow.dflow INFO: Standard output for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/85555475-f12c-40e5-a62a-5da147f424b4_isrForCrosstalkSources_855403_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/85555475-f12c-40e5-a62a-5da147f424b4_isrForCrosstalkSources_855403_2.stderr parsl.dataflow.dflow DEBUG: Task 1712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1712 has memoization hash b91368d5954019b7d1da365b15199eb3 parsl.dataflow.memoization INFO: Task 1712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1712 try 0 launched on executor multi with executor id 1713 parsl.dataflow.dflow INFO: Standard output for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5cfcf57e-d7e9-4239-a953-4c553d051a29_isrForCrosstalkSources_855413_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5cfcf57e-d7e9-4239-a953-4c553d051a29_isrForCrosstalkSources_855413_52.stderr parsl.dataflow.dflow DEBUG: Task 1713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1713 has memoization hash 783b2ddfd188c7bf5a6a3f19eab33812 parsl.dataflow.memoization INFO: Task 1713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1713 try 0 launched on executor multi with executor id 1714 parsl.dataflow.dflow INFO: Standard output for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c3b41fd2-eea0-4e22-85d8-212f0df63acc_isrForCrosstalkSources_855404_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c3b41fd2-eea0-4e22-85d8-212f0df63acc_isrForCrosstalkSources_855404_16.stderr parsl.dataflow.dflow DEBUG: Task 1714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1714 has memoization hash c82df0b9137dcf69c128aa8a7ebaec81 parsl.dataflow.memoization INFO: Task 1714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1714 try 0 launched on executor multi with executor id 1715 parsl.dataflow.dflow INFO: Standard output for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/95b999a8-330f-4fb9-8a69-f63cfd90ae21_isrForCrosstalkSources_855394_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/95b999a8-330f-4fb9-8a69-f63cfd90ae21_isrForCrosstalkSources_855394_1.stderr parsl.dataflow.dflow DEBUG: Task 1715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1715 has memoization hash acbf3a62e4bcb1882c4b91fd1dc9819a parsl.dataflow.memoization INFO: Task 1715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1715 try 0 launched on executor multi with executor id 1716 parsl.dataflow.dflow INFO: Standard output for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2f5f68d7-35a6-42eb-8265-cd01024994b0_isrForCrosstalkSources_855379_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2f5f68d7-35a6-42eb-8265-cd01024994b0_isrForCrosstalkSources_855379_48.stderr parsl.dataflow.dflow DEBUG: Task 1716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1716 has memoization hash 609573a1f9dc952bf1affb8bf63eb3a7 parsl.dataflow.memoization INFO: Task 1716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1716 try 0 launched on executor multi with executor id 1717 parsl.dataflow.dflow INFO: Standard output for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/408ad9b0-9d0e-49d5-b497-93e6226d5d79_isrForCrosstalkSources_855401_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/408ad9b0-9d0e-49d5-b497-93e6226d5d79_isrForCrosstalkSources_855401_55.stderr parsl.dataflow.dflow DEBUG: Task 1717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1717 has memoization hash 7b5aaa2b1e8688904caa81d4e01f7ab9 parsl.dataflow.memoization INFO: Task 1717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1717 try 0 launched on executor multi with executor id 1718 parsl.dataflow.dflow INFO: Standard output for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/96317402-358a-46a3-99d9-847af6c87701_isrForCrosstalkSources_855410_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/96317402-358a-46a3-99d9-847af6c87701_isrForCrosstalkSources_855410_29.stderr parsl.dataflow.dflow DEBUG: Task 1718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1718 has memoization hash 8d3569a262c5870385ee5a1cc9abda98 parsl.dataflow.memoization INFO: Task 1718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1718 try 0 launched on executor multi with executor id 1719 parsl.dataflow.dflow INFO: Standard output for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ab8d196c-7c79-435e-b71c-59e3e6da9129_isrForCrosstalkSources_855415_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ab8d196c-7c79-435e-b71c-59e3e6da9129_isrForCrosstalkSources_855415_33.stderr parsl.dataflow.dflow DEBUG: Task 1719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1719 has memoization hash ed22d0829471b831d43ca6e51885590e parsl.dataflow.memoization INFO: Task 1719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1719 try 0 launched on executor multi with executor id 1720 parsl.dataflow.dflow INFO: Standard output for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5650c506-2ae2-493b-bc2a-64ae9a06e258_isrForCrosstalkSources_855404_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5650c506-2ae2-493b-bc2a-64ae9a06e258_isrForCrosstalkSources_855404_51.stderr parsl.dataflow.dflow DEBUG: Task 1720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1720 has memoization hash 75aa3d2532eaff64385713b66bfbf71b parsl.dataflow.memoization INFO: Task 1720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1720 try 0 launched on executor multi with executor id 1721 parsl.dataflow.dflow INFO: Standard output for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/98fd582a-7fad-4d17-9310-ad30a207436f_isrForCrosstalkSources_855397_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/98fd582a-7fad-4d17-9310-ad30a207436f_isrForCrosstalkSources_855397_35.stderr parsl.dataflow.dflow DEBUG: Task 1721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1721 has memoization hash 6a48196984b2a9b5d941a6ca9be257d9 parsl.dataflow.memoization INFO: Task 1721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1721 try 0 launched on executor multi with executor id 1722 parsl.dataflow.dflow INFO: Standard output for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/73172a8e-f080-4f43-b3a7-781bc632b218_isrForCrosstalkSources_855396_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/73172a8e-f080-4f43-b3a7-781bc632b218_isrForCrosstalkSources_855396_17.stderr parsl.dataflow.dflow DEBUG: Task 1722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1722 has memoization hash fb32d665162190a92ebbf666de9b9562 parsl.dataflow.memoization INFO: Task 1722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1722 try 0 launched on executor multi with executor id 1723 parsl.dataflow.dflow INFO: Standard output for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e18181f8-a357-4dfd-b06f-490f7b18bc34_isrForCrosstalkSources_855458_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e18181f8-a357-4dfd-b06f-490f7b18bc34_isrForCrosstalkSources_855458_23.stderr parsl.dataflow.dflow DEBUG: Task 1723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1723 has memoization hash 03a3820efac4bdbf7f10c4ed796d8a86 parsl.dataflow.memoization INFO: Task 1723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1723 try 0 launched on executor multi with executor id 1724 parsl.dataflow.dflow INFO: Standard output for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/74dc9071-d140-4bb7-9f0f-24c275564e80_isrForCrosstalkSources_855455_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/74dc9071-d140-4bb7-9f0f-24c275564e80_isrForCrosstalkSources_855455_39.stderr parsl.dataflow.dflow DEBUG: Task 1724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1724 has memoization hash d43b2aba553ab3fcdc74e1507d0c9417 parsl.dataflow.memoization INFO: Task 1724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1724 try 0 launched on executor multi with executor id 1725 parsl.dataflow.dflow INFO: Standard output for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0fc3e80e-4919-4573-ab63-62da76ea80bf_isrForCrosstalkSources_855428_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0fc3e80e-4919-4573-ab63-62da76ea80bf_isrForCrosstalkSources_855428_3.stderr parsl.dataflow.dflow DEBUG: Task 1725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1725 has memoization hash 267f9d23e8b7d9a7cd65f923a890a2ea parsl.dataflow.memoization INFO: Task 1725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1725 try 0 launched on executor multi with executor id 1726 parsl.dataflow.dflow INFO: Standard output for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/65480f8c-4ce3-4c87-b61e-22653815c812_isrForCrosstalkSources_855421_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/65480f8c-4ce3-4c87-b61e-22653815c812_isrForCrosstalkSources_855421_35.stderr parsl.dataflow.dflow DEBUG: Task 1726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1726 has memoization hash 7f536c1afcbac63464bbd548ec5b5334 parsl.dataflow.memoization INFO: Task 1726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1726 try 0 launched on executor multi with executor id 1727 parsl.dataflow.dflow INFO: Standard output for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0892d2a7-4889-4d6d-b4ef-cf188bef40fe_isrForCrosstalkSources_855417_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0892d2a7-4889-4d6d-b4ef-cf188bef40fe_isrForCrosstalkSources_855417_9.stderr parsl.dataflow.dflow DEBUG: Task 1727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1727 has memoization hash 8019578db3d1f88379761b12209b3b23 parsl.dataflow.memoization INFO: Task 1727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1727 try 0 launched on executor multi with executor id 1728 parsl.dataflow.dflow INFO: Standard output for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4fd1b57c-3bd4-4282-b49b-d2b225ec9846_isrForCrosstalkSources_855419_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4fd1b57c-3bd4-4282-b49b-d2b225ec9846_isrForCrosstalkSources_855419_2.stderr parsl.dataflow.dflow DEBUG: Task 1728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1728 has memoization hash f8edbcfabcd120d2061d576372bb67e0 parsl.dataflow.memoization INFO: Task 1728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1728 try 0 launched on executor multi with executor id 1729 parsl.dataflow.dflow INFO: Standard output for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0d91cee1-17f8-4fdc-9b87-06414ace2d88_isrForCrosstalkSources_855404_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0d91cee1-17f8-4fdc-9b87-06414ace2d88_isrForCrosstalkSources_855404_22.stderr parsl.dataflow.dflow DEBUG: Task 1729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1729 has memoization hash ab09c5d266f056b6e92ebe6f8aaf3fe6 parsl.dataflow.memoization INFO: Task 1729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1729 try 0 launched on executor multi with executor id 1730 parsl.dataflow.dflow INFO: Standard output for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/57e2041e-1f88-47ad-9f4f-42d546ffeefd_isrForCrosstalkSources_855385_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/57e2041e-1f88-47ad-9f4f-42d546ffeefd_isrForCrosstalkSources_855385_38.stderr parsl.dataflow.dflow DEBUG: Task 1730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1730 has memoization hash cd01af9b45823307dd3570af851fc8c6 parsl.dataflow.memoization INFO: Task 1730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67accc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1730 try 0 launched on executor multi with executor id 1731 parsl.dataflow.dflow INFO: Standard output for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ad2ff7ba-ccf0-4e1f-bd3a-9d6f4271bb61_isrForCrosstalkSources_855430_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ad2ff7ba-ccf0-4e1f-bd3a-9d6f4271bb61_isrForCrosstalkSources_855430_55.stderr parsl.dataflow.dflow DEBUG: Task 1731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1731 has memoization hash 575cb31d842c68baeb066edb7e3dbc7c parsl.dataflow.memoization INFO: Task 1731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1731 try 0 launched on executor multi with executor id 1732 parsl.dataflow.dflow INFO: Standard output for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/144b1b44-cb43-461b-a17a-ff07f5ec7bb5_isrForCrosstalkSources_855455_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/144b1b44-cb43-461b-a17a-ff07f5ec7bb5_isrForCrosstalkSources_855455_14.stderr parsl.dataflow.dflow DEBUG: Task 1732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1732 has memoization hash 6e25eb3edd295de0bdf507df79ea4fd3 parsl.dataflow.memoization INFO: Task 1732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1732 try 0 launched on executor multi with executor id 1733 parsl.dataflow.dflow INFO: Standard output for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/55e245dd-3dab-4c1d-96f9-43e24b6ccb9e_isrForCrosstalkSources_855401_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/55e245dd-3dab-4c1d-96f9-43e24b6ccb9e_isrForCrosstalkSources_855401_45.stderr parsl.dataflow.dflow DEBUG: Task 1733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1733 has memoization hash e6240a3c5346e349f61e5d2a5376b454 parsl.dataflow.memoization INFO: Task 1733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1733 try 0 launched on executor multi with executor id 1734 parsl.dataflow.dflow INFO: Standard output for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/60d758d9-8221-412d-abf9-1244eedea323_isrForCrosstalkSources_855406_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/60d758d9-8221-412d-abf9-1244eedea323_isrForCrosstalkSources_855406_43.stderr parsl.dataflow.dflow DEBUG: Task 1734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1734 has memoization hash 3472a2261cd732b6c7c43e32d45d9c72 parsl.dataflow.memoization INFO: Task 1734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1734 try 0 launched on executor multi with executor id 1735 parsl.dataflow.dflow INFO: Standard output for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/350813de-7427-4538-bf05-bf8a3be802d7_isrForCrosstalkSources_855383_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/350813de-7427-4538-bf05-bf8a3be802d7_isrForCrosstalkSources_855383_50.stderr parsl.dataflow.dflow DEBUG: Task 1735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1735 has memoization hash 1e912636d8628ed2d357b02c34a552de parsl.dataflow.memoization INFO: Task 1735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1735 try 0 launched on executor multi with executor id 1736 parsl.dataflow.dflow INFO: Standard output for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/44fd24ef-cb85-4272-933d-4b70427b1906_isrForCrosstalkSources_855426_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/44fd24ef-cb85-4272-933d-4b70427b1906_isrForCrosstalkSources_855426_28.stderr parsl.dataflow.dflow DEBUG: Task 1736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1736 has memoization hash 6885c2e55168643e66a5e26de96354e7 parsl.dataflow.memoization INFO: Task 1736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1736 try 0 launched on executor multi with executor id 1737 parsl.dataflow.dflow INFO: Standard output for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0b2eab4e-904d-4d04-8633-f67a929bc488_isrForCrosstalkSources_855402_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0b2eab4e-904d-4d04-8633-f67a929bc488_isrForCrosstalkSources_855402_29.stderr parsl.dataflow.dflow DEBUG: Task 1737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1737 has memoization hash 203762c33a965a96e1398f87375df63f parsl.dataflow.memoization INFO: Task 1737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67acea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1737 try 0 launched on executor multi with executor id 1738 parsl.dataflow.dflow INFO: Standard output for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fe4f1090-1ad8-48da-8c78-f86d8d6ee59c_isrForCrosstalkSources_855459_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fe4f1090-1ad8-48da-8c78-f86d8d6ee59c_isrForCrosstalkSources_855459_10.stderr parsl.dataflow.dflow DEBUG: Task 1738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1738 has memoization hash 86d1e2736dc404e9c4f58ea011e9e18f parsl.dataflow.memoization INFO: Task 1738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1738 try 0 launched on executor multi with executor id 1739 parsl.dataflow.dflow INFO: Standard output for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9809c97e-1063-4145-aa53-516b025ea25f_isrForCrosstalkSources_855386_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9809c97e-1063-4145-aa53-516b025ea25f_isrForCrosstalkSources_855386_1.stderr parsl.dataflow.dflow DEBUG: Task 1739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1739 has memoization hash bc26f193ebdeb3e99764b281f4535598 parsl.dataflow.memoization INFO: Task 1739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1739 try 0 launched on executor multi with executor id 1740 parsl.dataflow.dflow INFO: Standard output for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1499d3cc-61d7-48f2-8f7d-a5cca5bae524_isrForCrosstalkSources_855403_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1499d3cc-61d7-48f2-8f7d-a5cca5bae524_isrForCrosstalkSources_855403_16.stderr parsl.dataflow.dflow DEBUG: Task 1740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1740 has memoization hash b39545a4c058dcb17a9c9010b69b293b parsl.dataflow.memoization INFO: Task 1740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1740 try 0 launched on executor multi with executor id 1741 parsl.dataflow.dflow INFO: Standard output for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/11fea568-f789-4716-858c-c3f425c4742f_isrForCrosstalkSources_855395_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/11fea568-f789-4716-858c-c3f425c4742f_isrForCrosstalkSources_855395_21.stderr parsl.dataflow.dflow DEBUG: Task 1741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1741 has memoization hash bdd2873aa2ea4fc91b433d95868a1277 parsl.dataflow.memoization INFO: Task 1741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1741 try 0 launched on executor multi with executor id 1742 parsl.dataflow.dflow INFO: Standard output for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/98eded79-5774-4173-a265-498850a32124_isrForCrosstalkSources_855425_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/98eded79-5774-4173-a265-498850a32124_isrForCrosstalkSources_855425_48.stderr parsl.dataflow.dflow DEBUG: Task 1742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1742 has memoization hash 959162171b919674722e0a4dbac01edf parsl.dataflow.memoization INFO: Task 1742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:21 proc_lsst.multi:170 [INFO] [multi] job_id 16964076 proc_lsst.multi INFO: [multi] job_id 16964076 2024-03-09 14:50:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 14:50:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964076 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964076 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16964076 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-09 14:50:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 14:50:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1742 try 0 launched on executor multi with executor id 1743 parsl.dataflow.dflow INFO: Standard output for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ba90c059-42b3-4510-a046-295483118435_isrForCrosstalkSources_855452_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ba90c059-42b3-4510-a046-295483118435_isrForCrosstalkSources_855452_40.stderr parsl.dataflow.dflow DEBUG: Task 1743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1743 has memoization hash 0a1202d862ca748ba36633d2b2fb7d67 parsl.dataflow.memoization INFO: Task 1743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1743 try 0 launched on executor multi with executor id 1744 parsl.dataflow.dflow INFO: Standard output for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9cd382b3-a385-4c09-9548-8bbb193d4c77_isrForCrosstalkSources_855428_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9cd382b3-a385-4c09-9548-8bbb193d4c77_isrForCrosstalkSources_855428_39.stderr parsl.dataflow.dflow DEBUG: Task 1744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1744 has memoization hash 87b9a8687a45478150d289c69013de49 parsl.dataflow.memoization INFO: Task 1744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1744 try 0 launched on executor multi with executor id 1745 parsl.dataflow.dflow INFO: Standard output for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4ba63552-2ff3-4253-b5d4-0b1eaf8c1a4b_isrForCrosstalkSources_855408_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4ba63552-2ff3-4253-b5d4-0b1eaf8c1a4b_isrForCrosstalkSources_855408_28.stderr parsl.dataflow.dflow DEBUG: Task 1745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1745 has memoization hash 903e8cecc619049f44e4791611fe45e0 parsl.dataflow.memoization INFO: Task 1745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1745 try 0 launched on executor multi with executor id 1746 parsl.dataflow.dflow INFO: Standard output for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b6a36e57-4821-4c4c-a310-c4293bf3ca6b_isrForCrosstalkSources_855402_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b6a36e57-4821-4c4c-a310-c4293bf3ca6b_isrForCrosstalkSources_855402_61.stderr parsl.dataflow.dflow DEBUG: Task 1746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1746 has memoization hash d06a3f4be59fc0768b25ee5a294f3f1e parsl.dataflow.memoization INFO: Task 1746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1746 try 0 launched on executor multi with executor id 1747 parsl.dataflow.dflow INFO: Standard output for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/945198b5-5785-4fb1-b642-087959952f4c_isrForCrosstalkSources_855441_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/945198b5-5785-4fb1-b642-087959952f4c_isrForCrosstalkSources_855441_53.stderr parsl.dataflow.dflow DEBUG: Task 1747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1747 has memoization hash 7ce316cfc0587a19995e5052981dd783 parsl.dataflow.memoization INFO: Task 1747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1747 try 0 launched on executor multi with executor id 1748 parsl.dataflow.dflow INFO: Standard output for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f96fe12a-4054-48aa-9fdf-bed6922ae76d_isrForCrosstalkSources_855420_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f96fe12a-4054-48aa-9fdf-bed6922ae76d_isrForCrosstalkSources_855420_23.stderr parsl.dataflow.dflow DEBUG: Task 1748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1748 has memoization hash bb10728f4f4aa5555b62c72dc09674bd parsl.dataflow.memoization INFO: Task 1748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1748 try 0 launched on executor multi with executor id 1749 parsl.dataflow.dflow INFO: Standard output for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c5385ff0-3604-49ed-a530-9dd2046b5d65_isrForCrosstalkSources_855419_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c5385ff0-3604-49ed-a530-9dd2046b5d65_isrForCrosstalkSources_855419_30.stderr parsl.dataflow.dflow DEBUG: Task 1749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1749 has memoization hash a1232ce651721d8a82cb79cbc852e9f2 parsl.dataflow.memoization INFO: Task 1749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1749 try 0 launched on executor multi with executor id 1750 parsl.dataflow.dflow INFO: Standard output for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff7ff9b9-9eb8-4344-a3e4-d3c66abc92ac_isrForCrosstalkSources_855405_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff7ff9b9-9eb8-4344-a3e4-d3c66abc92ac_isrForCrosstalkSources_855405_32.stderr parsl.dataflow.dflow DEBUG: Task 1750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1750 has memoization hash c7129bc81426cb203746446eb59c180f parsl.dataflow.memoization INFO: Task 1750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1750 try 0 launched on executor multi with executor id 1751 parsl.dataflow.dflow INFO: Standard output for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0ffe8e2-2e70-4ec6-96b1-7163c86fb6a7_isrForCrosstalkSources_855384_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0ffe8e2-2e70-4ec6-96b1-7163c86fb6a7_isrForCrosstalkSources_855384_62.stderr parsl.dataflow.dflow DEBUG: Task 1751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1751 has memoization hash c449f835c63909c297a039524cadc7ea parsl.dataflow.memoization INFO: Task 1751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1751 try 0 launched on executor multi with executor id 1752 parsl.dataflow.dflow INFO: Standard output for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ce1ed493-0de8-4184-bbdb-4e9fc762a0ba_isrForCrosstalkSources_855403_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ce1ed493-0de8-4184-bbdb-4e9fc762a0ba_isrForCrosstalkSources_855403_7.stderr parsl.dataflow.dflow DEBUG: Task 1752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1752 has memoization hash b4372432ecd107a6b141d11fa68fcf87 parsl.dataflow.memoization INFO: Task 1752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1752 try 0 launched on executor multi with executor id 1753 parsl.dataflow.dflow INFO: Standard output for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/aebe3e93-61ce-4ff5-b4bc-57918e95ad4e_isrForCrosstalkSources_855410_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/aebe3e93-61ce-4ff5-b4bc-57918e95ad4e_isrForCrosstalkSources_855410_7.stderr parsl.dataflow.dflow DEBUG: Task 1753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1753 has memoization hash 9c624e1a8ee03a0cc19f0384108068f2 parsl.dataflow.memoization INFO: Task 1753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1753 try 0 launched on executor multi with executor id 1754 parsl.dataflow.dflow INFO: Standard output for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de1f08af-5d7b-43d8-baf4-55a655068e37_isrForCrosstalkSources_855449_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de1f08af-5d7b-43d8-baf4-55a655068e37_isrForCrosstalkSources_855449_55.stderr parsl.dataflow.dflow DEBUG: Task 1754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1754 has memoization hash ade4032a1fc83a7006b524431a1467d1 parsl.dataflow.memoization INFO: Task 1754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1754 try 0 launched on executor multi with executor id 1755 parsl.dataflow.dflow INFO: Standard output for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1d1b4804-5c6a-4f0a-8b71-7e53ed87b4de_isrForCrosstalkSources_855412_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1d1b4804-5c6a-4f0a-8b71-7e53ed87b4de_isrForCrosstalkSources_855412_8.stderr parsl.dataflow.dflow DEBUG: Task 1755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1755 has memoization hash 7f4933778c92f99c678754aa28853202 parsl.dataflow.memoization INFO: Task 1755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1755 try 0 launched on executor multi with executor id 1756 parsl.dataflow.dflow INFO: Standard output for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e043fe1-b00f-4d84-b7fb-3f714b034101_isrForCrosstalkSources_855384_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e043fe1-b00f-4d84-b7fb-3f714b034101_isrForCrosstalkSources_855384_57.stderr parsl.dataflow.dflow DEBUG: Task 1756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1756 has memoization hash 1d3649496546f6e8dc0f052a8fb844c9 parsl.dataflow.memoization INFO: Task 1756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1756 try 0 launched on executor multi with executor id 1757 parsl.dataflow.dflow INFO: Standard output for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/799044fd-d423-415c-ae73-38243ebe223d_isrForCrosstalkSources_855435_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/799044fd-d423-415c-ae73-38243ebe223d_isrForCrosstalkSources_855435_5.stderr parsl.dataflow.dflow DEBUG: Task 1757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1757 has memoization hash c75523e7b0021163c7d0bdf2fbb644c4 parsl.dataflow.memoization INFO: Task 1757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67adb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1757 try 0 launched on executor multi with executor id 1758 parsl.dataflow.dflow INFO: Standard output for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cedd5363-ccf0-40c9-abf4-624ef9bba6bf_isrForCrosstalkSources_855415_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cedd5363-ccf0-40c9-abf4-624ef9bba6bf_isrForCrosstalkSources_855415_29.stderr parsl.dataflow.dflow DEBUG: Task 1758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1758 has memoization hash 8264ab09e7a91077ca2b9c2529f37423 parsl.dataflow.memoization INFO: Task 1758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1758 try 0 launched on executor multi with executor id 1759 parsl.dataflow.dflow INFO: Standard output for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fede1ab1-f75a-489c-8b82-00433584681e_isrForCrosstalkSources_855423_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fede1ab1-f75a-489c-8b82-00433584681e_isrForCrosstalkSources_855423_46.stderr parsl.dataflow.dflow DEBUG: Task 1759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1759 has memoization hash 3a8ebb890b1b04667df094da07476eb7 parsl.dataflow.memoization INFO: Task 1759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1759 try 0 launched on executor multi with executor id 1760 parsl.dataflow.dflow INFO: Standard output for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/96de1389-fb7e-4225-8fec-64a8a8d29bf6_isrForCrosstalkSources_855399_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/96de1389-fb7e-4225-8fec-64a8a8d29bf6_isrForCrosstalkSources_855399_31.stderr parsl.dataflow.dflow DEBUG: Task 1760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1760 has memoization hash d4b0f0efb6917a3187fd54e49f197042 parsl.dataflow.memoization INFO: Task 1760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67adc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1760 try 0 launched on executor multi with executor id 1761 parsl.dataflow.dflow INFO: Standard output for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ae52cbac-66e1-49d9-923b-6bb40a62de5a_isrForCrosstalkSources_855449_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ae52cbac-66e1-49d9-923b-6bb40a62de5a_isrForCrosstalkSources_855449_53.stderr parsl.dataflow.dflow DEBUG: Task 1761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1761 has memoization hash 2222e6b5b2e64daa14b00d68b61f3307 parsl.dataflow.memoization INFO: Task 1761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1761 try 0 launched on executor multi with executor id 1762 parsl.dataflow.dflow INFO: Standard output for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7c46d2eb-766b-4244-9222-c424f10f9b19_isrForCrosstalkSources_855413_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7c46d2eb-766b-4244-9222-c424f10f9b19_isrForCrosstalkSources_855413_13.stderr parsl.dataflow.dflow DEBUG: Task 1762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1762 has memoization hash cba04fd236d7d92adee1cb84caa23a09 parsl.dataflow.memoization INFO: Task 1762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ade40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1762 try 0 launched on executor multi with executor id 1763 parsl.dataflow.dflow INFO: Standard output for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/87728572-59e9-4264-8077-055a8ac01a1d_isrForCrosstalkSources_855450_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/87728572-59e9-4264-8077-055a8ac01a1d_isrForCrosstalkSources_855450_21.stderr parsl.dataflow.dflow DEBUG: Task 1763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1763 has memoization hash 423ffd6fdd8e08262ba79558dc6a0ffa parsl.dataflow.memoization INFO: Task 1763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67adda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1763 try 0 launched on executor multi with executor id 1764 parsl.dataflow.dflow INFO: Standard output for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6a49aa4f-c0c2-47ff-9cc5-99d321d96d56_isrForCrosstalkSources_855380_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6a49aa4f-c0c2-47ff-9cc5-99d321d96d56_isrForCrosstalkSources_855380_56.stderr parsl.dataflow.dflow DEBUG: Task 1764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1764 has memoization hash b5c30de376fd155560b87b8b62ae7e90 parsl.dataflow.memoization INFO: Task 1764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1764 try 0 launched on executor multi with executor id 1765 parsl.dataflow.dflow INFO: Standard output for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/25cad6c7-4f36-4ce3-8f88-e44379c04649_isrForCrosstalkSources_855450_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/25cad6c7-4f36-4ce3-8f88-e44379c04649_isrForCrosstalkSources_855450_31.stderr parsl.dataflow.dflow DEBUG: Task 1765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1765 has memoization hash 296626acd8c785cd786f1f7cf17dd98f parsl.dataflow.memoization INFO: Task 1765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1765 try 0 launched on executor multi with executor id 1766 parsl.dataflow.dflow INFO: Standard output for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/fc08558a-4b03-4d8f-bd46-c9e99944c0b4_isrForCrosstalkSources_855421_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/fc08558a-4b03-4d8f-bd46-c9e99944c0b4_isrForCrosstalkSources_855421_39.stderr parsl.dataflow.dflow DEBUG: Task 1766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1766 has memoization hash 646a340b73a56bb428e64546786f6d6e parsl.dataflow.memoization INFO: Task 1766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67adf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1766 try 0 launched on executor multi with executor id 1767 parsl.dataflow.dflow INFO: Standard output for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04ec688b-6542-40a4-99f6-c00a3a89a859_isrForCrosstalkSources_855412_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04ec688b-6542-40a4-99f6-c00a3a89a859_isrForCrosstalkSources_855412_47.stderr parsl.dataflow.dflow DEBUG: Task 1767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1767 has memoization hash c71b09b5de2c40c451d21394fd59866a parsl.dataflow.memoization INFO: Task 1767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1767 try 0 launched on executor multi with executor id 1768 parsl.dataflow.dflow INFO: Standard output for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2310f16b-cef8-49a2-a2ea-ca7f2655e064_isrForCrosstalkSources_855382_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2310f16b-cef8-49a2-a2ea-ca7f2655e064_isrForCrosstalkSources_855382_35.stderr parsl.dataflow.dflow DEBUG: Task 1768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1768 has memoization hash f590e2850b484fd949bdc3319b5d3a04 parsl.dataflow.memoization INFO: Task 1768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1768 try 0 launched on executor multi with executor id 1769 parsl.dataflow.dflow INFO: Standard output for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/86ff2a3e-0a67-47e1-8354-0832c6d76e55_isrForCrosstalkSources_855393_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/86ff2a3e-0a67-47e1-8354-0832c6d76e55_isrForCrosstalkSources_855393_60.stderr parsl.dataflow.dflow DEBUG: Task 1769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1769 has memoization hash c2b7fe819b5f906a88247b99f6be2526 parsl.dataflow.memoization INFO: Task 1769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1769 try 0 launched on executor multi with executor id 1770 parsl.dataflow.dflow INFO: Standard output for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d56a0fea-a2ba-4850-a1d4-1f974fcb6ad6_isrForCrosstalkSources_855444_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d56a0fea-a2ba-4850-a1d4-1f974fcb6ad6_isrForCrosstalkSources_855444_62.stderr parsl.dataflow.dflow DEBUG: Task 1770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1770 has memoization hash 1f49bf88fa89be3e021cd8c67c969933 parsl.dataflow.memoization INFO: Task 1770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1770 try 0 launched on executor multi with executor id 1771 parsl.dataflow.dflow INFO: Standard output for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/71b10347-ee5b-4bab-b570-2376c0895018_isrForCrosstalkSources_855380_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/71b10347-ee5b-4bab-b570-2376c0895018_isrForCrosstalkSources_855380_17.stderr parsl.dataflow.dflow DEBUG: Task 1771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1771 has memoization hash cf4a0ec8b98755e1e101913b0d63cd97 parsl.dataflow.memoization INFO: Task 1771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1771 try 0 launched on executor multi with executor id 1772 parsl.dataflow.dflow INFO: Standard output for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f8fe032a-650e-40e2-bb9b-7f1907a5a649_isrForCrosstalkSources_855384_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f8fe032a-650e-40e2-bb9b-7f1907a5a649_isrForCrosstalkSources_855384_27.stderr parsl.dataflow.dflow DEBUG: Task 1772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1772 has memoization hash ac389a8ff592628dd99239dd8b2917b8 parsl.dataflow.memoization INFO: Task 1772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1772 try 0 launched on executor multi with executor id 1773 parsl.dataflow.dflow INFO: Standard output for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/688b11f6-1bed-4fd6-981f-e4340b11aecf_isrForCrosstalkSources_855452_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/688b11f6-1bed-4fd6-981f-e4340b11aecf_isrForCrosstalkSources_855452_51.stderr parsl.dataflow.dflow DEBUG: Task 1773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1773 has memoization hash f3a7276d7d3c2b37769b5e920e15a8ec parsl.dataflow.memoization INFO: Task 1773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1773 try 0 launched on executor multi with executor id 1774 parsl.dataflow.dflow INFO: Standard output for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ec85978b-4bd0-4443-b869-870f14fc0b49_isrForCrosstalkSources_855381_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ec85978b-4bd0-4443-b869-870f14fc0b49_isrForCrosstalkSources_855381_24.stderr parsl.dataflow.dflow DEBUG: Task 1774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1774 has memoization hash fd85dddd7ff0aa45d667f88d5064c6b4 parsl.dataflow.memoization INFO: Task 1774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1774 try 0 launched on executor multi with executor id 1775 parsl.dataflow.dflow INFO: Standard output for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f1999fdb-0a39-4fea-8dc0-4385003f2e1a_isrForCrosstalkSources_855420_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f1999fdb-0a39-4fea-8dc0-4385003f2e1a_isrForCrosstalkSources_855420_56.stderr parsl.dataflow.dflow DEBUG: Task 1775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1775 has memoization hash 1e35c7cd7316c66160c7fa2875bb83ab parsl.dataflow.memoization INFO: Task 1775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1775 try 0 launched on executor multi with executor id 1776 parsl.dataflow.dflow INFO: Standard output for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/277b0d74-ad69-4db1-b16b-1eafe4e34014_isrForCrosstalkSources_855410_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/277b0d74-ad69-4db1-b16b-1eafe4e34014_isrForCrosstalkSources_855410_42.stderr parsl.dataflow.dflow DEBUG: Task 1776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1776 has memoization hash eef6bed81031b173d739a5d91bcd2e38 parsl.dataflow.memoization INFO: Task 1776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1776 try 0 launched on executor multi with executor id 1777 parsl.dataflow.dflow INFO: Standard output for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b714a4d3-c007-4170-a2d3-654111aa21fa_isrForCrosstalkSources_855454_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b714a4d3-c007-4170-a2d3-654111aa21fa_isrForCrosstalkSources_855454_12.stderr parsl.dataflow.dflow DEBUG: Task 1777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1777 has memoization hash 3f820705ca714b77c204ccc74b457e63 parsl.dataflow.memoization INFO: Task 1777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:22 proc_lsst.multi:170 [INFO] [multi] job_id 16964077 proc_lsst.multi INFO: [multi] job_id 16964077 2024-03-09 14:50:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 14:50:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964077 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964077 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16964077 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-09 14:50:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1777 try 0 launched on executor multi with executor id 1778 parsl.dataflow.dflow INFO: Standard output for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/4632e42e-ef8e-43cf-95e1-1929ab140f1e_isrForCrosstalkSources_855444_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/4632e42e-ef8e-43cf-95e1-1929ab140f1e_isrForCrosstalkSources_855444_45.stderr parsl.dataflow.dflow DEBUG: Task 1778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1778 has memoization hash 6cc15637379a2384ee085ed12cba32a4 parsl.dataflow.memoization INFO: Task 1778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1778 try 0 launched on executor multi with executor id 1779 parsl.dataflow.dflow INFO: Standard output for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/050287c7-5bee-4886-b278-c54fdc9d6981_isrForCrosstalkSources_855398_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/050287c7-5bee-4886-b278-c54fdc9d6981_isrForCrosstalkSources_855398_32.stderr parsl.dataflow.dflow DEBUG: Task 1779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1779 has memoization hash 4c15ed4a31829ad9ec60d795d9d9cb3b parsl.dataflow.memoization INFO: Task 1779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1779 try 0 launched on executor multi with executor id 1780 parsl.dataflow.dflow INFO: Standard output for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/77b28a06-daba-4f41-bba6-95cdbe9a339f_isrForCrosstalkSources_855417_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/77b28a06-daba-4f41-bba6-95cdbe9a339f_isrForCrosstalkSources_855417_52.stderr parsl.dataflow.dflow DEBUG: Task 1780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1780 has memoization hash ad2ddebfde8811eb7bd76e5679037a5b parsl.dataflow.memoization INFO: Task 1780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1780 try 0 launched on executor multi with executor id 1781 parsl.dataflow.dflow INFO: Standard output for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6474e59c-aad6-4ceb-ae67-67b8058ac657_isrForCrosstalkSources_855440_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6474e59c-aad6-4ceb-ae67-67b8058ac657_isrForCrosstalkSources_855440_32.stderr parsl.dataflow.dflow DEBUG: Task 1781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1781 has memoization hash 60ecdcef6972c0e14b8d48222dcc033f parsl.dataflow.memoization INFO: Task 1781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1781 try 0 launched on executor multi with executor id 1782 parsl.dataflow.dflow INFO: Standard output for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/857ebb41-15af-4d7c-9a47-6eae3a6941ca_isrForCrosstalkSources_855426_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/857ebb41-15af-4d7c-9a47-6eae3a6941ca_isrForCrosstalkSources_855426_5.stderr parsl.dataflow.dflow DEBUG: Task 1782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1782 has memoization hash 6f3fb433b72239827d93497d83f0e2fa parsl.dataflow.memoization INFO: Task 1782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1782 try 0 launched on executor multi with executor id 1783 parsl.dataflow.dflow INFO: Standard output for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/bf6a51de-0071-4421-8f86-0af91faf8c99_isrForCrosstalkSources_855390_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/bf6a51de-0071-4421-8f86-0af91faf8c99_isrForCrosstalkSources_855390_6.stderr parsl.dataflow.dflow DEBUG: Task 1783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1783 has memoization hash 93f28dfe6cf5be0f7113ab6adef2d675 parsl.dataflow.memoization INFO: Task 1783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1783 try 0 launched on executor multi with executor id 1784 parsl.dataflow.dflow INFO: Standard output for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8c0431f-be2a-4055-81fb-76a22d77ab01_isrForCrosstalkSources_855450_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8c0431f-be2a-4055-81fb-76a22d77ab01_isrForCrosstalkSources_855450_23.stderr parsl.dataflow.dflow DEBUG: Task 1784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1784 has memoization hash b24123168b639e599e06b91bd271bcae parsl.dataflow.memoization INFO: Task 1784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1784 try 0 launched on executor multi with executor id 1785 parsl.dataflow.dflow INFO: Standard output for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bdec0b49-bc7e-4976-a487-72edd78c6203_isrForCrosstalkSources_855380_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bdec0b49-bc7e-4976-a487-72edd78c6203_isrForCrosstalkSources_855380_37.stderr parsl.dataflow.dflow DEBUG: Task 1785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1785 has memoization hash 6038dd3745a2c25ab0a132dee345d535 parsl.dataflow.memoization INFO: Task 1785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1785 try 0 launched on executor multi with executor id 1786 parsl.dataflow.dflow INFO: Standard output for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0cbbabae-67f7-4971-8194-89a54e8a625d_isrForCrosstalkSources_855454_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0cbbabae-67f7-4971-8194-89a54e8a625d_isrForCrosstalkSources_855454_33.stderr parsl.dataflow.dflow DEBUG: Task 1786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1786 has memoization hash 845ac5ace1dc994c0490cb47cd527f3d parsl.dataflow.memoization INFO: Task 1786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1786 try 0 launched on executor multi with executor id 1787 parsl.dataflow.dflow INFO: Standard output for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/df4a6195-dd91-4361-a484-43b6018812a8_isrForCrosstalkSources_855419_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/df4a6195-dd91-4361-a484-43b6018812a8_isrForCrosstalkSources_855419_48.stderr parsl.dataflow.dflow DEBUG: Task 1787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1787 has memoization hash a89c12396d21f4ada22d6d597b5a527a parsl.dataflow.memoization INFO: Task 1787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1787 try 0 launched on executor multi with executor id 1788 parsl.dataflow.dflow INFO: Standard output for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/74805213-a793-4f2e-a764-33ca8a8eb45d_isrForCrosstalkSources_855440_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/74805213-a793-4f2e-a764-33ca8a8eb45d_isrForCrosstalkSources_855440_17.stderr parsl.dataflow.dflow DEBUG: Task 1788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1788 has memoization hash 9db400145be06612166cf273a77749c3 parsl.dataflow.memoization INFO: Task 1788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1788 try 0 launched on executor multi with executor id 1789 parsl.dataflow.dflow INFO: Standard output for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ee4a4fcc-e5b2-40f9-8ddf-30b8046925e9_isrForCrosstalkSources_855413_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ee4a4fcc-e5b2-40f9-8ddf-30b8046925e9_isrForCrosstalkSources_855413_56.stderr parsl.dataflow.dflow DEBUG: Task 1789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1789 has memoization hash baa068fb62192fe9f7c3040c0a0b3d61 parsl.dataflow.memoization INFO: Task 1789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1789 try 0 launched on executor multi with executor id 1790 parsl.dataflow.dflow INFO: Standard output for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f816d250-83ef-4f25-b3c8-21c8be0ab492_isrForCrosstalkSources_855397_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f816d250-83ef-4f25-b3c8-21c8be0ab492_isrForCrosstalkSources_855397_49.stderr parsl.dataflow.dflow DEBUG: Task 1790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1790 has memoization hash d909a259b0bbd2989a423e5047e3e672 parsl.dataflow.memoization INFO: Task 1790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1790 try 0 launched on executor multi with executor id 1791 parsl.dataflow.dflow INFO: Standard output for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/de1cea4a-7554-4141-b90f-6871f35ea3b8_isrForCrosstalkSources_855454_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/de1cea4a-7554-4141-b90f-6871f35ea3b8_isrForCrosstalkSources_855454_18.stderr parsl.dataflow.dflow DEBUG: Task 1791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1791 has memoization hash 50c338d2cb851f5fd09ff2bfcab860fa parsl.dataflow.memoization INFO: Task 1791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1791 try 0 launched on executor multi with executor id 1792 parsl.dataflow.dflow INFO: Standard output for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b6fe4a45-0d0b-47cb-a27b-efb6ae0eef7a_isrForCrosstalkSources_855428_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b6fe4a45-0d0b-47cb-a27b-efb6ae0eef7a_isrForCrosstalkSources_855428_26.stderr parsl.dataflow.dflow DEBUG: Task 1792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1792 has memoization hash b351b5809831c01028a36f6b75993ed7 parsl.dataflow.memoization INFO: Task 1792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1792 try 0 launched on executor multi with executor id 1793 parsl.dataflow.dflow INFO: Standard output for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/7920339d-1b00-4337-b1e0-f6ce81b0942b_isrForCrosstalkSources_855394_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/7920339d-1b00-4337-b1e0-f6ce81b0942b_isrForCrosstalkSources_855394_57.stderr parsl.dataflow.dflow DEBUG: Task 1793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1793 has memoization hash 137dc3ee7c68f56fd0bc676030536752 parsl.dataflow.memoization INFO: Task 1793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1793 try 0 launched on executor multi with executor id 1794 parsl.dataflow.dflow INFO: Standard output for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2e1a02dc-c049-4a62-9a0d-009cca33daee_isrForCrosstalkSources_855459_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2e1a02dc-c049-4a62-9a0d-009cca33daee_isrForCrosstalkSources_855459_9.stderr parsl.dataflow.dflow DEBUG: Task 1794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1794 has memoization hash dfb06a47a8ed17c9eb96f2ad91b785ae parsl.dataflow.memoization INFO: Task 1794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1794 try 0 launched on executor multi with executor id 1795 parsl.dataflow.dflow INFO: Standard output for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cc335d55-7238-441b-a06a-6fd266e70d06_isrForCrosstalkSources_855449_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cc335d55-7238-441b-a06a-6fd266e70d06_isrForCrosstalkSources_855449_40.stderr parsl.dataflow.dflow DEBUG: Task 1795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1795 has memoization hash d4615334a8349b4c3decd9cd91e4929c parsl.dataflow.memoization INFO: Task 1795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1795 try 0 launched on executor multi with executor id 1796 parsl.dataflow.dflow INFO: Standard output for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/710ce5a3-f4fa-47d4-8c07-0d595fd56f68_isrForCrosstalkSources_855392_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/710ce5a3-f4fa-47d4-8c07-0d595fd56f68_isrForCrosstalkSources_855392_36.stderr parsl.dataflow.dflow DEBUG: Task 1796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1796 has memoization hash f608de7ee66ec9f77d3808357144f727 parsl.dataflow.memoization INFO: Task 1796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1796 try 0 launched on executor multi with executor id 1797 parsl.dataflow.dflow INFO: Standard output for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b8ba462a-29b9-49a0-bc73-4ff7f90760f9_isrForCrosstalkSources_855439_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b8ba462a-29b9-49a0-bc73-4ff7f90760f9_isrForCrosstalkSources_855439_4.stderr parsl.dataflow.dflow DEBUG: Task 1797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1797 has memoization hash fed02f95bd328a85487088d203d5f209 parsl.dataflow.memoization INFO: Task 1797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1797 try 0 launched on executor multi with executor id 1798 parsl.dataflow.dflow INFO: Standard output for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35551444-7132-4726-8388-8e6b637ef4f6_isrForCrosstalkSources_855409_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35551444-7132-4726-8388-8e6b637ef4f6_isrForCrosstalkSources_855409_51.stderr parsl.dataflow.dflow DEBUG: Task 1798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1798 has memoization hash e2adbf9403201100f37e7be0a0694d61 parsl.dataflow.memoization INFO: Task 1798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1798 try 0 launched on executor multi with executor id 1799 parsl.dataflow.dflow INFO: Standard output for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c996ab30-2e5b-4d19-ab8a-a308b07bc538_isrForCrosstalkSources_855411_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c996ab30-2e5b-4d19-ab8a-a308b07bc538_isrForCrosstalkSources_855411_42.stderr parsl.dataflow.dflow DEBUG: Task 1799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1799 has memoization hash 20c3153101203f417cb857f6ad2b36c8 parsl.dataflow.memoization INFO: Task 1799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1799 try 0 launched on executor multi with executor id 1800 parsl.dataflow.dflow INFO: Standard output for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3f21396d-2a9d-43ba-bdb8-3b765dd29052_isrForCrosstalkSources_855400_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3f21396d-2a9d-43ba-bdb8-3b765dd29052_isrForCrosstalkSources_855400_30.stderr parsl.dataflow.dflow DEBUG: Task 1800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1800 has memoization hash e1de711683fe906b403ce5a68d30b60c parsl.dataflow.memoization INFO: Task 1800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1800 try 0 launched on executor multi with executor id 1801 parsl.dataflow.dflow INFO: Standard output for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1280ade4-4364-4f01-a61b-c1b9204f362d_isrForCrosstalkSources_855401_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1280ade4-4364-4f01-a61b-c1b9204f362d_isrForCrosstalkSources_855401_14.stderr parsl.dataflow.dflow DEBUG: Task 1801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1801 has memoization hash d66a5aa0fe86966a732eed01c1d2fb82 parsl.dataflow.memoization INFO: Task 1801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1801 try 0 launched on executor multi with executor id 1802 parsl.dataflow.dflow INFO: Standard output for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/206149cf-ce73-448b-889e-c4cbcd1009e7_isrForCrosstalkSources_855459_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/206149cf-ce73-448b-889e-c4cbcd1009e7_isrForCrosstalkSources_855459_47.stderr parsl.dataflow.dflow DEBUG: Task 1802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1802 has memoization hash 4f1f8144c9fa594bf97f63c9ce3c2817 parsl.dataflow.memoization INFO: Task 1802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1802 try 0 launched on executor multi with executor id 1803 parsl.dataflow.dflow INFO: Standard output for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f3b4ba15-ea67-4e95-9d51-0bb1374cb963_isrForCrosstalkSources_855384_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f3b4ba15-ea67-4e95-9d51-0bb1374cb963_isrForCrosstalkSources_855384_12.stderr parsl.dataflow.dflow DEBUG: Task 1803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1803 has memoization hash f2588f48f77f8a8cee110cb66fdcab0e parsl.dataflow.memoization INFO: Task 1803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1803 try 0 launched on executor multi with executor id 1804 parsl.dataflow.dflow INFO: Standard output for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c2d96fa2-2ea4-41af-864e-10b72be8a0a0_isrForCrosstalkSources_855431_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c2d96fa2-2ea4-41af-864e-10b72be8a0a0_isrForCrosstalkSources_855431_36.stderr parsl.dataflow.dflow DEBUG: Task 1804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1804 has memoization hash cf2dd613a05470f9274a247e3c302bc0 parsl.dataflow.memoization INFO: Task 1804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1804 try 0 launched on executor multi with executor id 1805 parsl.dataflow.dflow INFO: Standard output for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/46df093c-4e7b-47fd-a3b5-ba9df0cc6e62_isrForCrosstalkSources_855413_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/46df093c-4e7b-47fd-a3b5-ba9df0cc6e62_isrForCrosstalkSources_855413_62.stderr parsl.dataflow.dflow DEBUG: Task 1805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1805 has memoization hash 5502a4083360695a7ba714712bf0627a parsl.dataflow.memoization INFO: Task 1805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1805 try 0 launched on executor multi with executor id 1806 parsl.dataflow.dflow INFO: Standard output for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f7b9e6a6-147c-448f-8cb3-805538ee09ab_isrForCrosstalkSources_855443_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f7b9e6a6-147c-448f-8cb3-805538ee09ab_isrForCrosstalkSources_855443_28.stderr parsl.dataflow.dflow DEBUG: Task 1806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1806 has memoization hash 8748a1f89a948edff0e58f8d54170f7d parsl.dataflow.memoization INFO: Task 1806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1806 try 0 launched on executor multi with executor id 1807 parsl.dataflow.dflow INFO: Standard output for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8b2e9ce9-1b94-4853-a818-9ae1069a6ba9_isrForCrosstalkSources_855412_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8b2e9ce9-1b94-4853-a818-9ae1069a6ba9_isrForCrosstalkSources_855412_19.stderr parsl.dataflow.dflow DEBUG: Task 1807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1807 has memoization hash 95d2cf1609e2d9026b6039e34052dd59 parsl.dataflow.memoization INFO: Task 1807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67afba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:22 proc_lsst.multi:170 [INFO] [multi] job_id 16964078 proc_lsst.multi INFO: [multi] job_id 16964078 2024-03-09 14:50:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 14:50:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964078 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964078 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16964078 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-09 14:50:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 14:50:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1807 try 0 launched on executor multi with executor id 1808 parsl.dataflow.dflow INFO: Standard output for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/453a0617-e261-4f0f-982d-c13a22f235dd_isrForCrosstalkSources_855447_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/453a0617-e261-4f0f-982d-c13a22f235dd_isrForCrosstalkSources_855447_58.stderr parsl.dataflow.dflow DEBUG: Task 1808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1808 has memoization hash f3b79fc78fc80a41ad3c56ab4a0c3097 parsl.dataflow.memoization INFO: Task 1808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1808 try 0 launched on executor multi with executor id 1809 parsl.dataflow.dflow INFO: Standard output for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/71ca51bd-d941-4ac0-9aa2-a074f722d84f_isrForCrosstalkSources_855416_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/71ca51bd-d941-4ac0-9aa2-a074f722d84f_isrForCrosstalkSources_855416_25.stderr parsl.dataflow.dflow DEBUG: Task 1809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1809 has memoization hash ff0fdca7fcce234d17ea8c017ffc5502 parsl.dataflow.memoization INFO: Task 1809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1809 try 0 launched on executor multi with executor id 1810 parsl.dataflow.dflow INFO: Standard output for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7a58c13a-4866-4177-b7c1-1d7933cea294_isrForCrosstalkSources_855429_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7a58c13a-4866-4177-b7c1-1d7933cea294_isrForCrosstalkSources_855429_12.stderr parsl.dataflow.dflow DEBUG: Task 1810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1810 has memoization hash 22e963d09f9c5ecacbfbbea8000cf75d parsl.dataflow.memoization INFO: Task 1810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1810 try 0 launched on executor multi with executor id 1811 parsl.dataflow.dflow INFO: Standard output for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ebe85b08-4d71-4f19-9d37-3eefb655b582_isrForCrosstalkSources_855437_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ebe85b08-4d71-4f19-9d37-3eefb655b582_isrForCrosstalkSources_855437_17.stderr parsl.dataflow.dflow DEBUG: Task 1811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1811 has memoization hash f7789a6d8738d3060ed8a0a3706af480 parsl.dataflow.memoization INFO: Task 1811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67ad4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1811 try 0 launched on executor multi with executor id 1812 parsl.dataflow.dflow INFO: Standard output for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/39906da9-4ea3-4f30-b6cc-772b7b5eb4f9_isrForCrosstalkSources_855422_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/39906da9-4ea3-4f30-b6cc-772b7b5eb4f9_isrForCrosstalkSources_855422_37.stderr parsl.dataflow.dflow DEBUG: Task 1812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1812 has memoization hash 1d3f6f5080d59c2df076e5de3945d5de parsl.dataflow.memoization INFO: Task 1812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1812 try 0 launched on executor multi with executor id 1813 parsl.dataflow.dflow INFO: Standard output for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/bb0c4263-fd06-4d4c-b2c1-4988469a43fa_isrForCrosstalkSources_855455_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/bb0c4263-fd06-4d4c-b2c1-4988469a43fa_isrForCrosstalkSources_855455_7.stderr parsl.dataflow.dflow DEBUG: Task 1813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1813 has memoization hash e93a9bc0120ca1c06e0093ea90e0b229 parsl.dataflow.memoization INFO: Task 1813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1813 try 0 launched on executor multi with executor id 1814 parsl.dataflow.dflow INFO: Standard output for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4b17960-2029-48cc-b4df-6c3b4ae4e7f9_isrForCrosstalkSources_855393_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4b17960-2029-48cc-b4df-6c3b4ae4e7f9_isrForCrosstalkSources_855393_62.stderr parsl.dataflow.dflow DEBUG: Task 1814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1814 has memoization hash 254bdbd1b658a440cfb6b1db0e44665b parsl.dataflow.memoization INFO: Task 1814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1814 try 0 launched on executor multi with executor id 1815 parsl.dataflow.dflow INFO: Standard output for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/68ceb3ca-e88e-4f24-aa0c-e8e5fc0f3551_isrForCrosstalkSources_855421_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/68ceb3ca-e88e-4f24-aa0c-e8e5fc0f3551_isrForCrosstalkSources_855421_51.stderr parsl.dataflow.dflow DEBUG: Task 1815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1815 has memoization hash 86871559c7ceae7510eec203a8ddb47f parsl.dataflow.memoization INFO: Task 1815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1815 try 0 launched on executor multi with executor id 1816 parsl.dataflow.dflow INFO: Standard output for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f1b8b52b-04e1-4221-a6ea-3034fafb1d53_isrForCrosstalkSources_855392_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f1b8b52b-04e1-4221-a6ea-3034fafb1d53_isrForCrosstalkSources_855392_5.stderr parsl.dataflow.dflow DEBUG: Task 1816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1816 has memoization hash b9667916d1ada5a3ace08480414e3bc1 parsl.dataflow.memoization INFO: Task 1816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1816 try 0 launched on executor multi with executor id 1817 parsl.dataflow.dflow INFO: Standard output for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c4b6436f-2e35-425f-9901-528a36631b6f_isrForCrosstalkSources_855422_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c4b6436f-2e35-425f-9901-528a36631b6f_isrForCrosstalkSources_855422_6.stderr parsl.dataflow.dflow DEBUG: Task 1817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1817 has memoization hash 0dca48a8bc4660199e86a0045d997b41 parsl.dataflow.memoization INFO: Task 1817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1817 try 0 launched on executor multi with executor id 1818 parsl.dataflow.dflow INFO: Standard output for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/91f3a5dd-8eb8-4e80-b534-f9e57e9aba78_isrForCrosstalkSources_855445_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/91f3a5dd-8eb8-4e80-b534-f9e57e9aba78_isrForCrosstalkSources_855445_4.stderr parsl.dataflow.dflow DEBUG: Task 1818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1818 has memoization hash 61604737dca0ee2e7c4ec5d8c81160ad parsl.dataflow.memoization INFO: Task 1818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1818 try 0 launched on executor multi with executor id 1819 parsl.dataflow.dflow INFO: Standard output for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/589ece04-d72b-4ceb-b30f-df4dfd59c741_isrForCrosstalkSources_855393_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/589ece04-d72b-4ceb-b30f-df4dfd59c741_isrForCrosstalkSources_855393_21.stderr parsl.dataflow.dflow DEBUG: Task 1819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1819 has memoization hash cf22881e9fe67efd46d1570dbe35c134 parsl.dataflow.memoization INFO: Task 1819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1819 try 0 launched on executor multi with executor id 1820 parsl.dataflow.dflow INFO: Standard output for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/efb934a0-2c15-4d4e-ad16-c013b5aec410_isrForCrosstalkSources_855385_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/efb934a0-2c15-4d4e-ad16-c013b5aec410_isrForCrosstalkSources_855385_26.stderr parsl.dataflow.dflow DEBUG: Task 1820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1820 has memoization hash b4c2592dc1897ed3a51f17c246adb47b parsl.dataflow.memoization INFO: Task 1820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1820 try 0 launched on executor multi with executor id 1821 parsl.dataflow.dflow INFO: Standard output for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca57bf08-1443-456d-86b5-015ff17c8329_isrForCrosstalkSources_855437_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca57bf08-1443-456d-86b5-015ff17c8329_isrForCrosstalkSources_855437_21.stderr parsl.dataflow.dflow DEBUG: Task 1821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1821 has memoization hash 57d631cad749bf98d1e63febcf2e897b parsl.dataflow.memoization INFO: Task 1821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1821 try 0 launched on executor multi with executor id 1822 parsl.dataflow.dflow INFO: Standard output for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c88baf85-84a2-4b6a-a134-560fa0d35345_isrForCrosstalkSources_855440_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c88baf85-84a2-4b6a-a134-560fa0d35345_isrForCrosstalkSources_855440_39.stderr parsl.dataflow.dflow DEBUG: Task 1822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1822 has memoization hash 35e47b27678291839f35869bab9c62af parsl.dataflow.memoization INFO: Task 1822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1822 try 0 launched on executor multi with executor id 1823 parsl.dataflow.dflow INFO: Standard output for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8fa3c929-dfad-4b93-820a-8a9cffd05210_isrForCrosstalkSources_855425_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8fa3c929-dfad-4b93-820a-8a9cffd05210_isrForCrosstalkSources_855425_50.stderr parsl.dataflow.dflow DEBUG: Task 1823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1823 has memoization hash d71677e8c57d9fe2dd9d8129376c621b parsl.dataflow.memoization INFO: Task 1823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1823 try 0 launched on executor multi with executor id 1824 parsl.dataflow.dflow INFO: Standard output for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dee22110-bc50-4d50-84b3-d94197c64419_isrForCrosstalkSources_855396_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dee22110-bc50-4d50-84b3-d94197c64419_isrForCrosstalkSources_855396_2.stderr parsl.dataflow.dflow DEBUG: Task 1824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1824 has memoization hash 9b8c4c27fbde415f87abc52265d5fb3b parsl.dataflow.memoization INFO: Task 1824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1824 try 0 launched on executor multi with executor id 1825 parsl.dataflow.dflow INFO: Standard output for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7aef786a-7407-4f08-be65-e1dc733eb679_isrForCrosstalkSources_855380_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7aef786a-7407-4f08-be65-e1dc733eb679_isrForCrosstalkSources_855380_4.stderr parsl.dataflow.dflow DEBUG: Task 1825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1825 has memoization hash 3836d52f10e87e86a6ab7ba211c9c69a parsl.dataflow.memoization INFO: Task 1825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1825 try 0 launched on executor multi with executor id 1826 parsl.dataflow.dflow INFO: Standard output for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/16253fce-b4b4-42f0-9bd3-993b904c1754_isrForCrosstalkSources_855399_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/16253fce-b4b4-42f0-9bd3-993b904c1754_isrForCrosstalkSources_855399_58.stderr parsl.dataflow.dflow DEBUG: Task 1826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1826 has memoization hash b334c5a9d5a7a856978c7ac14313b39d parsl.dataflow.memoization INFO: Task 1826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1826 try 0 launched on executor multi with executor id 1827 parsl.dataflow.dflow INFO: Standard output for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8e29da3c-dc1d-4dc1-81f9-f8be5273c0b3_isrForCrosstalkSources_855419_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8e29da3c-dc1d-4dc1-81f9-f8be5273c0b3_isrForCrosstalkSources_855419_26.stderr parsl.dataflow.dflow DEBUG: Task 1827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1827 has memoization hash b9653abdb8cb67619a5b1a608ddacd78 parsl.dataflow.memoization INFO: Task 1827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1827 try 0 launched on executor multi with executor id 1828 parsl.dataflow.dflow INFO: Standard output for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/74dd5604-0701-465d-8783-f36f7541fbd9_isrForCrosstalkSources_855412_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/74dd5604-0701-465d-8783-f36f7541fbd9_isrForCrosstalkSources_855412_13.stderr parsl.dataflow.dflow DEBUG: Task 1828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1828 has memoization hash 488b7867c15a760e118d9de6b163add5 parsl.dataflow.memoization INFO: Task 1828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1828 try 0 launched on executor multi with executor id 1829 parsl.dataflow.dflow INFO: Standard output for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d9716ee5-e52a-4aac-99d6-1fbf48ac594b_isrForCrosstalkSources_855386_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d9716ee5-e52a-4aac-99d6-1fbf48ac594b_isrForCrosstalkSources_855386_35.stderr parsl.dataflow.dflow DEBUG: Task 1829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1829 has memoization hash e343244553517bfca22420e2c086b7db parsl.dataflow.memoization INFO: Task 1829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1829 try 0 launched on executor multi with executor id 1830 parsl.dataflow.dflow INFO: Standard output for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d9cd768d-5835-40a5-ae04-81af62f3a986_isrForCrosstalkSources_855436_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d9cd768d-5835-40a5-ae04-81af62f3a986_isrForCrosstalkSources_855436_51.stderr parsl.dataflow.dflow DEBUG: Task 1830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1830 has memoization hash a6aa59df0c968e1d7feb9b5f852653c2 parsl.dataflow.memoization INFO: Task 1830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1830 try 0 launched on executor multi with executor id 1831 parsl.dataflow.dflow INFO: Standard output for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc2fbac9-4959-4638-bef6-bec5edf9fd06_isrForCrosstalkSources_855403_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc2fbac9-4959-4638-bef6-bec5edf9fd06_isrForCrosstalkSources_855403_21.stderr parsl.dataflow.dflow DEBUG: Task 1831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1831 has memoization hash 49c653a7b98ef5c20454d4f42bcac31a parsl.dataflow.memoization INFO: Task 1831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1831 try 0 launched on executor multi with executor id 1832 parsl.dataflow.dflow INFO: Standard output for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f954c276-9690-40ce-b2ca-d6c67558a84b_isrForCrosstalkSources_855404_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f954c276-9690-40ce-b2ca-d6c67558a84b_isrForCrosstalkSources_855404_45.stderr parsl.dataflow.dflow DEBUG: Task 1832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1832 has memoization hash 7441ad79bed26e87c7ac5cc1870401b5 parsl.dataflow.memoization INFO: Task 1832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1832 try 0 launched on executor multi with executor id 1833 parsl.dataflow.dflow INFO: Standard output for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b779c11d-7edf-48da-8d28-ae0ad501e80f_isrForCrosstalkSources_855396_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b779c11d-7edf-48da-8d28-ae0ad501e80f_isrForCrosstalkSources_855396_16.stderr parsl.dataflow.dflow DEBUG: Task 1833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1833 has memoization hash 1cc6f4262004823a045dce2a40681dde parsl.dataflow.memoization INFO: Task 1833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1833 try 0 launched on executor multi with executor id 1834 parsl.dataflow.dflow INFO: Standard output for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/71b1b6bf-ef91-4728-b133-8e84bd2e75b9_isrForCrosstalkSources_855394_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/71b1b6bf-ef91-4728-b133-8e84bd2e75b9_isrForCrosstalkSources_855394_8.stderr parsl.dataflow.dflow DEBUG: Task 1834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1834 has memoization hash bc6e86a25e167e488eb2c94f92a4914c parsl.dataflow.memoization INFO: Task 1834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1834 try 0 launched on executor multi with executor id 1835 parsl.dataflow.dflow INFO: Standard output for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8401fc31-aef5-49ff-8d91-9421bc2edf05_isrForCrosstalkSources_855400_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8401fc31-aef5-49ff-8d91-9421bc2edf05_isrForCrosstalkSources_855400_39.stderr parsl.dataflow.dflow DEBUG: Task 1835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1835 has memoization hash 2fff43b2b0b6b2cc958802f21b02dcc5 parsl.dataflow.memoization INFO: Task 1835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1835 try 0 launched on executor multi with executor id 1836 parsl.dataflow.dflow INFO: Standard output for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/4b45e9c9-1d7a-486e-93e1-eadcb8d86b9d_isrForCrosstalkSources_855438_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/4b45e9c9-1d7a-486e-93e1-eadcb8d86b9d_isrForCrosstalkSources_855438_52.stderr parsl.dataflow.dflow DEBUG: Task 1836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1836 has memoization hash 09358479c79134ae55c792579d152595 parsl.dataflow.memoization INFO: Task 1836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:23 proc_lsst.multi:170 [INFO] [multi] job_id 16964079 proc_lsst.multi INFO: [multi] job_id 16964079 2024-03-09 14:50:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 14:50:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964079 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964079 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16964079 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-09 14:50:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-09 14:50:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 14:50:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1836 try 0 launched on executor multi with executor id 1837 parsl.dataflow.dflow INFO: Standard output for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/25b6cc8b-4c29-4bc9-afc8-efd1b95650d5_isrForCrosstalkSources_855406_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/25b6cc8b-4c29-4bc9-afc8-efd1b95650d5_isrForCrosstalkSources_855406_14.stderr parsl.dataflow.dflow DEBUG: Task 1837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1837 has memoization hash 3106a0edd901b17eee304cf89155cd53 parsl.dataflow.memoization INFO: Task 1837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1837 try 0 launched on executor multi with executor id 1838 parsl.dataflow.dflow INFO: Standard output for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/12621c1b-b93a-49b0-a9fe-9d88547e61b7_isrForCrosstalkSources_855387_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/12621c1b-b93a-49b0-a9fe-9d88547e61b7_isrForCrosstalkSources_855387_23.stderr parsl.dataflow.dflow DEBUG: Task 1838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1838 has memoization hash a52b314391ad73325490b6ae6d5c939d parsl.dataflow.memoization INFO: Task 1838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a67aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1838 try 0 launched on executor multi with executor id 1839 parsl.dataflow.dflow INFO: Standard output for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/582419fa-4ebd-4df0-b17b-ffe466d13fa6_isrForCrosstalkSources_855431_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/582419fa-4ebd-4df0-b17b-ffe466d13fa6_isrForCrosstalkSources_855431_46.stderr parsl.dataflow.dflow DEBUG: Task 1839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1839 has memoization hash 791e952e2ccbcad0f50da06b723f2d0b parsl.dataflow.memoization INFO: Task 1839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1839 try 0 launched on executor multi with executor id 1840 parsl.dataflow.dflow INFO: Standard output for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3328db37-cb2f-4abd-b283-c5ccad33d94c_isrForCrosstalkSources_855414_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3328db37-cb2f-4abd-b283-c5ccad33d94c_isrForCrosstalkSources_855414_52.stderr parsl.dataflow.dflow DEBUG: Task 1840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1840 has memoization hash bbc286e08be3358a9d613db03c769393 parsl.dataflow.memoization INFO: Task 1840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1840 try 0 launched on executor multi with executor id 1841 parsl.dataflow.dflow INFO: Standard output for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/b04b2089-f61b-4002-b8ce-c1dc71ed2f06_isrForCrosstalkSources_855390_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/b04b2089-f61b-4002-b8ce-c1dc71ed2f06_isrForCrosstalkSources_855390_60.stderr parsl.dataflow.dflow DEBUG: Task 1841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1841 has memoization hash ffa960a65c325a637a62036131515bed parsl.dataflow.memoization INFO: Task 1841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1841 try 0 launched on executor multi with executor id 1842 parsl.dataflow.dflow INFO: Standard output for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f67c4abd-837f-449e-b47c-b088d7900a15_isrForCrosstalkSources_855396_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f67c4abd-837f-449e-b47c-b088d7900a15_isrForCrosstalkSources_855396_20.stderr parsl.dataflow.dflow DEBUG: Task 1842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1842 has memoization hash f8af34aa1f2367e3287af3dd113eb026 parsl.dataflow.memoization INFO: Task 1842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1842 try 0 launched on executor multi with executor id 1843 parsl.dataflow.dflow INFO: Standard output for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/19106b9d-249d-4a6d-b127-9a82f066dbe0_isrForCrosstalkSources_855416_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/19106b9d-249d-4a6d-b127-9a82f066dbe0_isrForCrosstalkSources_855416_58.stderr parsl.dataflow.dflow DEBUG: Task 1843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1843 has memoization hash 2e8254cf29f64d95cdbb499246b45fa3 parsl.dataflow.memoization INFO: Task 1843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1843 try 0 launched on executor multi with executor id 1844 parsl.dataflow.dflow INFO: Standard output for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/3babbf68-ed10-4651-92e4-3b7966947cb9_isrForCrosstalkSources_855392_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/3babbf68-ed10-4651-92e4-3b7966947cb9_isrForCrosstalkSources_855392_45.stderr parsl.dataflow.dflow DEBUG: Task 1844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1844 has memoization hash 0aa90c9c67ccddec815c392a26b08816 parsl.dataflow.memoization INFO: Task 1844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1844 try 0 launched on executor multi with executor id 1845 parsl.dataflow.dflow INFO: Standard output for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8688f86c-e315-44dc-b3c9-d2d8f3c98851_isrForCrosstalkSources_855459_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8688f86c-e315-44dc-b3c9-d2d8f3c98851_isrForCrosstalkSources_855459_28.stderr parsl.dataflow.dflow DEBUG: Task 1845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1845 has memoization hash 3d4168bdc411f9e60e8bbda91b6656cc parsl.dataflow.memoization INFO: Task 1845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1845 try 0 launched on executor multi with executor id 1846 parsl.dataflow.dflow INFO: Standard output for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1e41b010-8bf4-4b95-bc6b-7fa62a6252a9_isrForCrosstalkSources_855427_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1e41b010-8bf4-4b95-bc6b-7fa62a6252a9_isrForCrosstalkSources_855427_27.stderr parsl.dataflow.dflow DEBUG: Task 1846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1846 has memoization hash 658dfa4c0d2eb230c299ae20e6d75987 parsl.dataflow.memoization INFO: Task 1846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1846 try 0 launched on executor multi with executor id 1847 parsl.dataflow.dflow INFO: Standard output for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0970ceb5-50d0-4b08-9c50-5f49a7a2571a_isrForCrosstalkSources_855458_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0970ceb5-50d0-4b08-9c50-5f49a7a2571a_isrForCrosstalkSources_855458_45.stderr parsl.dataflow.dflow DEBUG: Task 1847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1847 has memoization hash d415d454e99d9297bd509746ef54340f parsl.dataflow.memoization INFO: Task 1847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1847 try 0 launched on executor multi with executor id 1848 parsl.dataflow.dflow INFO: Standard output for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/1cf4752c-989c-49f3-8cc6-5c4039fc5fd5_isrForCrosstalkSources_855406_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/1cf4752c-989c-49f3-8cc6-5c4039fc5fd5_isrForCrosstalkSources_855406_31.stderr parsl.dataflow.dflow DEBUG: Task 1848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1848 has memoization hash dc575c1b962a12a976bf181dbf351dd4 parsl.dataflow.memoization INFO: Task 1848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1848 try 0 launched on executor multi with executor id 1849 parsl.dataflow.dflow INFO: Standard output for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3ebf37ba-a791-4692-a2df-72423164fad4_isrForCrosstalkSources_855438_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3ebf37ba-a791-4692-a2df-72423164fad4_isrForCrosstalkSources_855438_24.stderr parsl.dataflow.dflow DEBUG: Task 1849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1849 has memoization hash dc7b2a9b22b6f52713098345107fad64 parsl.dataflow.memoization INFO: Task 1849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1849 try 0 launched on executor multi with executor id 1850 parsl.dataflow.dflow INFO: Standard output for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8b299c75-6a97-48ab-846e-59201d6f69ff_isrForCrosstalkSources_855386_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8b299c75-6a97-48ab-846e-59201d6f69ff_isrForCrosstalkSources_855386_58.stderr parsl.dataflow.dflow DEBUG: Task 1850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1850 has memoization hash 878b4d5f43a656479a0bfb3bf0885dcf parsl.dataflow.memoization INFO: Task 1850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1850 try 0 launched on executor multi with executor id 1851 parsl.dataflow.dflow INFO: Standard output for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/6c0ad98b-8d39-45a6-be53-09d7cea48231_isrForCrosstalkSources_855454_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/6c0ad98b-8d39-45a6-be53-09d7cea48231_isrForCrosstalkSources_855454_31.stderr parsl.dataflow.dflow DEBUG: Task 1851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1851 has memoization hash 9db93201bdfcd3e648ce91433a850d89 parsl.dataflow.memoization INFO: Task 1851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1851 try 0 launched on executor multi with executor id 1852 parsl.dataflow.dflow INFO: Standard output for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3285667d-f96b-4289-aa70-8785b3699b2e_isrForCrosstalkSources_855394_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3285667d-f96b-4289-aa70-8785b3699b2e_isrForCrosstalkSources_855394_39.stderr parsl.dataflow.dflow DEBUG: Task 1852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1852 has memoization hash f64f7c8730f35bfc9136748a055dfce9 parsl.dataflow.memoization INFO: Task 1852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1852 try 0 launched on executor multi with executor id 1853 parsl.dataflow.dflow INFO: Standard output for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/e6de8865-2b78-40d2-af65-cfe4c52da8b8_isrForCrosstalkSources_855415_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/e6de8865-2b78-40d2-af65-cfe4c52da8b8_isrForCrosstalkSources_855415_11.stderr parsl.dataflow.dflow DEBUG: Task 1853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1853 has memoization hash e866d90337b7899ced2c035575d5ae3b parsl.dataflow.memoization INFO: Task 1853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1853 try 0 launched on executor multi with executor id 1854 parsl.dataflow.dflow INFO: Standard output for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5a159d82-15d3-4720-9f03-f01d7e457671_isrForCrosstalkSources_855430_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5a159d82-15d3-4720-9f03-f01d7e457671_isrForCrosstalkSources_855430_39.stderr parsl.dataflow.dflow DEBUG: Task 1854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1854 has memoization hash ff02ef03a6ea8ff4a026170e0fb8cfb9 parsl.dataflow.memoization INFO: Task 1854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1854 try 0 launched on executor multi with executor id 1855 parsl.dataflow.dflow INFO: Standard output for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/894365fa-3212-4f2b-8b81-432a1f323612_isrForCrosstalkSources_855433_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/894365fa-3212-4f2b-8b81-432a1f323612_isrForCrosstalkSources_855433_55.stderr parsl.dataflow.dflow DEBUG: Task 1855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1855 has memoization hash b474af37a9b2728b292782d9371dd45b parsl.dataflow.memoization INFO: Task 1855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1855 try 0 launched on executor multi with executor id 1856 parsl.dataflow.dflow INFO: Standard output for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6fe1e1f3-87f5-4c01-8b48-535de91ccf6f_isrForCrosstalkSources_855421_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6fe1e1f3-87f5-4c01-8b48-535de91ccf6f_isrForCrosstalkSources_855421_3.stderr parsl.dataflow.dflow DEBUG: Task 1856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1856 has memoization hash dd7d5d2f71a09ee1e636752931000062 parsl.dataflow.memoization INFO: Task 1856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1856 try 0 launched on executor multi with executor id 1857 parsl.dataflow.dflow INFO: Standard output for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c76999f4-f934-47b9-908d-21cb091d71a4_isrForCrosstalkSources_855383_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c76999f4-f934-47b9-908d-21cb091d71a4_isrForCrosstalkSources_855383_8.stderr parsl.dataflow.dflow DEBUG: Task 1857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1857 has memoization hash ab17b59798e1c265ce3eab2c89efeb01 parsl.dataflow.memoization INFO: Task 1857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1857 try 0 launched on executor multi with executor id 1858 parsl.dataflow.dflow INFO: Standard output for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7e39670a-1d29-4b6f-b8ec-60a616d8d406_isrForCrosstalkSources_855388_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7e39670a-1d29-4b6f-b8ec-60a616d8d406_isrForCrosstalkSources_855388_42.stderr parsl.dataflow.dflow DEBUG: Task 1858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1858 has memoization hash 50d03f7a856b74ce9b199f59d5a7e67f parsl.dataflow.memoization INFO: Task 1858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1858 try 0 launched on executor multi with executor id 1859 parsl.dataflow.dflow INFO: Standard output for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/58b95eab-d647-4de1-9845-d519a4564081_isrForCrosstalkSources_855459_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/58b95eab-d647-4de1-9845-d519a4564081_isrForCrosstalkSources_855459_24.stderr parsl.dataflow.dflow DEBUG: Task 1859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1859 has memoization hash 0a3af129bb9ff90219060891a21c7bb0 parsl.dataflow.memoization INFO: Task 1859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:24 proc_lsst.multi:170 [INFO] [multi] job_id 16964080 proc_lsst.multi INFO: [multi] job_id 16964080 2024-03-09 14:50:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16964080 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16964080 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16964080 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-09 14:50:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1859 try 0 launched on executor multi with executor id 1860 parsl.dataflow.dflow INFO: Standard output for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ef162b1-547d-4e84-8265-9c7d2245bb79_isrForCrosstalkSources_855379_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ef162b1-547d-4e84-8265-9c7d2245bb79_isrForCrosstalkSources_855379_36.stderr parsl.dataflow.dflow DEBUG: Task 1860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1860 has memoization hash c003a7e6451fc556df8abd27a2682a41 parsl.dataflow.memoization INFO: Task 1860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1860 try 0 launched on executor multi with executor id 1861 parsl.dataflow.dflow INFO: Standard output for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6e91be4c-fe7a-4b06-86a6-2d1984f35767_isrForCrosstalkSources_855407_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6e91be4c-fe7a-4b06-86a6-2d1984f35767_isrForCrosstalkSources_855407_7.stderr parsl.dataflow.dflow DEBUG: Task 1861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1861 has memoization hash de0e6f1c11565aa5e4e0e12a63ccd6fa parsl.dataflow.memoization INFO: Task 1861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1861 try 0 launched on executor multi with executor id 1862 parsl.dataflow.dflow INFO: Standard output for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/626982b9-0775-4657-b62e-94f7f2e97ceb_isrForCrosstalkSources_855431_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/626982b9-0775-4657-b62e-94f7f2e97ceb_isrForCrosstalkSources_855431_11.stderr parsl.dataflow.dflow DEBUG: Task 1862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1862 has memoization hash 4a0a92ac22a2e57770a9f059de7597b7 parsl.dataflow.memoization INFO: Task 1862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1862 try 0 launched on executor multi with executor id 1863 parsl.dataflow.dflow INFO: Standard output for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/42dbce6d-ed36-46ef-95a6-b580f9be7844_isrForCrosstalkSources_855398_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/42dbce6d-ed36-46ef-95a6-b580f9be7844_isrForCrosstalkSources_855398_56.stderr parsl.dataflow.dflow DEBUG: Task 1863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1863 has memoization hash 48be156b83bd45b91b54034a3b25d46f parsl.dataflow.memoization INFO: Task 1863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1863 try 0 launched on executor multi with executor id 1864 parsl.dataflow.dflow INFO: Standard output for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01e0e521-6a2e-4d15-baed-f8745149c72c_isrForCrosstalkSources_855433_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01e0e521-6a2e-4d15-baed-f8745149c72c_isrForCrosstalkSources_855433_19.stderr parsl.dataflow.dflow DEBUG: Task 1864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1864 has memoization hash aabe8cc14b12445dc99746888940dfe0 parsl.dataflow.memoization INFO: Task 1864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1864 try 0 launched on executor multi with executor id 1865 parsl.dataflow.dflow INFO: Standard output for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/778195b8-f715-48b5-8719-32a2d1e5ea14_isrForCrosstalkSources_855381_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/778195b8-f715-48b5-8719-32a2d1e5ea14_isrForCrosstalkSources_855381_27.stderr parsl.dataflow.dflow DEBUG: Task 1865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1865 has memoization hash 2736ea17eee523e83edc92fb25d85330 parsl.dataflow.memoization INFO: Task 1865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1865 try 0 launched on executor multi with executor id 1866 parsl.dataflow.dflow INFO: Standard output for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/eeba60ba-9db8-4fd6-abf8-4337697430f4_isrForCrosstalkSources_855444_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/eeba60ba-9db8-4fd6-abf8-4337697430f4_isrForCrosstalkSources_855444_22.stderr parsl.dataflow.dflow DEBUG: Task 1866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1866 has memoization hash cf7c3afd1ad2eeca56bd90232c6d2bd4 parsl.dataflow.memoization INFO: Task 1866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1866 try 0 launched on executor multi with executor id 1867 parsl.dataflow.dflow INFO: Standard output for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e67f1533-7fa1-425c-91ae-4d2cc8cf1ccb_isrForCrosstalkSources_855447_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e67f1533-7fa1-425c-91ae-4d2cc8cf1ccb_isrForCrosstalkSources_855447_6.stderr parsl.dataflow.dflow DEBUG: Task 1867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1867 has memoization hash 733065f6fb7a1feae76773f2edf5a0d2 parsl.dataflow.memoization INFO: Task 1867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1867 try 0 launched on executor multi with executor id 1868 parsl.dataflow.dflow INFO: Standard output for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0fb5b9c-f733-4cf4-b90b-b1415524481c_isrForCrosstalkSources_855380_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0fb5b9c-f733-4cf4-b90b-b1415524481c_isrForCrosstalkSources_855380_11.stderr parsl.dataflow.dflow DEBUG: Task 1868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1868 has memoization hash 22c0d08809080a0105405d8233d3ce74 parsl.dataflow.memoization INFO: Task 1868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1868 try 0 launched on executor multi with executor id 1869 parsl.dataflow.dflow INFO: Standard output for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/547e5cd9-3e7f-458d-af71-cc8eb0438374_isrForCrosstalkSources_855399_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/547e5cd9-3e7f-458d-af71-cc8eb0438374_isrForCrosstalkSources_855399_39.stderr parsl.dataflow.dflow DEBUG: Task 1869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1869 has memoization hash af99031d99876db2ba62289bc730a9f3 parsl.dataflow.memoization INFO: Task 1869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1869 try 0 launched on executor multi with executor id 1870 parsl.dataflow.dflow INFO: Standard output for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e5c93510-4d93-4fd7-9bc2-1bd1a392b102_isrForCrosstalkSources_855453_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e5c93510-4d93-4fd7-9bc2-1bd1a392b102_isrForCrosstalkSources_855453_37.stderr parsl.dataflow.dflow DEBUG: Task 1870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1870 has memoization hash 5aed7fab241ef8e6717de7283bcd0b50 parsl.dataflow.memoization INFO: Task 1870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1870 try 0 launched on executor multi with executor id 1871 parsl.dataflow.dflow INFO: Standard output for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/451a3e40-0c64-410a-9639-7923d8aaa797_isrForCrosstalkSources_855418_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/451a3e40-0c64-410a-9639-7923d8aaa797_isrForCrosstalkSources_855418_36.stderr parsl.dataflow.dflow DEBUG: Task 1871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1871 has memoization hash c688089bdcdae1c016683d7fcb47131c parsl.dataflow.memoization INFO: Task 1871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1871 try 0 launched on executor multi with executor id 1872 parsl.dataflow.dflow INFO: Standard output for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3bae6cc4-20d4-4bb4-98a1-4ce92136d249_isrForCrosstalkSources_855432_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3bae6cc4-20d4-4bb4-98a1-4ce92136d249_isrForCrosstalkSources_855432_41.stderr parsl.dataflow.dflow DEBUG: Task 1872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1872 has memoization hash 2d15e45134c7e9a05a54e8b523bc6ce5 parsl.dataflow.memoization INFO: Task 1872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1872 try 0 launched on executor multi with executor id 1873 parsl.dataflow.dflow INFO: Standard output for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ce112fcf-3710-4146-8b71-b200cf2b259a_isrForCrosstalkSources_855420_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ce112fcf-3710-4146-8b71-b200cf2b259a_isrForCrosstalkSources_855420_2.stderr parsl.dataflow.dflow DEBUG: Task 1873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1873 has memoization hash 7b604e4e7582619e2f1f7d43e49e9dce parsl.dataflow.memoization INFO: Task 1873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1873 try 0 launched on executor multi with executor id 1874 parsl.dataflow.dflow INFO: Standard output for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/cda88035-adac-491a-90e4-65d700d3e6fd_isrForCrosstalkSources_855419_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/cda88035-adac-491a-90e4-65d700d3e6fd_isrForCrosstalkSources_855419_34.stderr parsl.dataflow.dflow DEBUG: Task 1874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1874 has memoization hash aff32aba34f6f4f6572c9ff74617d375 parsl.dataflow.memoization INFO: Task 1874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1874 try 0 launched on executor multi with executor id 1875 parsl.dataflow.dflow INFO: Standard output for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e02b1aff-cbea-4a0b-a07e-cfe262dfdf78_isrForCrosstalkSources_855444_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e02b1aff-cbea-4a0b-a07e-cfe262dfdf78_isrForCrosstalkSources_855444_28.stderr parsl.dataflow.dflow DEBUG: Task 1875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1875 has memoization hash 5770a661a3ae48ea9bdb7bfb2c6839e5 parsl.dataflow.memoization INFO: Task 1875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1875 try 0 launched on executor multi with executor id 1876 parsl.dataflow.dflow INFO: Standard output for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9cb028d3-3a2a-4a24-8ca8-1a025ee85ec9_isrForCrosstalkSources_855439_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9cb028d3-3a2a-4a24-8ca8-1a025ee85ec9_isrForCrosstalkSources_855439_2.stderr parsl.dataflow.dflow DEBUG: Task 1876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1876 has memoization hash e210ed259cc81cb97836f705db9bd4af parsl.dataflow.memoization INFO: Task 1876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1876 try 0 launched on executor multi with executor id 1877 parsl.dataflow.dflow INFO: Standard output for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2498b053-89c2-46ab-be72-1454bbfd7ab4_isrForCrosstalkSources_855450_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2498b053-89c2-46ab-be72-1454bbfd7ab4_isrForCrosstalkSources_855450_34.stderr parsl.dataflow.dflow DEBUG: Task 1877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1877 has memoization hash 1ddc856fb746c40681d7b6ed200ee69c parsl.dataflow.memoization INFO: Task 1877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1877 try 0 launched on executor multi with executor id 1878 parsl.dataflow.dflow INFO: Standard output for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ab13f348-851c-4d4d-9cb7-a487d5a9c5a1_isrForCrosstalkSources_855447_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ab13f348-851c-4d4d-9cb7-a487d5a9c5a1_isrForCrosstalkSources_855447_48.stderr parsl.dataflow.dflow DEBUG: Task 1878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Storing app cache entry 4f134c616a06e86b177b41be31bdd868 with result from task 26 parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1878 has memoization hash adea9fb7481d583cf48ed51aaf5684e9 parsl.dataflow.memoization INFO: Task 1878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c9477436-68f7-4461-ac8b-2d4ece8edd6e_isrForCrosstalkSources_855436_28.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c9477436-68f7-4461-ac8b-2d4ece8edd6e_isrForCrosstalkSources_855436_28.stderr parsl.dataflow.dflow INFO: Parsl task 1878 try 0 launched on executor multi with executor id 1879 parsl.dataflow.dflow INFO: Standard output for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c1e3ddb-47b7-4ca9-8cf6-d084d4db4af1_isrForCrosstalkSources_855435_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c1e3ddb-47b7-4ca9-8cf6-d084d4db4af1_isrForCrosstalkSources_855435_59.stderr parsl.dataflow.dflow DEBUG: Task 1879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1879 has memoization hash efd3e555382c947ce8db7c4387ac3cfe parsl.dataflow.memoization INFO: Task 1879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1879 try 0 launched on executor multi with executor id 1880 parsl.dataflow.dflow INFO: Standard output for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/74ff7f73-0ccf-4d4e-9db7-fdb83997c5bf_isrForCrosstalkSources_855383_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/74ff7f73-0ccf-4d4e-9db7-fdb83997c5bf_isrForCrosstalkSources_855383_15.stderr parsl.dataflow.dflow DEBUG: Task 1880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1880 has memoization hash d19061b948654a47897bfe64cb4ac85e parsl.dataflow.memoization INFO: Task 1880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:24 proc_lsst.multi:170 [INFO] [multi] job_id 16964081 proc_lsst.multi INFO: [multi] job_id 16964081 2024-03-09 14:50:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964081 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964081 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16964081 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-09 14:50:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1880 try 0 launched on executor multi with executor id 1881 parsl.dataflow.dflow INFO: Standard output for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3364ee19-d665-477d-9b8f-13f17ab1745a_isrForCrosstalkSources_855390_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3364ee19-d665-477d-9b8f-13f17ab1745a_isrForCrosstalkSources_855390_25.stderr parsl.dataflow.dflow DEBUG: Task 1881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1881 has memoization hash 40f9d52799df65134b3d78090206b49e parsl.dataflow.memoization INFO: Task 1881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:26 proc_lsst.multi:170 [INFO] [multi] job_id 16964082 proc_lsst.multi INFO: [multi] job_id 16964082 2024-03-09 14:50:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:50:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964082 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964082 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16964082 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-09 14:50:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-09 14:50:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 14:50:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1881 try 0 launched on executor multi with executor id 1882 parsl.dataflow.dflow INFO: Standard output for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91e2138a-5fed-43d4-b2c4-bfcf03b7118d_isrForCrosstalkSources_855413_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91e2138a-5fed-43d4-b2c4-bfcf03b7118d_isrForCrosstalkSources_855413_54.stderr parsl.dataflow.dflow DEBUG: Task 1882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1882 has memoization hash bd156d898a43751ff672401c43ba55a9 parsl.dataflow.memoization INFO: Task 1882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6baa5878ae5e23ac3801548df319308 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fa9abeac-08f3-469f-bae0-af58c9a34b65_isrForCrosstalkSources_855413_27.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fa9abeac-08f3-469f-bae0-af58c9a34b65_isrForCrosstalkSources_855413_27.stderr parsl.dataflow.dflow INFO: Parsl task 1882 try 0 launched on executor multi with executor id 1883 parsl.dataflow.dflow INFO: Standard output for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/48ca5edd-34a6-4bd1-8aa4-a50d897851be_isrForCrosstalkSources_855445_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/48ca5edd-34a6-4bd1-8aa4-a50d897851be_isrForCrosstalkSources_855445_43.stderr parsl.dataflow.dflow DEBUG: Task 1883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1883 has memoization hash 3e2e36f373d02feb2902d1bf6896a1f9 parsl.dataflow.memoization INFO: Task 1883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1883 try 0 launched on executor multi with executor id 1884 parsl.dataflow.dflow INFO: Standard output for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fd49f2cd-90c7-4a1f-8f25-6fc35e55df28_isrForCrosstalkSources_855442_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fd49f2cd-90c7-4a1f-8f25-6fc35e55df28_isrForCrosstalkSources_855442_52.stderr parsl.dataflow.dflow DEBUG: Task 1884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1884 has memoization hash 76f5cda087c621a211beafe8291462a5 parsl.dataflow.memoization INFO: Task 1884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1884 try 0 launched on executor multi with executor id 1885 parsl.dataflow.dflow INFO: Standard output for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6233f9e1-ebd6-478e-8692-2dcf34aff3dc_isrForCrosstalkSources_855440_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6233f9e1-ebd6-478e-8692-2dcf34aff3dc_isrForCrosstalkSources_855440_58.stderr parsl.dataflow.dflow DEBUG: Task 1885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1885 has memoization hash 2617f1cf10a377319f61f747d30e1a9d parsl.dataflow.memoization INFO: Task 1885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07571ba26a1f4790b9b95aefce29b7b6 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d5c7e745-5541-42c6-99e6-f5a2a1e16a41_isrForCrosstalkSources_855435_1.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d5c7e745-5541-42c6-99e6-f5a2a1e16a41_isrForCrosstalkSources_855435_1.stderr parsl.dataflow.dflow INFO: Parsl task 1885 try 0 launched on executor multi with executor id 1886 parsl.dataflow.dflow INFO: Standard output for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c90706ad-c141-48f0-92d2-076b767568b4_isrForCrosstalkSources_855385_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c90706ad-c141-48f0-92d2-076b767568b4_isrForCrosstalkSources_855385_15.stderr parsl.dataflow.dflow DEBUG: Task 1886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1886 has memoization hash 4d625b30cc85fda5da85aebb5a2d8989 parsl.dataflow.memoization INFO: Task 1886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:28 proc_lsst.multi:170 [INFO] [multi] job_id 16964083 proc_lsst.multi INFO: [multi] job_id 16964083 2024-03-09 14:50:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:50:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964083 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964083 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16964083 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-09 14:50:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-09 14:50:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 14:50:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1886 try 0 launched on executor multi with executor id 1887 parsl.dataflow.dflow INFO: Standard output for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/315d54ab-d072-4363-b854-90af72f41549_isrForCrosstalkSources_855427_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/315d54ab-d072-4363-b854-90af72f41549_isrForCrosstalkSources_855427_60.stderr parsl.dataflow.dflow DEBUG: Task 1887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1887 has memoization hash 82041ef037e6a0eb13b511ade6f016c3 parsl.dataflow.memoization INFO: Task 1887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1887 try 0 launched on executor multi with executor id 1888 parsl.dataflow.dflow INFO: Standard output for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ad5e897e-4c2a-4097-9f78-2cc093682b93_isrForCrosstalkSources_855386_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ad5e897e-4c2a-4097-9f78-2cc093682b93_isrForCrosstalkSources_855386_37.stderr parsl.dataflow.dflow DEBUG: Task 1888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1888 has memoization hash 0ac507c8be67f3bc88e18c808e9be762 parsl.dataflow.memoization INFO: Task 1888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1888 try 0 launched on executor multi with executor id 1889 parsl.dataflow.dflow INFO: Standard output for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c142c38e-b5ed-4c47-adcc-717fb24ece10_isrForCrosstalkSources_855444_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c142c38e-b5ed-4c47-adcc-717fb24ece10_isrForCrosstalkSources_855444_5.stderr parsl.dataflow.dflow DEBUG: Task 1889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1889 has memoization hash d9db9a5a2e622cebbae5b03a4040d640 parsl.dataflow.memoization INFO: Task 1889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:29 proc_lsst.multi:170 [INFO] [multi] job_id 16964084 proc_lsst.multi INFO: [multi] job_id 16964084 2024-03-09 14:50:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:50:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964084 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964084 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16964084 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-09 14:50:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-09 14:50:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 14:50:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1889 try 0 launched on executor multi with executor id 1890 parsl.dataflow.dflow INFO: Standard output for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a8a2a8e4-a193-4e76-b541-7d4b0df0bc9f_isrForCrosstalkSources_855388_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a8a2a8e4-a193-4e76-b541-7d4b0df0bc9f_isrForCrosstalkSources_855388_50.stderr parsl.dataflow.dflow DEBUG: Task 1890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1890 has memoization hash ef624d439363af105bc68f4cdd4fd4f0 parsl.dataflow.memoization INFO: Task 1890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:30 proc_lsst.multi:170 [INFO] [multi] job_id 16964085 proc_lsst.multi INFO: [multi] job_id 16964085 2024-03-09 14:50:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964085 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964085 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16964085 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-09 14:50:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-09 14:50:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1890 try 0 launched on executor multi with executor id 1891 parsl.dataflow.dflow INFO: Standard output for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/09c368f0-2470-4d94-9428-e8e044ff43e6_isrForCrosstalkSources_855407_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/09c368f0-2470-4d94-9428-e8e044ff43e6_isrForCrosstalkSources_855407_59.stderr parsl.dataflow.dflow DEBUG: Task 1891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1891 has memoization hash dbbc7f409dbb559a2390ef54b406020f parsl.dataflow.memoization INFO: Task 1891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1891 try 0 launched on executor multi with executor id 1892 parsl.dataflow.dflow INFO: Standard output for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/fe542dbc-2f8f-482e-ac0f-8d51f159eee0_isrForCrosstalkSources_855410_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/fe542dbc-2f8f-482e-ac0f-8d51f159eee0_isrForCrosstalkSources_855410_22.stderr parsl.dataflow.dflow DEBUG: Task 1892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1892 has memoization hash 895a5436c4b80cbf31ccb7eeb09b6f4a parsl.dataflow.memoization INFO: Task 1892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1892 try 0 launched on executor multi with executor id 1893 parsl.dataflow.dflow INFO: Standard output for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7dc4e89e-f1c0-484c-bc75-9e65d669ee15_isrForCrosstalkSources_855459_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7dc4e89e-f1c0-484c-bc75-9e65d669ee15_isrForCrosstalkSources_855459_32.stderr parsl.dataflow.dflow DEBUG: Task 1893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1893 has memoization hash ad5f855aeda9fd61c56b5fc304c9e566 parsl.dataflow.memoization INFO: Task 1893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1893 try 0 launched on executor multi with executor id 1894 parsl.dataflow.dflow INFO: Standard output for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1ea5598-d71a-4c02-9c41-6a4c1480d065_isrForCrosstalkSources_855412_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1ea5598-d71a-4c02-9c41-6a4c1480d065_isrForCrosstalkSources_855412_45.stderr parsl.dataflow.dflow DEBUG: Task 1894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1894 has memoization hash cdf88075d7f6e271f20c62b3e5592b39 parsl.dataflow.memoization INFO: Task 1894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:33 proc_lsst.multi:170 [INFO] [multi] job_id 16964086 proc_lsst.multi INFO: [multi] job_id 16964086 2024-03-09 14:50:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 14:50:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964086 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964086 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16964086 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-09 14:50:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1894 try 0 launched on executor multi with executor id 1895 parsl.dataflow.dflow INFO: Standard output for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8a785cba-f527-44ca-a17c-fd23cb9a0a3c_isrForCrosstalkSources_855412_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8a785cba-f527-44ca-a17c-fd23cb9a0a3c_isrForCrosstalkSources_855412_56.stderr parsl.dataflow.dflow DEBUG: Task 1895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1895 has memoization hash cb66f8579234545bc245b1ebfd4e6605 parsl.dataflow.memoization INFO: Task 1895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1895 try 0 launched on executor multi with executor id 1896 parsl.dataflow.dflow INFO: Standard output for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/171516e8-8a45-445c-9c47-9cb7fa908dd6_isrForCrosstalkSources_855407_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/171516e8-8a45-445c-9c47-9cb7fa908dd6_isrForCrosstalkSources_855407_42.stderr parsl.dataflow.dflow DEBUG: Task 1896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1896 has memoization hash a1513acd87f2d8f4a899284627cd407c parsl.dataflow.memoization INFO: Task 1896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1896 try 0 launched on executor multi with executor id 1897 parsl.dataflow.dflow INFO: Standard output for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/98a9e8b0-fe15-4110-9ccb-54dc63d22816_isrForCrosstalkSources_855443_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/98a9e8b0-fe15-4110-9ccb-54dc63d22816_isrForCrosstalkSources_855443_58.stderr parsl.dataflow.dflow DEBUG: Task 1897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1897 has memoization hash c250cf5da4db7ece3fb57f6de7420ae7 parsl.dataflow.memoization INFO: Task 1897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1897 try 0 launched on executor multi with executor id 1898 parsl.dataflow.dflow INFO: Standard output for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c45fe67d-f1be-4a6a-a036-530fc7c8dd3d_isrForCrosstalkSources_855393_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c45fe67d-f1be-4a6a-a036-530fc7c8dd3d_isrForCrosstalkSources_855393_44.stderr parsl.dataflow.dflow DEBUG: Task 1898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1898 has memoization hash 7057ca4392567a00bcc549310125feca parsl.dataflow.memoization INFO: Task 1898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1898 try 0 launched on executor multi with executor id 1899 parsl.dataflow.dflow INFO: Standard output for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a692f81-41e6-4970-bc54-884499c36532_isrForCrosstalkSources_855379_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a692f81-41e6-4970-bc54-884499c36532_isrForCrosstalkSources_855379_16.stderr parsl.dataflow.dflow DEBUG: Task 1899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1899 has memoization hash c390d8867bd2c4c87651957fdbca6820 parsl.dataflow.memoization INFO: Task 1899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1899 try 0 launched on executor multi with executor id 1900 parsl.dataflow.dflow INFO: Standard output for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/121a4633-0677-4480-a5de-d6f769170ccc_isrForCrosstalkSources_855414_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/121a4633-0677-4480-a5de-d6f769170ccc_isrForCrosstalkSources_855414_49.stderr parsl.dataflow.dflow DEBUG: Task 1900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1900 has memoization hash c7687b50ba338a9caf0533950c386919 parsl.dataflow.memoization INFO: Task 1900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1900 try 0 launched on executor multi with executor id 1901 parsl.dataflow.dflow INFO: Standard output for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/59af85cd-c067-46c2-ae9f-ff46e120faaf_isrForCrosstalkSources_855457_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/59af85cd-c067-46c2-ae9f-ff46e120faaf_isrForCrosstalkSources_855457_50.stderr parsl.dataflow.dflow DEBUG: Task 1901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1901 has memoization hash 5b941fcc554f9b3d9080aed599cea4d8 parsl.dataflow.memoization INFO: Task 1901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1901 try 0 launched on executor multi with executor id 1902 parsl.dataflow.dflow INFO: Standard output for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/d71e285e-cd8a-4f2b-a447-7900e9343749_isrForCrosstalkSources_855394_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/d71e285e-cd8a-4f2b-a447-7900e9343749_isrForCrosstalkSources_855394_4.stderr parsl.dataflow.dflow DEBUG: Task 1902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout 2024-03-09 14:50:33 proc_lsst.multi:170 [INFO] [multi] job_id 16964087 proc_lsst.multi INFO: [multi] job_id 16964087 2024-03-09 14:50:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 14:50:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964087 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964087 parsl.dataflow.memoization DEBUG: Task 1902 has memoization hash 1b6bd94b06888a05d7f03a0f293ff60d parsl.dataflow.memoization INFO: Task 1902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16964087 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-09 14:50:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 14:50:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1902 try 0 launched on executor multi with executor id 1903 parsl.dataflow.dflow INFO: Standard output for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e423fb9-4d30-4f31-bfd6-3b25c89cb74f_isrForCrosstalkSources_855412_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e423fb9-4d30-4f31-bfd6-3b25c89cb74f_isrForCrosstalkSources_855412_24.stderr parsl.dataflow.dflow DEBUG: Task 1903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1903 has memoization hash 05a76c201573590c8048d07899a4339c parsl.dataflow.memoization INFO: Task 1903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a674ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1903 try 0 launched on executor multi with executor id 1904 parsl.dataflow.dflow INFO: Standard output for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b9d49ebf-a5b1-4f71-8959-cc8c891a352c_isrForCrosstalkSources_855389_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b9d49ebf-a5b1-4f71-8959-cc8c891a352c_isrForCrosstalkSources_855389_11.stderr parsl.dataflow.dflow DEBUG: Task 1904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1904 has memoization hash 5ff41c01e58cbeae8d39f0070871ad81 parsl.dataflow.memoization INFO: Task 1904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a9ee82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c646d30df5c5abaf961a9b7fd25ad71a with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/59be1e37-9104-4258-a842-13482fb6ee07_isrForCrosstalkSources_855415_19.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/59be1e37-9104-4258-a842-13482fb6ee07_isrForCrosstalkSources_855415_19.stderr parsl.dataflow.dflow INFO: Parsl task 1904 try 0 launched on executor multi with executor id 1905 parsl.dataflow.dflow INFO: Standard output for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b3f07334-6780-461b-b266-3492f1abffdc_isrForCrosstalkSources_855421_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b3f07334-6780-461b-b266-3492f1abffdc_isrForCrosstalkSources_855421_46.stderr parsl.dataflow.dflow DEBUG: Task 1905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1905 has memoization hash 870b1729864e53a99a7bd58a5ad5047a parsl.dataflow.memoization INFO: Task 1905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1905 try 0 launched on executor multi with executor id 1906 parsl.dataflow.dflow INFO: Standard output for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8b4990ff-537f-4da1-99fe-a293dc3243bf_isrForCrosstalkSources_855392_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8b4990ff-537f-4da1-99fe-a293dc3243bf_isrForCrosstalkSources_855392_51.stderr parsl.dataflow.dflow DEBUG: Task 1906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1906 has memoization hash 79bfc861033631e5be4dd5a7268718a2 parsl.dataflow.memoization INFO: Task 1906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1906 try 0 launched on executor multi with executor id 1907 parsl.dataflow.dflow INFO: Standard output for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/33eb5894-e6a4-45ba-927a-e2aa9a25830d_isrForCrosstalkSources_855386_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/33eb5894-e6a4-45ba-927a-e2aa9a25830d_isrForCrosstalkSources_855386_60.stderr parsl.dataflow.dflow DEBUG: Task 1907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1907 has memoization hash 1b00780516c5543a99d78ef033cb24eb parsl.dataflow.memoization INFO: Task 1907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1907 try 0 launched on executor multi with executor id 1908 parsl.dataflow.dflow INFO: Standard output for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f8419ac3-49a7-4689-8330-a33e380dfc6c_isrForCrosstalkSources_855405_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f8419ac3-49a7-4689-8330-a33e380dfc6c_isrForCrosstalkSources_855405_57.stderr parsl.dataflow.dflow DEBUG: Task 1908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1908 has memoization hash a3c268f6aaf1262e5683e15c639f6055 parsl.dataflow.memoization INFO: Task 1908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1908 try 0 launched on executor multi with executor id 1909 parsl.dataflow.dflow INFO: Standard output for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e329a1ef-6cbd-4d53-8f87-987f06ebc622_isrForCrosstalkSources_855451_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e329a1ef-6cbd-4d53-8f87-987f06ebc622_isrForCrosstalkSources_855451_8.stderr parsl.dataflow.dflow DEBUG: Task 1909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1909 has memoization hash 6ff2f7ca27985a79f12348c0b22f5488 parsl.dataflow.memoization INFO: Task 1909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1909 try 0 launched on executor multi with executor id 1910 parsl.dataflow.dflow INFO: Standard output for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1a135b36-98b9-4092-b243-296e39482c65_isrForCrosstalkSources_855439_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1a135b36-98b9-4092-b243-296e39482c65_isrForCrosstalkSources_855439_55.stderr parsl.dataflow.dflow DEBUG: Task 1910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr 2024-03-09 14:50:34 proc_lsst.multi:170 [INFO] [multi] job_id 16964088 proc_lsst.multi INFO: [multi] job_id 16964088 2024-03-09 14:50:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 14:50:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964088 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964088 parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1910 has memoization hash f92b4420ef220f20c58d63dfc91adca8 parsl.dataflow.memoization INFO: Task 1910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6804f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16964088 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-09 14:50:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1910 try 0 launched on executor multi with executor id 1911 parsl.dataflow.dflow INFO: Standard output for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/3577d9c2-b7f8-4eff-8844-8a9123ad38fa_isrForCrosstalkSources_855384_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/3577d9c2-b7f8-4eff-8844-8a9123ad38fa_isrForCrosstalkSources_855384_52.stderr parsl.dataflow.dflow DEBUG: Task 1911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1911 has memoization hash bd7701fe557eb147870908cf409d43de parsl.dataflow.memoization INFO: Task 1911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:34 proc_lsst.multi:170 [INFO] [multi] job_id 16964089 proc_lsst.multi INFO: [multi] job_id 16964089 2024-03-09 14:50:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 14:50:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964089 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964089 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16964089 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-09 14:50:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 14:50:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1911 try 0 launched on executor multi with executor id 1912 parsl.dataflow.dflow INFO: Standard output for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/21195a7e-bb75-4026-9fcd-ef144a982fce_isrForCrosstalkSources_855410_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/21195a7e-bb75-4026-9fcd-ef144a982fce_isrForCrosstalkSources_855410_1.stderr parsl.dataflow.dflow DEBUG: Task 1912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1912 has memoization hash efc56b9a416f3480f22c171e727cc510 parsl.dataflow.memoization INFO: Task 1912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e3be45bcd49934e89d9ae5f8f013141 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8a36f9d7-cf0d-4782-8853-5ce0c1f5453c_isrForCrosstalkSources_855406_25.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8a36f9d7-cf0d-4782-8853-5ce0c1f5453c_isrForCrosstalkSources_855406_25.stderr 2024-03-09 14:50:35 proc_lsst.multi:170 [INFO] [multi] job_id 16964090 proc_lsst.multi INFO: [multi] job_id 16964090 2024-03-09 14:50:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 14:50:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964090 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964090 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16964090 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-09 14:50:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-09 14:50:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 14:50:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d05f7d005740cf4541525585fd552071 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3bb82c5f-8e7c-45ba-a91e-eb517e84535e_isrForCrosstalkSources_855387_46.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3bb82c5f-8e7c-45ba-a91e-eb517e84535e_isrForCrosstalkSources_855387_46.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4da9ba1f2e7c59cb7d761a09a4c2075d with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/11015b11-2f75-4d0b-a02c-f2759bff18d2_isrForCrosstalkSources_855417_15.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/11015b11-2f75-4d0b-a02c-f2759bff18d2_isrForCrosstalkSources_855417_15.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8698971eadf711f67f44ff57637171 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0245fed-668e-45ac-be65-a03f3331d3d0_isrForCrosstalkSources_855388_24.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0245fed-668e-45ac-be65-a03f3331d3d0_isrForCrosstalkSources_855388_24.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0622f94ab343f670f9e5d1883a30ca2 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bdbb9eb5-e48d-479d-95e7-382d943fc3ff_isrForCrosstalkSources_855420_12.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bdbb9eb5-e48d-479d-95e7-382d943fc3ff_isrForCrosstalkSources_855420_12.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d5d1b12ca3e4997bf5f1c2bfeb32d04 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7ee20a5b-c053-42fe-8506-1ed792996921_isrForCrosstalkSources_855446_6.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7ee20a5b-c053-42fe-8506-1ed792996921_isrForCrosstalkSources_855446_6.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc056b43b1f14c3c4c3595e2a0083772 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/676d8b5d-b321-473f-a231-dc3001974725_isrForCrosstalkSources_855406_12.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/676d8b5d-b321-473f-a231-dc3001974725_isrForCrosstalkSources_855406_12.stderr 2024-03-09 14:50:38 proc_lsst.multi:170 [INFO] [multi] job_id 16964091 proc_lsst.multi INFO: [multi] job_id 16964091 2024-03-09 14:50:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 14:50:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964091 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964091 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16964091 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-09 14:50:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-09 14:50:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 14:50:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29224622adb208294893572e298633f5 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0c2a7950-45ff-47b5-a565-20873c31c3a5_isrForCrosstalkSources_855425_29.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0c2a7950-45ff-47b5-a565-20873c31c3a5_isrForCrosstalkSources_855425_29.stderr parsl.dataflow.dflow INFO: Parsl task 1912 try 0 launched on executor multi with executor id 1913 parsl.dataflow.dflow INFO: Standard output for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e2e6a2b6-59f7-4de6-ad5a-91ef1d0341a0_isrForCrosstalkSources_855392_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e2e6a2b6-59f7-4de6-ad5a-91ef1d0341a0_isrForCrosstalkSources_855392_30.stderr parsl.dataflow.dflow DEBUG: Task 1913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1913 has memoization hash b02e7341736d39bf99bf9d8735da53b0 parsl.dataflow.memoization INFO: Task 1913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1913 try 0 launched on executor multi with executor id 1914 parsl.dataflow.dflow INFO: Standard output for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e6e364e2-6d21-4928-ae7e-e487d6fe67a3_isrForCrosstalkSources_855406_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e6e364e2-6d21-4928-ae7e-e487d6fe67a3_isrForCrosstalkSources_855406_41.stderr parsl.dataflow.dflow DEBUG: Task 1914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1914 has memoization hash 8fc79ad30e237e723ae591f255f8b3dc parsl.dataflow.memoization INFO: Task 1914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1914 try 0 launched on executor multi with executor id 1915 parsl.dataflow.dflow INFO: Standard output for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b2e283e1-eddf-4f82-8fac-44d98ee934e4_isrForCrosstalkSources_855402_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b2e283e1-eddf-4f82-8fac-44d98ee934e4_isrForCrosstalkSources_855402_31.stderr parsl.dataflow.dflow DEBUG: Task 1915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1915 has memoization hash fdb16412fd2b038e7d5837e8651d3d0b parsl.dataflow.memoization INFO: Task 1915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1915 try 0 launched on executor multi with executor id 1916 parsl.dataflow.dflow INFO: Standard output for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/b1c8267a-cbdb-44a1-81a8-c0ce77b00bfa_isrForCrosstalkSources_855387_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/b1c8267a-cbdb-44a1-81a8-c0ce77b00bfa_isrForCrosstalkSources_855387_55.stderr parsl.dataflow.dflow DEBUG: Task 1916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1916 has memoization hash f94040856128cec5c331c86e6f0df529 parsl.dataflow.memoization INFO: Task 1916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1916 try 0 launched on executor multi with executor id 1917 parsl.dataflow.dflow INFO: Standard output for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/264f50e2-b534-4dbe-bb04-25b3e14cc334_isrForCrosstalkSources_855397_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/264f50e2-b534-4dbe-bb04-25b3e14cc334_isrForCrosstalkSources_855397_31.stderr parsl.dataflow.dflow DEBUG: Task 1917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1917 has memoization hash 34e9be1fa6c8d760b1c07ca02ff847f0 parsl.dataflow.memoization INFO: Task 1917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1917 try 0 launched on executor multi with executor id 1918 parsl.dataflow.dflow INFO: Standard output for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/2e267d5f-8875-4758-8816-c510fd7d766f_isrForCrosstalkSources_855431_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/2e267d5f-8875-4758-8816-c510fd7d766f_isrForCrosstalkSources_855431_58.stderr parsl.dataflow.dflow DEBUG: Task 1918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1918 has memoization hash 6ece644447128fa9a0cef4ab7b17e8d0 parsl.dataflow.memoization INFO: Task 1918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1918 try 0 launched on executor multi with executor id 1919 parsl.dataflow.dflow INFO: Standard output for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/86524e44-ef25-46cd-b5dc-d4b510c17ab8_isrForCrosstalkSources_855424_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/86524e44-ef25-46cd-b5dc-d4b510c17ab8_isrForCrosstalkSources_855424_7.stderr parsl.dataflow.dflow DEBUG: Task 1919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1919 has memoization hash 100a5d7da982b2e8ee146ea753f133f0 parsl.dataflow.memoization INFO: Task 1919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1919 try 0 launched on executor multi with executor id 1920 parsl.dataflow.dflow INFO: Standard output for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d8de3be5-9645-4fba-b414-ae559ac5b33d_isrForCrosstalkSources_855389_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d8de3be5-9645-4fba-b414-ae559ac5b33d_isrForCrosstalkSources_855389_34.stderr parsl.dataflow.dflow DEBUG: Task 1920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1920 has memoization hash fdd2247f958bc23aa2ed8052fa6556e3 parsl.dataflow.memoization INFO: Task 1920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1920 try 0 launched on executor multi with executor id 1921 parsl.dataflow.dflow INFO: Standard output for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/db09b754-5d2b-42aa-b271-e795ead71bbe_isrForCrosstalkSources_855455_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/db09b754-5d2b-42aa-b271-e795ead71bbe_isrForCrosstalkSources_855455_29.stderr parsl.dataflow.dflow DEBUG: Task 1921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1921 has memoization hash eb0527a5969d4e1e1b2043e390dedbe0 parsl.dataflow.memoization INFO: Task 1921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:39 proc_lsst.multi:170 [INFO] [multi] job_id 16964092 proc_lsst.multi INFO: [multi] job_id 16964092 2024-03-09 14:50:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 14:50:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964092 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964092 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16964092 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-09 14:50:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1921 try 0 launched on executor multi with executor id 1922 parsl.dataflow.dflow INFO: Standard output for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/940dc7ff-7a7d-457e-8ab6-08cbf3d3853c_isrForCrosstalkSources_855429_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/940dc7ff-7a7d-457e-8ab6-08cbf3d3853c_isrForCrosstalkSources_855429_39.stderr parsl.dataflow.dflow DEBUG: Task 1922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1922 has memoization hash 431a1c04f332c44832afe15b9db556da parsl.dataflow.memoization INFO: Task 1922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6806340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1922 try 0 launched on executor multi with executor id 1923 parsl.dataflow.dflow INFO: Standard output for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2b9ce8c8-2d0f-4615-a7e7-f5c1b089809f_isrForCrosstalkSources_855409_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2b9ce8c8-2d0f-4615-a7e7-f5c1b089809f_isrForCrosstalkSources_855409_44.stderr parsl.dataflow.dflow DEBUG: Task 1923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1923 has memoization hash b4a42ae73d03b07138a72ff77cac629a parsl.dataflow.memoization INFO: Task 1923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1923 try 0 launched on executor multi with executor id 1924 parsl.dataflow.dflow INFO: Standard output for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/632a077a-4a2e-45b5-8380-f4d81fc98397_isrForCrosstalkSources_855402_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/632a077a-4a2e-45b5-8380-f4d81fc98397_isrForCrosstalkSources_855402_6.stderr parsl.dataflow.dflow DEBUG: Task 1924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1924 has memoization hash e2863a77775343fa2ab431eb61dfe373 parsl.dataflow.memoization INFO: Task 1924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6805f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34d38d17f363013118cd1a43b093f349 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ea27ad5f-2308-4707-9d89-11196a52c900_isrForCrosstalkSources_855416_47.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ea27ad5f-2308-4707-9d89-11196a52c900_isrForCrosstalkSources_855416_47.stderr parsl.dataflow.dflow INFO: Parsl task 1924 try 0 launched on executor multi with executor id 1925 parsl.dataflow.dflow INFO: Standard output for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/499d1780-4255-42fc-a166-2ee06a63996f_isrForCrosstalkSources_855440_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/499d1780-4255-42fc-a166-2ee06a63996f_isrForCrosstalkSources_855440_44.stderr parsl.dataflow.dflow DEBUG: Task 1925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1925 has memoization hash 429de8704ffe33634e270d28d5a8334d parsl.dataflow.memoization INFO: Task 1925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0ea67d43bc5109ee5afd7030e52db2c with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3ad48710-90a3-429c-b3b7-13367f656c83_isrForCrosstalkSources_855457_53.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3ad48710-90a3-429c-b3b7-13367f656c83_isrForCrosstalkSources_855457_53.stderr parsl.dataflow.dflow INFO: Parsl task 1925 try 0 launched on executor multi with executor id 1926 parsl.dataflow.dflow INFO: Standard output for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9eb1faf3-0daf-4faa-886c-59d78f33b402_isrForCrosstalkSources_855382_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9eb1faf3-0daf-4faa-886c-59d78f33b402_isrForCrosstalkSources_855382_62.stderr parsl.dataflow.dflow DEBUG: Task 1926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1926 has memoization hash 808202056f1674cde653c29894bfdae0 parsl.dataflow.memoization INFO: Task 1926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1926 try 0 launched on executor multi with executor id 1927 parsl.dataflow.dflow INFO: Standard output for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2264d4ab-f993-4b31-abd4-e90fbb39d7f9_isrForCrosstalkSources_855404_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2264d4ab-f993-4b31-abd4-e90fbb39d7f9_isrForCrosstalkSources_855404_48.stderr parsl.dataflow.dflow DEBUG: Task 1927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1927 has memoization hash 9f56117b218ab6a165120ce77c1bf64b parsl.dataflow.memoization INFO: Task 1927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1927 try 0 launched on executor multi with executor id 1928 parsl.dataflow.dflow INFO: Standard output for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2df23423-f927-4549-8b75-55533ee5073c_isrForCrosstalkSources_855441_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2df23423-f927-4549-8b75-55533ee5073c_isrForCrosstalkSources_855441_62.stderr parsl.dataflow.dflow DEBUG: Task 1928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1928 has memoization hash c2e52ea9baa6803319af51e072743f7e parsl.dataflow.memoization INFO: Task 1928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1928 try 0 launched on executor multi with executor id 1929 parsl.dataflow.dflow INFO: Standard output for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1113e670-6436-4030-b097-0754a87fb00c_isrForCrosstalkSources_855424_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1113e670-6436-4030-b097-0754a87fb00c_isrForCrosstalkSources_855424_56.stderr parsl.dataflow.dflow DEBUG: Task 1929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1929 has memoization hash 9b98f96c7f9bb8d57e06a2dc7fea1707 parsl.dataflow.memoization INFO: Task 1929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1929 try 0 launched on executor multi with executor id 1930 parsl.dataflow.dflow INFO: Standard output for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/65041afa-92e9-4393-b94e-b3e3c04f94df_isrForCrosstalkSources_855422_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/65041afa-92e9-4393-b94e-b3e3c04f94df_isrForCrosstalkSources_855422_8.stderr parsl.dataflow.dflow DEBUG: Task 1930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1930 has memoization hash 3c9953660bd0c8a527f0c3f3bf040f35 parsl.dataflow.memoization INFO: Task 1930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:39 proc_lsst.multi:170 [INFO] [multi] job_id 16964093 proc_lsst.multi INFO: [multi] job_id 16964093 2024-03-09 14:50:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 14:50:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964093 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964093 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16964093 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-09 14:50:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1930 try 0 launched on executor multi with executor id 1931 parsl.dataflow.dflow INFO: Standard output for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bec1640a-e947-43b0-89c9-374608d593f0_isrForCrosstalkSources_855396_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bec1640a-e947-43b0-89c9-374608d593f0_isrForCrosstalkSources_855396_7.stderr parsl.dataflow.dflow DEBUG: Task 1931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1931 has memoization hash 754dc4d0e0fdd48ffc74748c2875e704 parsl.dataflow.memoization INFO: Task 1931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1931 try 0 launched on executor multi with executor id 1932 parsl.dataflow.dflow INFO: Standard output for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a61d5f90-15b9-4325-a461-bd45cf7e984d_isrForCrosstalkSources_855408_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a61d5f90-15b9-4325-a461-bd45cf7e984d_isrForCrosstalkSources_855408_32.stderr parsl.dataflow.dflow DEBUG: Task 1932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1932 has memoization hash 0255c703a430fbd75fda354b6a1b1764 parsl.dataflow.memoization INFO: Task 1932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1932 try 0 launched on executor multi with executor id 1933 parsl.dataflow.dflow INFO: Standard output for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b00798bf-8bcf-41c2-880c-308d9a85a370_isrForCrosstalkSources_855439_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b00798bf-8bcf-41c2-880c-308d9a85a370_isrForCrosstalkSources_855439_1.stderr parsl.dataflow.dflow DEBUG: Task 1933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1933 has memoization hash 0caf66eb79b2929cfa0c9f147d3c9c4f parsl.dataflow.memoization INFO: Task 1933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1933 try 0 launched on executor multi with executor id 1934 parsl.dataflow.dflow INFO: Standard output for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0e07786e-be2f-4290-a2af-91064a7a9d2a_isrForCrosstalkSources_855384_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0e07786e-be2f-4290-a2af-91064a7a9d2a_isrForCrosstalkSources_855384_36.stderr parsl.dataflow.dflow DEBUG: Task 1934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1934 has memoization hash e846602ca252a9a168d0794d7ecd2001 parsl.dataflow.memoization INFO: Task 1934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52572a212fb501cbf9715e089c73061c with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/57f1a0fe-e744-4673-99c9-b0dc2ccf4d6f_isrForCrosstalkSources_855395_47.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/57f1a0fe-e744-4673-99c9-b0dc2ccf4d6f_isrForCrosstalkSources_855395_47.stderr 2024-03-09 14:50:39 proc_lsst.multi:170 [INFO] [multi] job_id 16964094 proc_lsst.multi INFO: [multi] job_id 16964094 2024-03-09 14:50:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 14:50:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964094 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964094 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16964094 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-09 14:50:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 14:50:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b140f164b696b1b3348d7c9c9b3b5d4d with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/341f4089-1e42-46db-91ca-288da28dcac6_isrForCrosstalkSources_855452_49.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/341f4089-1e42-46db-91ca-288da28dcac6_isrForCrosstalkSources_855452_49.stderr 2024-03-09 14:50:40 proc_lsst.multi:170 [INFO] [multi] job_id 16964095 proc_lsst.multi INFO: [multi] job_id 16964095 2024-03-09 14:50:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 14:50:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964095 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964095 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16964095 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-09 14:50:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1934 try 0 launched on executor multi with executor id 1935 parsl.dataflow.dflow INFO: Standard output for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/850245b8-8240-44b8-9efe-4b45563a8bcf_isrForCrosstalkSources_855436_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/850245b8-8240-44b8-9efe-4b45563a8bcf_isrForCrosstalkSources_855436_32.stderr parsl.dataflow.dflow DEBUG: Task 1935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1935 has memoization hash b0ca8fa9f42b214c4296009d8a591beb parsl.dataflow.memoization INFO: Task 1935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270d48a0a1faf7d9cc758c9edf59f23a with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b822200b-c46f-41a4-bc29-45be0deb746c_isrForCrosstalkSources_855416_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b822200b-c46f-41a4-bc29-45be0deb746c_isrForCrosstalkSources_855416_27.stderr parsl.dataflow.dflow INFO: Parsl task 1935 try 0 launched on executor multi with executor id 1936 parsl.dataflow.dflow INFO: Standard output for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8ab65490-f988-48a3-a695-d217c272ec7a_isrForCrosstalkSources_855454_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8ab65490-f988-48a3-a695-d217c272ec7a_isrForCrosstalkSources_855454_20.stderr parsl.dataflow.dflow DEBUG: Task 1936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1936 has memoization hash 9d657832b22b690e24c38b9f384a1c90 parsl.dataflow.memoization INFO: Task 1936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:40 proc_lsst.multi:170 [INFO] [multi] job_id 16964096 proc_lsst.multi INFO: [multi] job_id 16964096 2024-03-09 14:50:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 14:50:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964096 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964096 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16964096 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-09 14:50:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 14:50:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28cd26354445bc8b7191785dca445449 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5e67b706-c429-4fbf-922d-c84b84ef02d6_isrForCrosstalkSources_855439_36.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5e67b706-c429-4fbf-922d-c84b84ef02d6_isrForCrosstalkSources_855439_36.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb2ce2b8b02a4a77847e9c3fb4dc3ce with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a6b6e051-f44c-48fa-b78c-4e40926eb8e8_isrForCrosstalkSources_855455_57.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a6b6e051-f44c-48fa-b78c-4e40926eb8e8_isrForCrosstalkSources_855455_57.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 053ca3e76cfce1c2f7cca2203fc7273f with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bcca0090-4ea9-42b4-8235-338868fd6b34_isrForCrosstalkSources_855396_9.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bcca0090-4ea9-42b4-8235-338868fd6b34_isrForCrosstalkSources_855396_9.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb46a85be4c13cb079d64d11ccde07b8 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f653bc54-0004-4c21-a524-0df884431f99_isrForCrosstalkSources_855409_52.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f653bc54-0004-4c21-a524-0df884431f99_isrForCrosstalkSources_855409_52.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3455ea119eef5598ad5738cec7640bf with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/5d8dd0ac-8162-4917-a416-284334efb83a_isrForCrosstalkSources_855437_12.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/5d8dd0ac-8162-4917-a416-284334efb83a_isrForCrosstalkSources_855437_12.stderr 2024-03-09 14:50:41 proc_lsst.multi:170 [INFO] [multi] job_id 16964097 proc_lsst.multi INFO: [multi] job_id 16964097 2024-03-09 14:50:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 14:50:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964097 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964097 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16964097 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-09 14:50:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-09 14:50:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 14:50:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1936 try 0 launched on executor multi with executor id 1937 parsl.dataflow.dflow INFO: Standard output for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/21e265d9-4b97-4bdf-ab2b-10288144d2be_isrForCrosstalkSources_855419_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/21e265d9-4b97-4bdf-ab2b-10288144d2be_isrForCrosstalkSources_855419_21.stderr parsl.dataflow.dflow DEBUG: Task 1937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1937 has memoization hash da1c11049e66327377a8f28460c4c609 parsl.dataflow.memoization INFO: Task 1937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:42 proc_lsst.multi:170 [INFO] [multi] job_id 16964099 proc_lsst.multi INFO: [multi] job_id 16964099 2024-03-09 14:50:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 14:50:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964099 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964099 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16964099 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-09 14:50:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-09 14:50:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 14:50:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 274cf1fe899dad91ae8740c29043781b with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/60c6f57c-5037-4d7b-8838-7445c24c159a_isrForCrosstalkSources_855428_32.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/60c6f57c-5037-4d7b-8838-7445c24c159a_isrForCrosstalkSources_855428_32.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caa393a7c1230f88f7d0a726d4eaf725 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fe4ada47-c2c7-41a6-a999-ec317e9e7414_isrForCrosstalkSources_855442_13.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fe4ada47-c2c7-41a6-a999-ec317e9e7414_isrForCrosstalkSources_855442_13.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebfa6ca5ecc85c888cb9eadd0a7b6e7 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a59e67f1-9f14-4bf6-b15e-d688a2f6219e_isrForCrosstalkSources_855434_15.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a59e67f1-9f14-4bf6-b15e-d688a2f6219e_isrForCrosstalkSources_855434_15.stderr parsl.dataflow.dflow INFO: Parsl task 1937 try 0 launched on executor multi with executor id 1938 parsl.dataflow.dflow INFO: Standard output for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/8ce61220-b620-4364-9847-da9968f9167f_isrForCrosstalkSources_855409_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/8ce61220-b620-4364-9847-da9968f9167f_isrForCrosstalkSources_855409_42.stderr parsl.dataflow.dflow DEBUG: Task 1938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1938 has memoization hash 6d7f0bbea69ad38b598366886e15fa07 parsl.dataflow.memoization INFO: Task 1938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1938 try 0 launched on executor multi with executor id 1939 parsl.dataflow.dflow INFO: Standard output for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e126bb40-ead6-4ad8-954a-d3dfc0165acd_isrForCrosstalkSources_855434_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e126bb40-ead6-4ad8-954a-d3dfc0165acd_isrForCrosstalkSources_855434_60.stderr parsl.dataflow.dflow DEBUG: Task 1939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1939 has memoization hash 7e45b9a04a88f19f1ae4f188acba2290 parsl.dataflow.memoization INFO: Task 1939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5357a08326808915125e8222718a1952 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a0d75c98-36f0-45a6-99de-7c102f1cd47b_isrForCrosstalkSources_855441_6.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a0d75c98-36f0-45a6-99de-7c102f1cd47b_isrForCrosstalkSources_855441_6.stderr 2024-03-09 14:50:43 proc_lsst.multi:170 [INFO] [multi] job_id 16964100 proc_lsst.multi INFO: [multi] job_id 16964100 2024-03-09 14:50:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 14:50:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964100 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964100 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16964100 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-09 14:50:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1939 try 0 launched on executor multi with executor id 1940 parsl.dataflow.dflow INFO: Standard output for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/29765ba4-3dbb-4a26-b60a-55381865609a_isrForCrosstalkSources_855402_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/29765ba4-3dbb-4a26-b60a-55381865609a_isrForCrosstalkSources_855402_37.stderr parsl.dataflow.dflow DEBUG: Task 1940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1940 has memoization hash b88716743f249d2f3bc318db3c3cf12f parsl.dataflow.memoization INFO: Task 1940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3155f4930706de2d41805e85098f74f9 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/500cc5f7-b8ec-4148-83c2-379041637744_isrForCrosstalkSources_855430_49.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/500cc5f7-b8ec-4148-83c2-379041637744_isrForCrosstalkSources_855430_49.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c374d6c954dbcef0194d19af79fae844 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/12b9c846-6d82-458e-bf47-24938027d510_isrForCrosstalkSources_855457_35.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/12b9c846-6d82-458e-bf47-24938027d510_isrForCrosstalkSources_855457_35.stderr parsl.dataflow.dflow INFO: Parsl task 1940 try 0 launched on executor multi with executor id 1941 parsl.dataflow.dflow INFO: Standard output for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a476bb34-a9bc-4eea-bf60-a5cd7f433a60_isrForCrosstalkSources_855434_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a476bb34-a9bc-4eea-bf60-a5cd7f433a60_isrForCrosstalkSources_855434_39.stderr parsl.dataflow.dflow DEBUG: Task 1941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1941 has memoization hash 9336087f25e308e460e944ce94ed1bbf parsl.dataflow.memoization INFO: Task 1941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1941 try 0 launched on executor multi with executor id 1942 parsl.dataflow.dflow INFO: Standard output for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/72aa123a-2599-47f1-82ca-2a725d82f23f_isrForCrosstalkSources_855393_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/72aa123a-2599-47f1-82ca-2a725d82f23f_isrForCrosstalkSources_855393_23.stderr parsl.dataflow.dflow DEBUG: Task 1942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1942 has memoization hash 01bcf8f90e86cf92956f173f913d19fd parsl.dataflow.memoization INFO: Task 1942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:43 proc_lsst.multi:170 [INFO] [multi] job_id 16964101 proc_lsst.multi INFO: [multi] job_id 16964101 2024-03-09 14:50:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964101 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964101 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16964101 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-09 14:50:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1942 try 0 launched on executor multi with executor id 1943 parsl.dataflow.dflow INFO: Standard output for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b8ebbaeb-c48f-4d72-953c-9f522be6c62a_isrForCrosstalkSources_855432_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b8ebbaeb-c48f-4d72-953c-9f522be6c62a_isrForCrosstalkSources_855432_25.stderr parsl.dataflow.dflow DEBUG: Task 1943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1943 has memoization hash a92bf2020c618f02c886d42ac5444bde parsl.dataflow.memoization INFO: Task 1943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9355d5e45a724f3d12448266ca8bd7c with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/f0a22717-b76d-457a-9975-945f2361d5cf_isrForCrosstalkSources_855406_1.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/f0a22717-b76d-457a-9975-945f2361d5cf_isrForCrosstalkSources_855406_1.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a25a0013227ee5ddd2cb4674c08c8f6e with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/10687160-3f29-4068-9916-84e5fe45fe4c_isrForCrosstalkSources_855401_37.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/10687160-3f29-4068-9916-84e5fe45fe4c_isrForCrosstalkSources_855401_37.stderr parsl.dataflow.dflow INFO: Parsl task 1943 try 0 launched on executor multi with executor id 1944 parsl.dataflow.dflow INFO: Standard output for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a129c6ed-7fa5-494d-84f3-8d42ea706761_isrForCrosstalkSources_855384_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a129c6ed-7fa5-494d-84f3-8d42ea706761_isrForCrosstalkSources_855384_4.stderr parsl.dataflow.dflow DEBUG: Task 1944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1944 has memoization hash 36e72837f835b36965f42d533b3c2be2 parsl.dataflow.memoization INFO: Task 1944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1944 try 0 launched on executor multi with executor id 1945 parsl.dataflow.dflow INFO: Standard output for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f3ba20f4-7186-4e3e-95b0-87cabf2fbfc7_isrForCrosstalkSources_855393_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f3ba20f4-7186-4e3e-95b0-87cabf2fbfc7_isrForCrosstalkSources_855393_17.stderr parsl.dataflow.dflow DEBUG: Task 1945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1945 has memoization hash 2acb04aee918732028fcbfd7a07b62da parsl.dataflow.memoization INFO: Task 1945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1945 try 0 launched on executor multi with executor id 1946 parsl.dataflow.dflow INFO: Standard output for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36fd4de0-7ad6-400a-8f59-430a8ee2b1a2_isrForCrosstalkSources_855390_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36fd4de0-7ad6-400a-8f59-430a8ee2b1a2_isrForCrosstalkSources_855390_15.stderr parsl.dataflow.dflow DEBUG: Task 1946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1946 has memoization hash 59df9c4e2fc7a7da2fd0a93eda8f8490 parsl.dataflow.memoization INFO: Task 1946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1946 try 0 launched on executor multi with executor id 1947 parsl.dataflow.dflow INFO: Standard output for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/58a6e9a4-3f11-4603-becd-6f49817981de_isrForCrosstalkSources_855448_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/58a6e9a4-3f11-4603-becd-6f49817981de_isrForCrosstalkSources_855448_22.stderr parsl.dataflow.dflow DEBUG: Task 1947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1947 has memoization hash a995fe995d38b7f30ab71d3762a0ad32 parsl.dataflow.memoization INFO: Task 1947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:43 proc_lsst.multi:170 [INFO] [multi] job_id 16964102 proc_lsst.multi INFO: [multi] job_id 16964102 2024-03-09 14:50:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 14:50:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964102 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964102 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16964102 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-09 14:50:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 14:50:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1947 try 0 launched on executor multi with executor id 1948 parsl.dataflow.dflow INFO: Standard output for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/961bc602-d802-4d2b-a13b-586dd12e4893_isrForCrosstalkSources_855397_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/961bc602-d802-4d2b-a13b-586dd12e4893_isrForCrosstalkSources_855397_56.stderr parsl.dataflow.dflow DEBUG: Task 1948 will be sent to executor multi parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1948 has memoization hash e0579c09819e2bb2583492a8a9f3712b parsl.dataflow.memoization INFO: Task 1948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1948 try 0 launched on executor multi with executor id 1949 parsl.dataflow.dflow INFO: Standard output for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/78389a41-b710-4f5c-8bbc-b7d3d412b714_isrForCrosstalkSources_855399_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/78389a41-b710-4f5c-8bbc-b7d3d412b714_isrForCrosstalkSources_855399_14.stderr parsl.dataflow.dflow DEBUG: Task 1949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1949 has memoization hash ac02a026bf13fca8aa6482639db6951a parsl.dataflow.memoization INFO: Task 1949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f098efa7771d599dedb0a3c0263616b with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7bbcdcc6-f6b5-497d-a2c9-4b15c27f3590_isrForCrosstalkSources_855440_57.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7bbcdcc6-f6b5-497d-a2c9-4b15c27f3590_isrForCrosstalkSources_855440_57.stderr parsl.dataflow.dflow INFO: Parsl task 1949 try 0 launched on executor multi with executor id 1950 parsl.dataflow.dflow INFO: Standard output for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6e5ce65d-65ba-4015-8e3f-9a4392fdb14d_isrForCrosstalkSources_855436_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6e5ce65d-65ba-4015-8e3f-9a4392fdb14d_isrForCrosstalkSources_855436_29.stderr parsl.dataflow.dflow DEBUG: Task 1950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1950 has memoization hash f789e522b82763cd457d1757ee72f02f parsl.dataflow.memoization INFO: Task 1950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1950 try 0 launched on executor multi with executor id 1951 parsl.dataflow.dflow INFO: Standard output for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e6dcff97-45ff-400b-88ed-f93754e23183_isrForCrosstalkSources_855447_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e6dcff97-45ff-400b-88ed-f93754e23183_isrForCrosstalkSources_855447_54.stderr parsl.dataflow.dflow DEBUG: Task 1951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1951 has memoization hash cf111da78bda043bd3fb5e0eba46f93a parsl.dataflow.memoization INFO: Task 1951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1951 try 0 launched on executor multi with executor id 1952 parsl.dataflow.dflow INFO: Standard output for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/250fd2bc-b6d9-4c9b-8de5-9c3a04b74d34_isrForCrosstalkSources_855459_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/250fd2bc-b6d9-4c9b-8de5-9c3a04b74d34_isrForCrosstalkSources_855459_20.stderr parsl.dataflow.dflow DEBUG: Task 1952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1952 has memoization hash 4925311349e7055b4598c74467fd0179 parsl.dataflow.memoization INFO: Task 1952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1952 try 0 launched on executor multi with executor id 1953 parsl.dataflow.dflow INFO: Standard output for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f6d529e3-ec54-4df3-9139-6dc76cc84244_isrForCrosstalkSources_855384_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f6d529e3-ec54-4df3-9139-6dc76cc84244_isrForCrosstalkSources_855384_43.stderr parsl.dataflow.dflow DEBUG: Task 1953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1953 has memoization hash 79f4ebe69ecb017b7e36287fd411aade parsl.dataflow.memoization INFO: Task 1953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1953 try 0 launched on executor multi with executor id 1954 parsl.dataflow.dflow INFO: Standard output for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c20ecdbd-11be-4605-94f1-96c6a05601d1_isrForCrosstalkSources_855456_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c20ecdbd-11be-4605-94f1-96c6a05601d1_isrForCrosstalkSources_855456_46.stderr parsl.dataflow.dflow DEBUG: Task 1954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1954 has memoization hash 00d8062b54bce6f1450a7b76bcbf3b85 parsl.dataflow.memoization INFO: Task 1954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1954 try 0 launched on executor multi with executor id 1955 parsl.dataflow.dflow INFO: Standard output for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/449b6622-843f-4e6d-b708-7a3004773eb4_isrForCrosstalkSources_855450_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/449b6622-843f-4e6d-b708-7a3004773eb4_isrForCrosstalkSources_855450_26.stderr parsl.dataflow.dflow DEBUG: Task 1955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1955 has memoization hash a4ea0b1451abd4d2671148c95640b63a parsl.dataflow.memoization INFO: Task 1955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1955 try 0 launched on executor multi with executor id 1956 parsl.dataflow.dflow INFO: Standard output for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbf36d2d-f079-4b87-b3d2-75298b7455c1_isrForCrosstalkSources_855431_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbf36d2d-f079-4b87-b3d2-75298b7455c1_isrForCrosstalkSources_855431_17.stderr parsl.dataflow.dflow DEBUG: Task 1956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1956 has memoization hash b47152d26bbd627e22de0585aa5675c1 parsl.dataflow.memoization INFO: Task 1956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:44 proc_lsst.multi:170 [INFO] [multi] job_id 16964103 proc_lsst.multi INFO: [multi] job_id 16964103 2024-03-09 14:50:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 14:50:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964103 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964103 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16964103 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-09 14:50:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1956 try 0 launched on executor multi with executor id 1957 parsl.dataflow.dflow INFO: Standard output for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8c35a59b-b7d9-4998-90f8-50be2ab1e551_isrForCrosstalkSources_855391_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8c35a59b-b7d9-4998-90f8-50be2ab1e551_isrForCrosstalkSources_855391_20.stderr parsl.dataflow.dflow DEBUG: Task 1957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1957 has memoization hash d7264e67aff37027d0c00311e4b9cfe7 parsl.dataflow.memoization INFO: Task 1957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1957 try 0 launched on executor multi with executor id 1958 parsl.dataflow.dflow INFO: Standard output for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6895d392-8be2-463f-8105-cf4ef1228024_isrForCrosstalkSources_855456_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6895d392-8be2-463f-8105-cf4ef1228024_isrForCrosstalkSources_855456_56.stderr parsl.dataflow.dflow DEBUG: Task 1958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1958 has memoization hash 12336cbde60981a01294dc2f0411c6f8 parsl.dataflow.memoization INFO: Task 1958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1958 try 0 launched on executor multi with executor id 1959 parsl.dataflow.dflow INFO: Standard output for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/55fad503-a752-4174-abf2-f5ce5b34a68d_isrForCrosstalkSources_855386_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/55fad503-a752-4174-abf2-f5ce5b34a68d_isrForCrosstalkSources_855386_47.stderr parsl.dataflow.dflow DEBUG: Task 1959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1959 has memoization hash 7043481fa64f33c4a651ab59796d9520 parsl.dataflow.memoization INFO: Task 1959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1959 try 0 launched on executor multi with executor id 1960 parsl.dataflow.dflow INFO: Standard output for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/63f1d73f-fdbb-4e5b-9069-be014fb73f59_isrForCrosstalkSources_855451_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/63f1d73f-fdbb-4e5b-9069-be014fb73f59_isrForCrosstalkSources_855451_61.stderr parsl.dataflow.dflow DEBUG: Task 1960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1960 has memoization hash 0e6810112e511ffccecb2ee8b11614d7 parsl.dataflow.memoization INFO: Task 1960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1960 try 0 launched on executor multi with executor id 1961 parsl.dataflow.dflow INFO: Standard output for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9b1d0284-62f1-4108-87cc-3325d15f29f2_isrForCrosstalkSources_855427_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9b1d0284-62f1-4108-87cc-3325d15f29f2_isrForCrosstalkSources_855427_23.stderr parsl.dataflow.dflow DEBUG: Task 1961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1961 has memoization hash f7b29b3b6d8c3d0fb42e6f5c9f32597a parsl.dataflow.memoization INFO: Task 1961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:44 proc_lsst.multi:170 [INFO] [multi] job_id 16964104 proc_lsst.multi INFO: [multi] job_id 16964104 2024-03-09 14:50:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 14:50:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964104 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964104 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16964104 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-09 14:50:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 14:50:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1961 try 0 launched on executor multi with executor id 1962 parsl.dataflow.dflow INFO: Standard output for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dde268da-04f5-43ce-b303-ff26d9fb13ba_isrForCrosstalkSources_855456_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dde268da-04f5-43ce-b303-ff26d9fb13ba_isrForCrosstalkSources_855456_41.stderr parsl.dataflow.dflow DEBUG: Task 1962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1962 has memoization hash f25fd89b71fff385e4cb917be858cbb0 parsl.dataflow.memoization INFO: Task 1962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1962 try 0 launched on executor multi with executor id 1963 parsl.dataflow.dflow INFO: Standard output for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0ac35c29-a51a-4ee8-82e0-af617690bb2b_isrForCrosstalkSources_855430_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0ac35c29-a51a-4ee8-82e0-af617690bb2b_isrForCrosstalkSources_855430_5.stderr parsl.dataflow.dflow DEBUG: Task 1963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1963 has memoization hash ead55a59a8e0bd530478fb1f001123ca parsl.dataflow.memoization INFO: Task 1963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1963 try 0 launched on executor multi with executor id 1964 parsl.dataflow.dflow INFO: Standard output for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/76c3ab63-9f5b-420f-8820-8ce827dba1d9_isrForCrosstalkSources_855410_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/76c3ab63-9f5b-420f-8820-8ce827dba1d9_isrForCrosstalkSources_855410_49.stderr parsl.dataflow.dflow DEBUG: Task 1964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1964 has memoization hash 0f7fb64041e9c80e957d5af748ab318f parsl.dataflow.memoization INFO: Task 1964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1964 try 0 launched on executor multi with executor id 1965 parsl.dataflow.dflow INFO: Standard output for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/32d12d65-9cfd-4173-8962-1ab1bac4a6cd_isrForCrosstalkSources_855444_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/32d12d65-9cfd-4173-8962-1ab1bac4a6cd_isrForCrosstalkSources_855444_25.stderr parsl.dataflow.dflow DEBUG: Task 1965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1965 has memoization hash aa27aa6a22fad22be2797d57e8ee283b parsl.dataflow.memoization INFO: Task 1965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1965 try 0 launched on executor multi with executor id 1966 parsl.dataflow.dflow INFO: Standard output for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e9e96450-7c54-47fa-a711-9a527c21f26c_isrForCrosstalkSources_855407_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e9e96450-7c54-47fa-a711-9a527c21f26c_isrForCrosstalkSources_855407_30.stderr parsl.dataflow.dflow DEBUG: Task 1966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1966 has memoization hash 7980031865fa7e23ab07d9733b125c35 parsl.dataflow.memoization INFO: Task 1966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1966 try 0 launched on executor multi with executor id 1967 parsl.dataflow.dflow INFO: Standard output for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/826cb2c0-8134-4c3b-b546-bd325759507e_isrForCrosstalkSources_855453_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/826cb2c0-8134-4c3b-b546-bd325759507e_isrForCrosstalkSources_855453_57.stderr parsl.dataflow.dflow DEBUG: Task 1967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1967 has memoization hash 35c11c795f9d6a10448af567edab7128 parsl.dataflow.memoization INFO: Task 1967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1967 try 0 launched on executor multi with executor id 1968 parsl.dataflow.dflow INFO: Standard output for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/286d3e28-9b38-4a27-a9cb-9a9cc4b2e136_isrForCrosstalkSources_855388_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/286d3e28-9b38-4a27-a9cb-9a9cc4b2e136_isrForCrosstalkSources_855388_40.stderr parsl.dataflow.dflow DEBUG: Task 1968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1968 has memoization hash bcb78f65808c370597ed5213cb8aea3c parsl.dataflow.memoization INFO: Task 1968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1968 try 0 launched on executor multi with executor id 1969 parsl.dataflow.dflow INFO: Standard output for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/23e08001-f68b-487b-beb2-c32e9af3eea8_isrForCrosstalkSources_855424_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/23e08001-f68b-487b-beb2-c32e9af3eea8_isrForCrosstalkSources_855424_33.stderr parsl.dataflow.dflow DEBUG: Task 1969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1969 has memoization hash f068bc185ef4f8528bf6a91fe379443a parsl.dataflow.memoization INFO: Task 1969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1969 try 0 launched on executor multi with executor id 1970 parsl.dataflow.dflow INFO: Standard output for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b7c909f3-c8b3-46f0-a63e-222561f37adf_isrForCrosstalkSources_855385_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b7c909f3-c8b3-46f0-a63e-222561f37adf_isrForCrosstalkSources_855385_8.stderr parsl.dataflow.dflow DEBUG: Task 1970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1970 has memoization hash 1f38251a88a22716b839de4f916636ce parsl.dataflow.memoization INFO: Task 1970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1970 try 0 launched on executor multi with executor id 1971 parsl.dataflow.dflow INFO: Standard output for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/1b42d58d-1b77-4b95-abfa-ea64bbccd5df_isrForCrosstalkSources_855449_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/1b42d58d-1b77-4b95-abfa-ea64bbccd5df_isrForCrosstalkSources_855449_23.stderr parsl.dataflow.dflow DEBUG: Task 1971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1971 has memoization hash ccbc29f71476092d2f81a36b88da7007 parsl.dataflow.memoization INFO: Task 1971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1971 try 0 launched on executor multi with executor id 1972 parsl.dataflow.dflow INFO: Standard output for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a25d29e6-66e9-4c4f-9cde-049f27e6cff6_isrForCrosstalkSources_855401_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a25d29e6-66e9-4c4f-9cde-049f27e6cff6_isrForCrosstalkSources_855401_42.stderr parsl.dataflow.dflow DEBUG: Task 1972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1972 has memoization hash 9267df5cc8d037bb859848b873201cdf parsl.dataflow.memoization INFO: Task 1972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1972 try 0 launched on executor multi with executor id 1973 parsl.dataflow.dflow INFO: Standard output for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a82587f8-603a-4ca7-97a3-443abad24d43_isrForCrosstalkSources_855415_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a82587f8-603a-4ca7-97a3-443abad24d43_isrForCrosstalkSources_855415_31.stderr parsl.dataflow.dflow DEBUG: Task 1973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1973 has memoization hash 57d44c140458e6e57941f2504b166804 parsl.dataflow.memoization INFO: Task 1973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1973 try 0 launched on executor multi with executor id 1974 parsl.dataflow.dflow INFO: Standard output for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/db595576-d727-46f6-95b0-ccf44e84fabf_isrForCrosstalkSources_855416_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/db595576-d727-46f6-95b0-ccf44e84fabf_isrForCrosstalkSources_855416_24.stderr parsl.dataflow.dflow DEBUG: Task 1974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1974 has memoization hash ccacad4239eec19dd5ab0e6a81d052d5 parsl.dataflow.memoization INFO: Task 1974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:46 proc_lsst.multi:170 [INFO] [multi] job_id 16964105 proc_lsst.multi INFO: [multi] job_id 16964105 2024-03-09 14:50:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 14:50:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964105 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964105 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16964105 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-09 14:50:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1974 try 0 launched on executor multi with executor id 1975 parsl.dataflow.dflow INFO: Standard output for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b1986399-46f8-4043-9a91-b0c8b01f3062_isrForCrosstalkSources_855420_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b1986399-46f8-4043-9a91-b0c8b01f3062_isrForCrosstalkSources_855420_49.stderr parsl.dataflow.dflow DEBUG: Task 1975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1975 has memoization hash 708647a803c9fa56086238256ed21dba parsl.dataflow.memoization INFO: Task 1975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1975 try 0 launched on executor multi with executor id 1976 parsl.dataflow.dflow INFO: Standard output for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d90f960-bd89-4928-9d57-d9edaa64e95f_isrForCrosstalkSources_855459_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d90f960-bd89-4928-9d57-d9edaa64e95f_isrForCrosstalkSources_855459_36.stderr parsl.dataflow.dflow DEBUG: Task 1976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1976 has memoization hash d8d20c7f0d04e9ab3869474678c16c88 parsl.dataflow.memoization INFO: Task 1976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1976 try 0 launched on executor multi with executor id 1977 parsl.dataflow.dflow INFO: Standard output for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7a9e82a0-1b98-4f28-a556-ec6bbb9d3952_isrForCrosstalkSources_855402_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7a9e82a0-1b98-4f28-a556-ec6bbb9d3952_isrForCrosstalkSources_855402_9.stderr parsl.dataflow.dflow DEBUG: Task 1977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1977 has memoization hash 1f6a61fef0fa522c13c38653ab174909 parsl.dataflow.memoization INFO: Task 1977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1977 try 0 launched on executor multi with executor id 1978 parsl.dataflow.dflow INFO: Standard output for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/01e9aebf-9640-4ce8-9cec-854de39acb74_isrForCrosstalkSources_855457_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/01e9aebf-9640-4ce8-9cec-854de39acb74_isrForCrosstalkSources_855457_18.stderr parsl.dataflow.dflow DEBUG: Task 1978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1978 has memoization hash 55cee7b48badbb113c808e8e401323ff parsl.dataflow.memoization INFO: Task 1978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1978 try 0 launched on executor multi with executor id 1979 parsl.dataflow.dflow INFO: Standard output for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/18770fe1-8839-4a78-b345-0eabaab45efd_isrForCrosstalkSources_855445_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/18770fe1-8839-4a78-b345-0eabaab45efd_isrForCrosstalkSources_855445_47.stderr parsl.dataflow.dflow DEBUG: Task 1979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1979 has memoization hash ed36760a194ac88104db54c3923cdb5e parsl.dataflow.memoization INFO: Task 1979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1979 try 0 launched on executor multi with executor id 1980 parsl.dataflow.dflow INFO: Standard output for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7e707fae-158d-442e-ad93-c2022a669134_isrForCrosstalkSources_855421_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7e707fae-158d-442e-ad93-c2022a669134_isrForCrosstalkSources_855421_24.stderr parsl.dataflow.dflow DEBUG: Task 1980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1980 has memoization hash 0d7e58c2351166decd62fcad48f05c9c parsl.dataflow.memoization INFO: Task 1980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1980 try 0 launched on executor multi with executor id 1981 parsl.dataflow.dflow INFO: Standard output for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a8ae788d-5e07-46b6-88e9-b8ca659efcec_isrForCrosstalkSources_855442_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a8ae788d-5e07-46b6-88e9-b8ca659efcec_isrForCrosstalkSources_855442_25.stderr parsl.dataflow.dflow DEBUG: Task 1981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1981 has memoization hash fdeba43162ca2ff2ca2e08a79b15069d parsl.dataflow.memoization INFO: Task 1981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1981 try 0 launched on executor multi with executor id 1982 parsl.dataflow.dflow INFO: Standard output for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/94569c96-1225-4f42-aa66-5dd75579b88f_isrForCrosstalkSources_855385_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/94569c96-1225-4f42-aa66-5dd75579b88f_isrForCrosstalkSources_855385_31.stderr parsl.dataflow.dflow DEBUG: Task 1982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1982 has memoization hash b23116d579bd1f5b67db5411ced228cf parsl.dataflow.memoization INFO: Task 1982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1982 try 0 launched on executor multi with executor id 1983 parsl.dataflow.dflow INFO: Standard output for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10911095-9f38-4e8c-85a2-01dc73263e6b_isrForCrosstalkSources_855412_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10911095-9f38-4e8c-85a2-01dc73263e6b_isrForCrosstalkSources_855412_7.stderr parsl.dataflow.dflow DEBUG: Task 1983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1983 has memoization hash b03730bef1c96def8b3befe1fbd30dc9 parsl.dataflow.memoization INFO: Task 1983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1983 try 0 launched on executor multi with executor id 1984 parsl.dataflow.dflow INFO: Standard output for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e672a244-2f45-460e-800a-1890050a0227_isrForCrosstalkSources_855451_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e672a244-2f45-460e-800a-1890050a0227_isrForCrosstalkSources_855451_50.stderr parsl.dataflow.dflow DEBUG: Task 1984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1984 has memoization hash 2447f7525e8ae69ecf162183fd6b9895 parsl.dataflow.memoization INFO: Task 1984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1984 try 0 launched on executor multi with executor id 1985 parsl.dataflow.dflow INFO: Standard output for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/534f5b42-d310-4d58-afd2-9c9957cb47fb_isrForCrosstalkSources_855454_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/534f5b42-d310-4d58-afd2-9c9957cb47fb_isrForCrosstalkSources_855454_47.stderr parsl.dataflow.dflow DEBUG: Task 1985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1985 has memoization hash 6cca5ff3b391d4a5ccb2124fea9e115c parsl.dataflow.memoization INFO: Task 1985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1985 try 0 launched on executor multi with executor id 1986 parsl.dataflow.dflow INFO: Standard output for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/704b92d2-59ef-4efd-b43a-afd3c6cebadb_isrForCrosstalkSources_855438_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/704b92d2-59ef-4efd-b43a-afd3c6cebadb_isrForCrosstalkSources_855438_51.stderr parsl.dataflow.dflow DEBUG: Task 1986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1986 has memoization hash 9c069223f213b2228b332bb34cbf5f1e parsl.dataflow.memoization INFO: Task 1986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1986 try 0 launched on executor multi with executor id 1987 parsl.dataflow.dflow INFO: Standard output for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/63b8c2de-f8ab-4d97-b9fb-5737f5c54617_isrForCrosstalkSources_855383_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/63b8c2de-f8ab-4d97-b9fb-5737f5c54617_isrForCrosstalkSources_855383_51.stderr parsl.dataflow.dflow DEBUG: Task 1987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1987 has memoization hash 2c419c3f4c9b4cf1f881dc970029efc3 parsl.dataflow.memoization INFO: Task 1987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1987 try 0 launched on executor multi with executor id 1988 parsl.dataflow.dflow INFO: Standard output for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fcd71c4a-3355-46da-849e-07fd561ffbdc_isrForCrosstalkSources_855412_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fcd71c4a-3355-46da-849e-07fd561ffbdc_isrForCrosstalkSources_855412_3.stderr parsl.dataflow.dflow DEBUG: Task 1988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1988 has memoization hash c0b68b3aca6bab55fca45205f65e3e22 parsl.dataflow.memoization INFO: Task 1988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1988 try 0 launched on executor multi with executor id 1989 parsl.dataflow.dflow INFO: Standard output for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/587399b0-894e-4b0f-94f6-0c39693e5a60_isrForCrosstalkSources_855394_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/587399b0-894e-4b0f-94f6-0c39693e5a60_isrForCrosstalkSources_855394_9.stderr parsl.dataflow.dflow DEBUG: Task 1989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1989 has memoization hash 5d4a367932c4bc03e315a0cef96de3d0 parsl.dataflow.memoization INFO: Task 1989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1989 try 0 launched on executor multi with executor id 1990 parsl.dataflow.dflow INFO: Standard output for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c2be462b-ebff-4e7e-a94e-c053a2a43ecd_isrForCrosstalkSources_855410_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c2be462b-ebff-4e7e-a94e-c053a2a43ecd_isrForCrosstalkSources_855410_50.stderr parsl.dataflow.dflow DEBUG: Task 1990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1990 has memoization hash b4c0d614a6781806fb43d915b9e2ad7d parsl.dataflow.memoization INFO: Task 1990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:46 proc_lsst.multi:170 [INFO] [multi] job_id 16964106 proc_lsst.multi INFO: [multi] job_id 16964106 2024-03-09 14:50:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 14:50:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964106 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964106 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16964106 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-09 14:50:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 14:50:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1990 try 0 launched on executor multi with executor id 1991 parsl.dataflow.dflow INFO: Standard output for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/75e2ffb3-6754-436f-9f4b-db03ab2e175b_isrForCrosstalkSources_855440_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/75e2ffb3-6754-436f-9f4b-db03ab2e175b_isrForCrosstalkSources_855440_30.stderr parsl.dataflow.dflow DEBUG: Task 1991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1991 has memoization hash ecf8af5e8768822741673516d7981da5 parsl.dataflow.memoization INFO: Task 1991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1991 try 0 launched on executor multi with executor id 1992 parsl.dataflow.dflow INFO: Standard output for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4a11e37-7432-452b-8ace-c642b0aa8978_isrForCrosstalkSources_855416_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4a11e37-7432-452b-8ace-c642b0aa8978_isrForCrosstalkSources_855416_28.stderr parsl.dataflow.dflow DEBUG: Task 1992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1992 has memoization hash eedd02a0fe00a530abcef8f64abf3b00 parsl.dataflow.memoization INFO: Task 1992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1992 try 0 launched on executor multi with executor id 1993 parsl.dataflow.dflow INFO: Standard output for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6f7778e2-edbc-437e-8f5f-73ec8ede16f6_isrForCrosstalkSources_855431_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6f7778e2-edbc-437e-8f5f-73ec8ede16f6_isrForCrosstalkSources_855431_24.stderr parsl.dataflow.dflow DEBUG: Task 1993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1993 has memoization hash 9ed6ddeca0855ced9265b9f9b50bd8b0 parsl.dataflow.memoization INFO: Task 1993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1993 try 0 launched on executor multi with executor id 1994 parsl.dataflow.dflow INFO: Standard output for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/fd320069-d73d-45d6-af2f-7f3e4865041e_isrForCrosstalkSources_855446_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/fd320069-d73d-45d6-af2f-7f3e4865041e_isrForCrosstalkSources_855446_5.stderr parsl.dataflow.dflow DEBUG: Task 1994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1994 has memoization hash c70a28d0b89ea66d9e06cc9c94d4470a parsl.dataflow.memoization INFO: Task 1994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1994 try 0 launched on executor multi with executor id 1995 parsl.dataflow.dflow INFO: Standard output for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e78afb0-3815-4d4f-aa98-7299d01d489b_isrForCrosstalkSources_855408_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e78afb0-3815-4d4f-aa98-7299d01d489b_isrForCrosstalkSources_855408_1.stderr parsl.dataflow.dflow DEBUG: Task 1995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1995 has memoization hash 31101982eb8a0a1e843e2350eafa4218 parsl.dataflow.memoization INFO: Task 1995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1995 try 0 launched on executor multi with executor id 1996 parsl.dataflow.dflow INFO: Standard output for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d67df830-cf9c-4057-8e4a-59c711403029_isrForCrosstalkSources_855392_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d67df830-cf9c-4057-8e4a-59c711403029_isrForCrosstalkSources_855392_55.stderr parsl.dataflow.dflow DEBUG: Task 1996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1996 has memoization hash 98dbc79d803cc363ea36d402b8fb07c7 parsl.dataflow.memoization INFO: Task 1996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1996 try 0 launched on executor multi with executor id 1997 parsl.dataflow.dflow INFO: Standard output for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a584c91c-ba0f-4996-9884-13965bc4e644_isrForCrosstalkSources_855384_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a584c91c-ba0f-4996-9884-13965bc4e644_isrForCrosstalkSources_855384_34.stderr parsl.dataflow.dflow DEBUG: Task 1997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1997 has memoization hash a85992ebbedc5c4f48671b284f92c07f parsl.dataflow.memoization INFO: Task 1997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1997 try 0 launched on executor multi with executor id 1998 parsl.dataflow.dflow INFO: Standard output for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5aa24a98-c1fa-4fcf-b8e0-c09b0d1db49b_isrForCrosstalkSources_855425_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5aa24a98-c1fa-4fcf-b8e0-c09b0d1db49b_isrForCrosstalkSources_855425_53.stderr parsl.dataflow.dflow DEBUG: Task 1998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1998 has memoization hash 08185ec1cee834382cd40469adc59ee9 parsl.dataflow.memoization INFO: Task 1998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1998 try 0 launched on executor multi with executor id 1999 parsl.dataflow.dflow INFO: Standard output for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8cc765be-0e5f-48ed-ab5b-2212ec6febac_isrForCrosstalkSources_855398_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8cc765be-0e5f-48ed-ab5b-2212ec6febac_isrForCrosstalkSources_855398_41.stderr parsl.dataflow.dflow DEBUG: Task 1999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1999 has memoization hash 3ef3188a80d6aea267d7964a50f6c601 parsl.dataflow.memoization INFO: Task 1999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1999 try 0 launched on executor multi with executor id 2000 parsl.dataflow.dflow INFO: Standard output for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7d6b8a24-a7cb-4f19-aa6f-835efe763a5f_isrForCrosstalkSources_855449_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7d6b8a24-a7cb-4f19-aa6f-835efe763a5f_isrForCrosstalkSources_855449_34.stderr parsl.dataflow.dflow DEBUG: Task 2000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2000 has memoization hash e4b5a4404841f1d2fa93241fe8471ac1 parsl.dataflow.memoization INFO: Task 2000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2000 try 0 launched on executor multi with executor id 2001 parsl.dataflow.dflow INFO: Standard output for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/57ca9624-bd15-4ee6-81b9-8cc380f63c02_isrForCrosstalkSources_855438_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/57ca9624-bd15-4ee6-81b9-8cc380f63c02_isrForCrosstalkSources_855438_49.stderr parsl.dataflow.dflow DEBUG: Task 2001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2001 has memoization hash 1941404f65be821f3038aabab301782e parsl.dataflow.memoization INFO: Task 2001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6674ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2001 try 0 launched on executor multi with executor id 2002 parsl.dataflow.dflow INFO: Standard output for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/2986b154-ee66-4c68-94db-8ffe39f0cfff_isrForCrosstalkSources_855458_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/2986b154-ee66-4c68-94db-8ffe39f0cfff_isrForCrosstalkSources_855458_13.stderr parsl.dataflow.dflow DEBUG: Task 2002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2002 has memoization hash aa891eb5f1aa886529daf8a7ce995b49 parsl.dataflow.memoization INFO: Task 2002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2002 try 0 launched on executor multi with executor id 2003 parsl.dataflow.dflow INFO: Standard output for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/3f5f8e1d-9433-48f8-a4f2-3b205378eb80_isrForCrosstalkSources_855437_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/3f5f8e1d-9433-48f8-a4f2-3b205378eb80_isrForCrosstalkSources_855437_52.stderr parsl.dataflow.dflow DEBUG: Task 2003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2003 has memoization hash 48908e2de20b2a144e02c224f00c15bb parsl.dataflow.memoization INFO: Task 2003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2003 try 0 launched on executor multi with executor id 2004 parsl.dataflow.dflow INFO: Standard output for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f9db51d2-3dcd-42f0-912a-27c2a5a39b9a_isrForCrosstalkSources_855395_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f9db51d2-3dcd-42f0-912a-27c2a5a39b9a_isrForCrosstalkSources_855395_56.stderr parsl.dataflow.dflow DEBUG: Task 2004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2004 has memoization hash aaec8c0b43492714d1919d474d81bf70 parsl.dataflow.memoization INFO: Task 2004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2004 try 0 launched on executor multi with executor id 2005 parsl.dataflow.dflow INFO: Standard output for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/cd56e708-4268-44b4-ad57-3b40635c0da4_isrForCrosstalkSources_855424_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/cd56e708-4268-44b4-ad57-3b40635c0da4_isrForCrosstalkSources_855424_46.stderr parsl.dataflow.dflow DEBUG: Task 2005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2005 has memoization hash 712fbd8d37c4bc05598c522e4a7f1c03 parsl.dataflow.memoization INFO: Task 2005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:47 proc_lsst.multi:170 [INFO] [multi] job_id 16964107 proc_lsst.multi INFO: [multi] job_id 16964107 2024-03-09 14:50:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 14:50:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964107 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964107 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16964107 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-09 14:50:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-09 14:50:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 14:50:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2005 try 0 launched on executor multi with executor id 2006 parsl.dataflow.dflow INFO: Standard output for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f238862c-26ae-4a34-93e5-9c4d5aca61a5_isrForCrosstalkSources_855412_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f238862c-26ae-4a34-93e5-9c4d5aca61a5_isrForCrosstalkSources_855412_51.stderr parsl.dataflow.dflow DEBUG: Task 2006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2006 has memoization hash e4d90f3a86c74a4361f2bb272fbbc328 parsl.dataflow.memoization INFO: Task 2006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2006 try 0 launched on executor multi with executor id 2007 parsl.dataflow.dflow INFO: Standard output for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e751dbcf-9443-493b-b53e-caf469446971_isrForCrosstalkSources_855421_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e751dbcf-9443-493b-b53e-caf469446971_isrForCrosstalkSources_855421_31.stderr parsl.dataflow.dflow DEBUG: Task 2007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2007 has memoization hash c7c73f02c2c16e9e73ee535fa451e646 parsl.dataflow.memoization INFO: Task 2007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2007 try 0 launched on executor multi with executor id 2008 parsl.dataflow.dflow INFO: Standard output for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7e19c8a0-402c-419f-8265-b1c05020a675_isrForCrosstalkSources_855381_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7e19c8a0-402c-419f-8265-b1c05020a675_isrForCrosstalkSources_855381_60.stderr parsl.dataflow.dflow DEBUG: Task 2008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2008 has memoization hash 03ea40607eafcbd959b94a6072978f72 parsl.dataflow.memoization INFO: Task 2008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6807600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2008 try 0 launched on executor multi with executor id 2009 parsl.dataflow.dflow INFO: Standard output for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ea1a670-6a2e-45bc-bcfe-bf3f016705a7_isrForCrosstalkSources_855429_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ea1a670-6a2e-45bc-bcfe-bf3f016705a7_isrForCrosstalkSources_855429_18.stderr parsl.dataflow.dflow DEBUG: Task 2009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2009 has memoization hash 4be3fc914e136c39a76c71aa55deeca8 parsl.dataflow.memoization INFO: Task 2009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2009 try 0 launched on executor multi with executor id 2010 parsl.dataflow.dflow INFO: Standard output for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9a0614b-e370-4588-a200-ceee279c072e_isrForCrosstalkSources_855421_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9a0614b-e370-4588-a200-ceee279c072e_isrForCrosstalkSources_855421_4.stderr parsl.dataflow.dflow DEBUG: Task 2010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2010 has memoization hash d93ab9107e27425bcae0c28282820290 parsl.dataflow.memoization INFO: Task 2010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2010 try 0 launched on executor multi with executor id 2011 parsl.dataflow.dflow INFO: Standard output for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3dfe20a0-263a-4be8-96d2-8d9c665c544b_isrForCrosstalkSources_855416_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3dfe20a0-263a-4be8-96d2-8d9c665c544b_isrForCrosstalkSources_855416_42.stderr parsl.dataflow.dflow DEBUG: Task 2011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2011 has memoization hash 85824f6aa7d1e7e9fa5359b1eaa272db parsl.dataflow.memoization INFO: Task 2011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2011 try 0 launched on executor multi with executor id 2012 parsl.dataflow.dflow INFO: Standard output for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/a1f6b7ac-4d22-46c5-b4a3-82267bd67d52_isrForCrosstalkSources_855392_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/a1f6b7ac-4d22-46c5-b4a3-82267bd67d52_isrForCrosstalkSources_855392_16.stderr parsl.dataflow.dflow DEBUG: Task 2012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2012 has memoization hash 63cb3ce130aed94d4d4c92c7fb4aef2a parsl.dataflow.memoization INFO: Task 2012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2012 try 0 launched on executor multi with executor id 2013 parsl.dataflow.dflow INFO: Standard output for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a879fe82-2669-42c9-bf10-e470c1909d33_isrForCrosstalkSources_855455_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a879fe82-2669-42c9-bf10-e470c1909d33_isrForCrosstalkSources_855455_28.stderr parsl.dataflow.dflow DEBUG: Task 2013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2013 has memoization hash 610e7f214bd9dc4a9595004f64785e27 parsl.dataflow.memoization INFO: Task 2013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2013 try 0 launched on executor multi with executor id 2014 parsl.dataflow.dflow INFO: Standard output for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/637cde83-aac4-45ee-bc6a-f57139fa0421_isrForCrosstalkSources_855380_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/637cde83-aac4-45ee-bc6a-f57139fa0421_isrForCrosstalkSources_855380_46.stderr parsl.dataflow.dflow DEBUG: Task 2014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2014 has memoization hash 8e8a7fc60a8b660981cda5cc13f57828 parsl.dataflow.memoization INFO: Task 2014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2014 try 0 launched on executor multi with executor id 2015 parsl.dataflow.dflow INFO: Standard output for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/27821f74-9d97-4af6-80c8-b86ebad0dbf3_isrForCrosstalkSources_855386_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/27821f74-9d97-4af6-80c8-b86ebad0dbf3_isrForCrosstalkSources_855386_43.stderr parsl.dataflow.dflow DEBUG: Task 2015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2015 has memoization hash 428deed87a56db920b5573310a66085a parsl.dataflow.memoization INFO: Task 2015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2015 try 0 launched on executor multi with executor id 2016 parsl.dataflow.dflow INFO: Standard output for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1efc6253-115f-4e72-800c-983c0327c020_isrForCrosstalkSources_855447_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1efc6253-115f-4e72-800c-983c0327c020_isrForCrosstalkSources_855447_44.stderr parsl.dataflow.dflow DEBUG: Task 2016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2016 has memoization hash afcade48dc852bb89902614510e22290 parsl.dataflow.memoization INFO: Task 2016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2016 try 0 launched on executor multi with executor id 2017 parsl.dataflow.dflow INFO: Standard output for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3865894-a9a7-4f20-9d80-07ea690abaa6_isrForCrosstalkSources_855379_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3865894-a9a7-4f20-9d80-07ea690abaa6_isrForCrosstalkSources_855379_3.stderr parsl.dataflow.dflow DEBUG: Task 2017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2017 has memoization hash 81162bf8c741810602906b4bc2144cfd parsl.dataflow.memoization INFO: Task 2017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2017 try 0 launched on executor multi with executor id 2018 parsl.dataflow.dflow INFO: Standard output for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d231f72c-75a7-4acc-b85c-3a6e3bc5dd3d_isrForCrosstalkSources_855428_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d231f72c-75a7-4acc-b85c-3a6e3bc5dd3d_isrForCrosstalkSources_855428_23.stderr parsl.dataflow.dflow DEBUG: Task 2018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2018 has memoization hash 2e15649753590282583ae83384f2caeb parsl.dataflow.memoization INFO: Task 2018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2018 try 0 launched on executor multi with executor id 2019 parsl.dataflow.dflow INFO: Standard output for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/dd81a42b-9b0f-4f84-b467-a01ee476377b_isrForCrosstalkSources_855448_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/dd81a42b-9b0f-4f84-b467-a01ee476377b_isrForCrosstalkSources_855448_36.stderr parsl.dataflow.dflow DEBUG: Task 2019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2019 has memoization hash fb9d93ad4be2799fe1086073bf2b91ca parsl.dataflow.memoization INFO: Task 2019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2019 try 0 launched on executor multi with executor id 2020 parsl.dataflow.dflow INFO: Standard output for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f7b665dc-08ca-4aac-98e9-65f1546dde36_isrForCrosstalkSources_855438_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f7b665dc-08ca-4aac-98e9-65f1546dde36_isrForCrosstalkSources_855438_54.stderr parsl.dataflow.dflow DEBUG: Task 2020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2020 has memoization hash 975441cf9fc110ad26afc2c8d07286d8 parsl.dataflow.memoization INFO: Task 2020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2020 try 0 launched on executor multi with executor id 2021 parsl.dataflow.dflow INFO: Standard output for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d38d1b23-3fb4-460d-8797-6ec7e2170343_isrForCrosstalkSources_855442_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d38d1b23-3fb4-460d-8797-6ec7e2170343_isrForCrosstalkSources_855442_58.stderr parsl.dataflow.dflow DEBUG: Task 2021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2021 has memoization hash 0b8d584de3b9044c95693f5fe8d77276 parsl.dataflow.memoization INFO: Task 2021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2021 try 0 launched on executor multi with executor id 2022 parsl.dataflow.dflow INFO: Standard output for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3fca9484-443c-488d-a9bc-b69bc4c26f68_isrForCrosstalkSources_855408_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3fca9484-443c-488d-a9bc-b69bc4c26f68_isrForCrosstalkSources_855408_24.stderr parsl.dataflow.dflow DEBUG: Task 2022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2022 has memoization hash 9d109bb031051d07ea74b0dcd6d6bb62 parsl.dataflow.memoization INFO: Task 2022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2022 try 0 launched on executor multi with executor id 2023 parsl.dataflow.dflow INFO: Standard output for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4efe991c-5ed3-45a8-bdcc-b11cea328692_isrForCrosstalkSources_855383_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4efe991c-5ed3-45a8-bdcc-b11cea328692_isrForCrosstalkSources_855383_22.stderr parsl.dataflow.dflow DEBUG: Task 2023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2023 has memoization hash 5fbd21acadfae3f2b3f82a5990dd123c parsl.dataflow.memoization INFO: Task 2023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2023 try 0 launched on executor multi with executor id 2024 parsl.dataflow.dflow INFO: Standard output for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e3b53adc-d253-4ac9-98f1-0af128e8dd66_isrForCrosstalkSources_855416_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e3b53adc-d253-4ac9-98f1-0af128e8dd66_isrForCrosstalkSources_855416_21.stderr parsl.dataflow.dflow DEBUG: Task 2024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2024 has memoization hash 2c08823281e9b6803aa0496d7fa31f83 parsl.dataflow.memoization INFO: Task 2024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2024 try 0 launched on executor multi with executor id 2025 parsl.dataflow.dflow INFO: Standard output for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/28b4293e-de17-441e-aaac-d6fb9b6b1300_isrForCrosstalkSources_855442_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/28b4293e-de17-441e-aaac-d6fb9b6b1300_isrForCrosstalkSources_855442_47.stderr parsl.dataflow.dflow DEBUG: Task 2025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2025 has memoization hash a765c033999da806e24a1905ef507ac6 parsl.dataflow.memoization INFO: Task 2025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2025 try 0 launched on executor multi with executor id 2026 parsl.dataflow.dflow INFO: Standard output for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a74c2d4-bc4c-49e1-9f77-07ef58a9e4ef_isrForCrosstalkSources_855447_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a74c2d4-bc4c-49e1-9f77-07ef58a9e4ef_isrForCrosstalkSources_855447_5.stderr parsl.dataflow.dflow DEBUG: Task 2026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2026 has memoization hash dbdaa91fc807c6e97d04e6f43a8878da parsl.dataflow.memoization INFO: Task 2026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2026 try 0 launched on executor multi with executor id 2027 parsl.dataflow.dflow INFO: Standard output for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/b1412fa1-6820-468b-a712-76e8370a7f81_isrForCrosstalkSources_855429_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/b1412fa1-6820-468b-a712-76e8370a7f81_isrForCrosstalkSources_855429_51.stderr parsl.dataflow.dflow DEBUG: Task 2027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2027 has memoization hash 98f186b0e0b50126e316bc93227f0ce4 parsl.dataflow.memoization INFO: Task 2027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2027 try 0 launched on executor multi with executor id 2028 parsl.dataflow.dflow INFO: Standard output for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b4585347-7155-4f0c-87f6-b0095c553b81_isrForCrosstalkSources_855458_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b4585347-7155-4f0c-87f6-b0095c553b81_isrForCrosstalkSources_855458_6.stderr parsl.dataflow.dflow DEBUG: Task 2028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2028 has memoization hash 3aec05c4091a6e402df166092917a1a0 parsl.dataflow.memoization INFO: Task 2028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2028 try 0 launched on executor multi with executor id 2029 parsl.dataflow.dflow INFO: Standard output for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/59537ff1-7b88-4e0f-b4d6-d0cfaf174381_isrForCrosstalkSources_855389_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/59537ff1-7b88-4e0f-b4d6-d0cfaf174381_isrForCrosstalkSources_855389_32.stderr parsl.dataflow.dflow DEBUG: Task 2029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2029 has memoization hash dab7ea5ed423ec6d36a97ecb6bdcbfcd parsl.dataflow.memoization INFO: Task 2029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2029 try 0 launched on executor multi with executor id 2030 parsl.dataflow.dflow INFO: Standard output for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/0ea58fd6-b16b-4cde-aeca-440f315a4d20_isrForCrosstalkSources_855397_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/0ea58fd6-b16b-4cde-aeca-440f315a4d20_isrForCrosstalkSources_855397_2.stderr parsl.dataflow.dflow DEBUG: Task 2030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2030 has memoization hash ec418bec519a6a34071ba1d2f26dd6dd parsl.dataflow.memoization INFO: Task 2030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2030 try 0 launched on executor multi with executor id 2031 parsl.dataflow.dflow INFO: Standard output for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/9c66eb67-7d1e-4b4d-abe7-796a3119d987_isrForCrosstalkSources_855416_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/9c66eb67-7d1e-4b4d-abe7-796a3119d987_isrForCrosstalkSources_855416_14.stderr parsl.dataflow.dflow DEBUG: Task 2031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2031 has memoization hash 8d4521763c45bc16f34cf2d88b9204ef parsl.dataflow.memoization INFO: Task 2031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2031 try 0 launched on executor multi with executor id 2032 parsl.dataflow.dflow INFO: Standard output for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/815a8a35-cfe3-42e6-a071-c63693401cd0_isrForCrosstalkSources_855394_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/815a8a35-cfe3-42e6-a071-c63693401cd0_isrForCrosstalkSources_855394_20.stderr parsl.dataflow.dflow DEBUG: Task 2032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2032 has memoization hash 99dcd577d353af5935bd53b7b621d447 parsl.dataflow.memoization INFO: Task 2032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:48 proc_lsst.multi:170 [INFO] [multi] job_id 16964108 proc_lsst.multi INFO: [multi] job_id 16964108 2024-03-09 14:50:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 14:50:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964108 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964108 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16964108 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-09 14:50:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2032 try 0 launched on executor multi with executor id 2033 parsl.dataflow.dflow INFO: Standard output for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78ef59ed-dbad-42a2-82ea-b29dba0a1efa_isrForCrosstalkSources_855387_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78ef59ed-dbad-42a2-82ea-b29dba0a1efa_isrForCrosstalkSources_855387_3.stderr parsl.dataflow.dflow DEBUG: Task 2033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2033 has memoization hash 06de22bfed01ced7615218817ac62635 parsl.dataflow.memoization INFO: Task 2033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2033 try 0 launched on executor multi with executor id 2034 parsl.dataflow.dflow INFO: Standard output for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ae1c6241-b1d5-4c32-b456-85acd61d526c_isrForCrosstalkSources_855398_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ae1c6241-b1d5-4c32-b456-85acd61d526c_isrForCrosstalkSources_855398_28.stderr parsl.dataflow.dflow DEBUG: Task 2034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2034 has memoization hash d853fa5b06b0062212c3e95acf653b3f parsl.dataflow.memoization INFO: Task 2034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2034 try 0 launched on executor multi with executor id 2035 parsl.dataflow.dflow INFO: Standard output for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9f9e479-ebf5-424c-9998-943d98a921a0_isrForCrosstalkSources_855421_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9f9e479-ebf5-424c-9998-943d98a921a0_isrForCrosstalkSources_855421_13.stderr parsl.dataflow.dflow DEBUG: Task 2035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2035 has memoization hash 9635fe8a5eb7400ff682f664e1992405 parsl.dataflow.memoization INFO: Task 2035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b671f59e43654371eb78ce40983607fb with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3f1534a1-b5d9-4315-9567-b16ebfade7e1_isrForCrosstalkSources_855454_43.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3f1534a1-b5d9-4315-9567-b16ebfade7e1_isrForCrosstalkSources_855454_43.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 237014cbbc13ee69866655ed3aaa3def with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bb182782-bb17-4a0f-8077-1ae22fd4b004_isrForCrosstalkSources_855412_38.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bb182782-bb17-4a0f-8077-1ae22fd4b004_isrForCrosstalkSources_855412_38.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d97668dbfac40b0802e9bc2ddd51902c with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e1afd0c2-8ab6-437b-82d7-7b1ef48be40a_isrForCrosstalkSources_855423_39.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e1afd0c2-8ab6-437b-82d7-7b1ef48be40a_isrForCrosstalkSources_855423_39.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a363909203120fd680aaea16780b77b with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c36c11e5-9e6c-4b04-8ba4-cf94262ba824_isrForCrosstalkSources_855435_7.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c36c11e5-9e6c-4b04-8ba4-cf94262ba824_isrForCrosstalkSources_855435_7.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07c1b6c6164878cb8c890b745f37eb0c with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/51dbf996-c0ef-4405-9892-e7bb4dff6053_isrForCrosstalkSources_855402_16.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/51dbf996-c0ef-4405-9892-e7bb4dff6053_isrForCrosstalkSources_855402_16.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ac2abdc54376b82c28d2837b41e592f with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec36a123-bad2-42ec-81f8-7ed94362e912_isrForCrosstalkSources_855391_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec36a123-bad2-42ec-81f8-7ed94362e912_isrForCrosstalkSources_855391_2.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a619fa75e14eba3371f32b6358b829e8 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/08c15e73-0ab4-4038-a094-1a47f9c2fe81_isrForCrosstalkSources_855422_7.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/08c15e73-0ab4-4038-a094-1a47f9c2fe81_isrForCrosstalkSources_855422_7.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbff304fe8b9a0e83ef6b13462759d01 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/edceabb3-7ed1-42d7-9c60-c854ce19697f_isrForCrosstalkSources_855410_3.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/edceabb3-7ed1-42d7-9c60-c854ce19697f_isrForCrosstalkSources_855410_3.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29ca7a996b9242ca171f19f4db28042e with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0595f7d5-6cf7-4329-b490-de818f0e6743_isrForCrosstalkSources_855385_45.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0595f7d5-6cf7-4329-b490-de818f0e6743_isrForCrosstalkSources_855385_45.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b8a736e8e3bfa2947557c599212857f with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1919a9dc-5e4a-44e9-b39e-c67fa67658c9_isrForCrosstalkSources_855419_40.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1919a9dc-5e4a-44e9-b39e-c67fa67658c9_isrForCrosstalkSources_855419_40.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e954c7d7c1f32e4314d01139a7e43bb0 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9c626707-fa5b-4831-adc6-1711699c3e9d_isrForCrosstalkSources_855397_51.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9c626707-fa5b-4831-adc6-1711699c3e9d_isrForCrosstalkSources_855397_51.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12d6b1ea8af0d065a1b54c0addd0712a with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/94fc10d9-9397-4d33-90dc-77f9b839975a_isrForCrosstalkSources_855434_54.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/94fc10d9-9397-4d33-90dc-77f9b839975a_isrForCrosstalkSources_855434_54.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3a268b34276d8aaf369b59f5c3b3ed0 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d090e80f-fc19-4680-bd1e-73fdbc2b4be0_isrForCrosstalkSources_855383_44.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d090e80f-fc19-4680-bd1e-73fdbc2b4be0_isrForCrosstalkSources_855383_44.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8c110d3963cc2b894c8cecd2af7cd59 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/879884ae-8d84-4871-8528-cd8ca1babd05_isrForCrosstalkSources_855430_47.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/879884ae-8d84-4871-8528-cd8ca1babd05_isrForCrosstalkSources_855430_47.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b682dce7ac1f56bbba0281a9a6ffff with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/96dbfa9e-0797-4897-bf95-90d45c783bb4_isrForCrosstalkSources_855443_24.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/96dbfa9e-0797-4897-bf95-90d45c783bb4_isrForCrosstalkSources_855443_24.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe2fc251e67a4c1872303622a2b550b with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e1133b7d-3099-40e3-8428-9bfac04ef96a_isrForCrosstalkSources_855441_1.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e1133b7d-3099-40e3-8428-9bfac04ef96a_isrForCrosstalkSources_855441_1.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e010b781bfdbe137c035fe8fbb1a23f6 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e0a56c3a-b340-4d97-b6e4-1f764cfb1362_isrForCrosstalkSources_855411_56.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e0a56c3a-b340-4d97-b6e4-1f764cfb1362_isrForCrosstalkSources_855411_56.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58ee2f46733c51b3e4017dc24cae0a0e with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81b82c2d-eb84-43d1-80e9-594431a9e1ce_isrForCrosstalkSources_855398_22.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81b82c2d-eb84-43d1-80e9-594431a9e1ce_isrForCrosstalkSources_855398_22.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 828f3c0d5d891b2cc91724ff364d6d4a with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2b7c3f03-16c4-4095-b008-99548cabfb66_isrForCrosstalkSources_855384_22.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2b7c3f03-16c4-4095-b008-99548cabfb66_isrForCrosstalkSources_855384_22.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6da97dcb2c71c9418f8530362d72a0f with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/eedc41b2-f0b4-47dd-a978-0db5d08ec0f5_isrForCrosstalkSources_855433_30.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/eedc41b2-f0b4-47dd-a978-0db5d08ec0f5_isrForCrosstalkSources_855433_30.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06dc7b49a309a692358364024e552dec with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5f608c1d-46df-4c43-8926-f85071c81e32_isrForCrosstalkSources_855381_43.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5f608c1d-46df-4c43-8926-f85071c81e32_isrForCrosstalkSources_855381_43.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e78e00c796502efe34f04a4d6442b96f with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/74415227-741f-456c-b042-6f029a0cd4ef_isrForCrosstalkSources_855395_57.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/74415227-741f-456c-b042-6f029a0cd4ef_isrForCrosstalkSources_855395_57.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79430137afdfa2218b77f3c591babd8f with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/bbf8b392-9510-4b8b-b20e-40282fb0a452_isrForCrosstalkSources_855415_26.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/bbf8b392-9510-4b8b-b20e-40282fb0a452_isrForCrosstalkSources_855415_26.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ea10079c88793914b09285ac546bbd with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/711ffe4c-604e-448c-aec3-4bb98b825c98_isrForCrosstalkSources_855382_17.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/711ffe4c-604e-448c-aec3-4bb98b825c98_isrForCrosstalkSources_855382_17.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c8bdf22e83efa7c6225fdaea4e7b60 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/3acda221-f75d-4abb-95c4-3ed31777aa7d_isrForCrosstalkSources_855406_38.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/3acda221-f75d-4abb-95c4-3ed31777aa7d_isrForCrosstalkSources_855406_38.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aad47b592411aeb101ffac4274c6e8c1 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c7366544-d8e4-400f-bdd8-51d2e1737f90_isrForCrosstalkSources_855404_13.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c7366544-d8e4-400f-bdd8-51d2e1737f90_isrForCrosstalkSources_855404_13.stderr parsl.dataflow.dflow INFO: Parsl task 2035 try 0 launched on executor multi with executor id 2036 parsl.dataflow.dflow INFO: Standard output for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df5c5641-200d-46aa-917a-f4fd47c77ea7_isrForCrosstalkSources_855422_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df5c5641-200d-46aa-917a-f4fd47c77ea7_isrForCrosstalkSources_855422_54.stderr parsl.dataflow.dflow DEBUG: Task 2036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2036 has memoization hash aa5980be11b386b812ba8b17c6ddc18f parsl.dataflow.memoization INFO: Task 2036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2036 try 0 launched on executor multi with executor id 2037 parsl.dataflow.dflow INFO: Standard output for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6ddef829-2254-4ef4-b6b6-6373d1e43c72_isrForCrosstalkSources_855443_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6ddef829-2254-4ef4-b6b6-6373d1e43c72_isrForCrosstalkSources_855443_18.stderr parsl.dataflow.dflow DEBUG: Task 2037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2037 has memoization hash 72a07e3e34c02c7a8f2715c028407b18 parsl.dataflow.memoization INFO: Task 2037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2037 try 0 launched on executor multi with executor id 2038 parsl.dataflow.dflow INFO: Standard output for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/87e64576-c024-491a-89ef-4737e6b2eb3f_isrForCrosstalkSources_855434_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/87e64576-c024-491a-89ef-4737e6b2eb3f_isrForCrosstalkSources_855434_30.stderr parsl.dataflow.dflow DEBUG: Task 2038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2038 has memoization hash 4b4610a26e54e6992fa1334a0985e53b parsl.dataflow.memoization INFO: Task 2038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2038 try 0 launched on executor multi with executor id 2039 parsl.dataflow.dflow INFO: Standard output for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b870f740-302b-4c57-a489-eb174b55514a_isrForCrosstalkSources_855408_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b870f740-302b-4c57-a489-eb174b55514a_isrForCrosstalkSources_855408_34.stderr parsl.dataflow.dflow DEBUG: Task 2039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2039 has memoization hash 792bb065d3b30cf27c572f993ba9478d parsl.dataflow.memoization INFO: Task 2039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2039 try 0 launched on executor multi with executor id 2040 parsl.dataflow.dflow INFO: Standard output for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9854d422-a423-4b0f-bdba-5d8eac545b96_isrForCrosstalkSources_855448_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9854d422-a423-4b0f-bdba-5d8eac545b96_isrForCrosstalkSources_855448_18.stderr parsl.dataflow.dflow DEBUG: Task 2040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2040 has memoization hash 650019115c10617615b765d1641f3605 parsl.dataflow.memoization INFO: Task 2040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2040 try 0 launched on executor multi with executor id 2041 parsl.dataflow.dflow INFO: Standard output for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/e4d947eb-7fe2-4fff-aec8-4115f0fc3405_isrForCrosstalkSources_855381_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/e4d947eb-7fe2-4fff-aec8-4115f0fc3405_isrForCrosstalkSources_855381_19.stderr parsl.dataflow.dflow DEBUG: Task 2041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2041 has memoization hash 96eac7e80ae010b269a9ed3acb34bdd7 parsl.dataflow.memoization INFO: Task 2041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2041 try 0 launched on executor multi with executor id 2042 parsl.dataflow.dflow INFO: Standard output for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06f8acf7-155e-49c6-8a9d-dfc09b52e608_isrForCrosstalkSources_855454_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06f8acf7-155e-49c6-8a9d-dfc09b52e608_isrForCrosstalkSources_855454_59.stderr parsl.dataflow.dflow DEBUG: Task 2042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2042 has memoization hash ddc13f700ee53188c55e62ff759b39aa parsl.dataflow.memoization INFO: Task 2042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2042 try 0 launched on executor multi with executor id 2043 parsl.dataflow.dflow INFO: Standard output for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f50f3fbb-7157-461f-b43b-a37ae2128306_isrForCrosstalkSources_855409_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f50f3fbb-7157-461f-b43b-a37ae2128306_isrForCrosstalkSources_855409_48.stderr parsl.dataflow.dflow DEBUG: Task 2043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2043 has memoization hash 6c4e60f667ebc02ff2887e91a4f040f5 parsl.dataflow.memoization INFO: Task 2043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2043 try 0 launched on executor multi with executor id 2044 parsl.dataflow.dflow INFO: Standard output for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/fa7902bf-f75d-4df7-a71f-f5693b424445_isrForCrosstalkSources_855436_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/fa7902bf-f75d-4df7-a71f-f5693b424445_isrForCrosstalkSources_855436_39.stderr parsl.dataflow.dflow DEBUG: Task 2044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2044 has memoization hash d98014ed561155266f91b95e56983866 parsl.dataflow.memoization INFO: Task 2044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2044 try 0 launched on executor multi with executor id 2045 parsl.dataflow.dflow INFO: Standard output for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/bc638559-8c7e-45d0-ba24-da0c63588592_isrForCrosstalkSources_855393_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/bc638559-8c7e-45d0-ba24-da0c63588592_isrForCrosstalkSources_855393_45.stderr parsl.dataflow.dflow DEBUG: Task 2045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2045 has memoization hash d6d673ff54c1a43f0fd083f0ee32f67b parsl.dataflow.memoization INFO: Task 2045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2045 try 0 launched on executor multi with executor id 2046 parsl.dataflow.dflow INFO: Standard output for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/143594e5-c3f8-41fc-9796-034f04d7245f_isrForCrosstalkSources_855436_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/143594e5-c3f8-41fc-9796-034f04d7245f_isrForCrosstalkSources_855436_61.stderr parsl.dataflow.dflow DEBUG: Task 2046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies 2024-03-09 14:50:48 proc_lsst.multi:170 [INFO] [multi] job_id 16964109 proc_lsst.multi INFO: [multi] job_id 16964109 2024-03-09 14:50:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 14:50:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964109 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964109 parsl.dataflow.dflow INFO: Task 2046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2046 has memoization hash ce58cb2e7a9bfffdd17d21ee2969dcce parsl.dataflow.memoization INFO: Task 2046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16964109 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-09 14:50:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2046 try 0 launched on executor multi with executor id 2047 parsl.dataflow.dflow INFO: Standard output for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/acfa45a7-df25-4fe1-b448-a5a6d6633526_isrForCrosstalkSources_855391_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/acfa45a7-df25-4fe1-b448-a5a6d6633526_isrForCrosstalkSources_855391_55.stderr parsl.dataflow.dflow DEBUG: Task 2047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2047 has memoization hash e24de92c4ad6d9d1c5e6d19af10d88d6 parsl.dataflow.memoization INFO: Task 2047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2047 try 0 launched on executor multi with executor id 2048 parsl.dataflow.dflow INFO: Standard output for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/86344761-d67c-4c1d-b347-c1e371fc5352_isrForCrosstalkSources_855459_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/86344761-d67c-4c1d-b347-c1e371fc5352_isrForCrosstalkSources_855459_45.stderr parsl.dataflow.dflow DEBUG: Task 2048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2048 has memoization hash 5f4640c8d289cf4d303e38aa0ad7ffe1 parsl.dataflow.memoization INFO: Task 2048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6676e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2048 try 0 launched on executor multi with executor id 2049 parsl.dataflow.dflow INFO: Standard output for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9cc6f4e4-59fd-406b-af78-1e0f678db0dd_isrForCrosstalkSources_855408_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9cc6f4e4-59fd-406b-af78-1e0f678db0dd_isrForCrosstalkSources_855408_11.stderr parsl.dataflow.dflow DEBUG: Task 2049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2049 has memoization hash 0acc5b3639e7921bfee313e2b028d88c parsl.dataflow.memoization INFO: Task 2049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2049 try 0 launched on executor multi with executor id 2050 parsl.dataflow.dflow INFO: Standard output for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/924fafe3-7379-43f5-9ca0-c389247014a5_isrForCrosstalkSources_855398_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/924fafe3-7379-43f5-9ca0-c389247014a5_isrForCrosstalkSources_855398_53.stderr parsl.dataflow.dflow DEBUG: Task 2050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2050 has memoization hash 5820bdd953c02bf4bafbd08eeda0d789 parsl.dataflow.memoization INFO: Task 2050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2050 try 0 launched on executor multi with executor id 2051 parsl.dataflow.dflow INFO: Standard output for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f1eea7b1-2ff9-4239-9a7b-ed3d11932273_isrForCrosstalkSources_855444_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f1eea7b1-2ff9-4239-9a7b-ed3d11932273_isrForCrosstalkSources_855444_47.stderr parsl.dataflow.dflow DEBUG: Task 2051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2051 has memoization hash b866fccce1bc8e927094baeaee8c2295 parsl.dataflow.memoization INFO: Task 2051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2051 try 0 launched on executor multi with executor id 2052 parsl.dataflow.dflow INFO: Standard output for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1176dc03-5ab3-4bc3-a18c-e22fc0a95661_isrForCrosstalkSources_855450_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1176dc03-5ab3-4bc3-a18c-e22fc0a95661_isrForCrosstalkSources_855450_22.stderr parsl.dataflow.dflow DEBUG: Task 2052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Task 2052 has memoization hash 64fd292af5305885ca3554650a9549ea parsl.dataflow.memoization INFO: Task 2052 had no result in cache parsl.dataflow.memoization DEBUG: Storing app cache entry 007729bd26d58ff047a432ccb6ae358d with result from task 28 parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c3e45e64-c049-4821-bab6-8138e817bf71_isrForCrosstalkSources_855415_62.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c3e45e64-c049-4821-bab6-8138e817bf71_isrForCrosstalkSources_855415_62.stderr parsl.dataflow.dflow INFO: Parsl task 2052 try 0 launched on executor multi with executor id 2053 parsl.dataflow.dflow INFO: Standard output for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4ecc0614-68fa-4922-b095-63ad18be6b9a_isrForCrosstalkSources_855412_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4ecc0614-68fa-4922-b095-63ad18be6b9a_isrForCrosstalkSources_855412_46.stderr parsl.dataflow.dflow DEBUG: Task 2053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2053 has memoization hash f5468a3889783197f9d20146d18d1e49 parsl.dataflow.memoization INFO: Task 2053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2053 try 0 launched on executor multi with executor id 2054 parsl.dataflow.dflow INFO: Standard output for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b9fbccd2-c4d5-4be8-a5df-ace971615556_isrForCrosstalkSources_855438_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b9fbccd2-c4d5-4be8-a5df-ace971615556_isrForCrosstalkSources_855438_20.stderr parsl.dataflow.dflow DEBUG: Task 2054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2054 has memoization hash 8fdc20e92cc9a9c31c70487ab1ba9e07 parsl.dataflow.memoization INFO: Task 2054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2054 try 0 launched on executor multi with executor id 2055 parsl.dataflow.dflow INFO: Standard output for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7a22be39-9b64-4bab-b717-d47e0365d750_isrForCrosstalkSources_855453_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7a22be39-9b64-4bab-b717-d47e0365d750_isrForCrosstalkSources_855453_62.stderr parsl.dataflow.dflow DEBUG: Task 2055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2055 has memoization hash d5836349961fcddb350ace374bd83915 parsl.dataflow.memoization INFO: Task 2055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2055 try 0 launched on executor multi with executor id 2056 parsl.dataflow.dflow INFO: Standard output for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b77ef49a-d761-439f-9644-6e797e5197ba_isrForCrosstalkSources_855440_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b77ef49a-d761-439f-9644-6e797e5197ba_isrForCrosstalkSources_855440_49.stderr parsl.dataflow.dflow DEBUG: Task 2056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2056 has memoization hash 99ae4887f68d13d0884578e5f8a3f643 parsl.dataflow.memoization INFO: Task 2056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:48 proc_lsst.multi:170 [INFO] [multi] job_id 16964110 proc_lsst.multi INFO: [multi] job_id 16964110 2024-03-09 14:50:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 14:50:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964110 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964110 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16964110 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-09 14:50:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 14:50:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2056 try 0 launched on executor multi with executor id 2057 parsl.dataflow.dflow INFO: Standard output for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/05a93d35-7f63-447b-b2c9-628fa66a1105_isrForCrosstalkSources_855437_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/05a93d35-7f63-447b-b2c9-628fa66a1105_isrForCrosstalkSources_855437_58.stderr parsl.dataflow.dflow DEBUG: Task 2057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2057 has memoization hash 3a63816c6ba6a7b198cbae45b914c843 parsl.dataflow.memoization INFO: Task 2057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a68077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:49 proc_lsst.multi:170 [INFO] [multi] job_id 16964111 proc_lsst.multi INFO: [multi] job_id 16964111 2024-03-09 14:50:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964111 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964111 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16964111 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-09 14:50:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:50:49 proc_lsst.multi:170 [INFO] [multi] job_id 16964112 proc_lsst.multi INFO: [multi] job_id 16964112 2024-03-09 14:50:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-09 14:50:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964112 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964112 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16964112 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-09 14:50:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-09 14:50:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02dd53f80392c0b742e7434e4bfe3926 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b89bcc1c-853a-4a69-b9bd-adb530f12e8a_isrForCrosstalkSources_855419_23.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b89bcc1c-853a-4a69-b9bd-adb530f12e8a_isrForCrosstalkSources_855419_23.stderr parsl.dataflow.dflow INFO: Parsl task 2057 try 0 launched on executor multi with executor id 2058 parsl.dataflow.dflow INFO: Standard output for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/1b8f074b-9f17-4d13-a46c-039b83c64c44_isrForCrosstalkSources_855410_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/1b8f074b-9f17-4d13-a46c-039b83c64c44_isrForCrosstalkSources_855410_10.stderr parsl.dataflow.dflow DEBUG: Task 2058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2058 has memoization hash 435c153b64416b938ec396a07fb84e48 parsl.dataflow.memoization INFO: Task 2058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a99b7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2058 try 0 launched on executor multi with executor id 2059 parsl.dataflow.dflow INFO: Standard output for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/def14c90-f076-4901-870b-1198836beca6_isrForCrosstalkSources_855457_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/def14c90-f076-4901-870b-1198836beca6_isrForCrosstalkSources_855457_8.stderr parsl.dataflow.dflow DEBUG: Task 2059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2059 has memoization hash 974f28ee28da6b7c96bf53a180353550 parsl.dataflow.memoization INFO: Task 2059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2059 try 0 launched on executor multi with executor id 2060 parsl.dataflow.dflow INFO: Standard output for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/51a61add-6fa8-4af3-a98f-5dc5db8e5d3c_isrForCrosstalkSources_855410_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/51a61add-6fa8-4af3-a98f-5dc5db8e5d3c_isrForCrosstalkSources_855410_62.stderr parsl.dataflow.dflow DEBUG: Task 2060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2060 has memoization hash 95a51e9b04f234fa4a5433027a9dc15e parsl.dataflow.memoization INFO: Task 2060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2060 try 0 launched on executor multi with executor id 2061 parsl.dataflow.dflow INFO: Standard output for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/8a5c0463-567c-49e5-8338-15fc6b93e5a0_isrForCrosstalkSources_855403_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/8a5c0463-567c-49e5-8338-15fc6b93e5a0_isrForCrosstalkSources_855403_5.stderr parsl.dataflow.dflow DEBUG: Task 2061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2061 has memoization hash a850c83d5ac4601f2da4e133bd359f83 parsl.dataflow.memoization INFO: Task 2061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2061 try 0 launched on executor multi with executor id 2062 parsl.dataflow.dflow INFO: Standard output for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/075ecc01-19a6-423b-b39c-da83ae682311_isrForCrosstalkSources_855455_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/075ecc01-19a6-423b-b39c-da83ae682311_isrForCrosstalkSources_855455_41.stderr parsl.dataflow.dflow DEBUG: Task 2062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2062 has memoization hash 2b0f162b2cfe4b622d6bcf2b9d6fb252 parsl.dataflow.memoization INFO: Task 2062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2062 try 0 launched on executor multi with executor id 2063 parsl.dataflow.dflow INFO: Standard output for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/bf548217-67d9-41e7-8cd0-28d5b454a586_isrForCrosstalkSources_855411_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/bf548217-67d9-41e7-8cd0-28d5b454a586_isrForCrosstalkSources_855411_41.stderr parsl.dataflow.dflow DEBUG: Task 2063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2063 has memoization hash 8146886aa77092799d35e69f1eaa0f5f parsl.dataflow.memoization INFO: Task 2063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:51 proc_lsst.multi:170 [INFO] [multi] job_id 16964113 proc_lsst.multi INFO: [multi] job_id 16964113 2024-03-09 14:50:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-09 14:50:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964113 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964113 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16964113 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-09 14:50:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-09 14:50:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-09 14:50:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36414fa91943bb2aed3ecf6472dee93e with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3e2556c6-e1c4-4d4f-bf73-8dfd0521177a_isrForCrosstalkSources_855382_15.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3e2556c6-e1c4-4d4f-bf73-8dfd0521177a_isrForCrosstalkSources_855382_15.stderr parsl.dataflow.dflow INFO: Parsl task 2063 try 0 launched on executor multi with executor id 2064 parsl.dataflow.dflow INFO: Standard output for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2b3e1556-1818-422a-87a8-1579a5172393_isrForCrosstalkSources_855449_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2b3e1556-1818-422a-87a8-1579a5172393_isrForCrosstalkSources_855449_22.stderr parsl.dataflow.dflow DEBUG: Task 2064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2064 has memoization hash 2bb962bcdc8a29d21fd1ded339916167 parsl.dataflow.memoization INFO: Task 2064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 499300cb7606c4552d09d19bedc2608d with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/94fcddcc-67f8-4d83-8285-90b8d75a962e_isrForCrosstalkSources_855408_25.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/94fcddcc-67f8-4d83-8285-90b8d75a962e_isrForCrosstalkSources_855408_25.stderr parsl.dataflow.dflow INFO: Parsl task 2064 try 0 launched on executor multi with executor id 2065 parsl.dataflow.dflow INFO: Standard output for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/014a0e5c-9567-45cb-a4dd-c075bcf7069c_isrForCrosstalkSources_855389_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/014a0e5c-9567-45cb-a4dd-c075bcf7069c_isrForCrosstalkSources_855389_6.stderr parsl.dataflow.dflow DEBUG: Task 2065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2065 has memoization hash f273482e0fb38cb358ef772c1c06be36 parsl.dataflow.memoization INFO: Task 2065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2065 try 0 launched on executor multi with executor id 2066 parsl.dataflow.dflow INFO: Standard output for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d2bd6483-2394-4e86-a0ae-8c09055840a5_isrForCrosstalkSources_855445_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d2bd6483-2394-4e86-a0ae-8c09055840a5_isrForCrosstalkSources_855445_7.stderr parsl.dataflow.dflow DEBUG: Task 2066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2066 has memoization hash d167ec207f26f507708bfc0726affbf3 parsl.dataflow.memoization INFO: Task 2066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2066 try 0 launched on executor multi with executor id 2067 parsl.dataflow.dflow INFO: Standard output for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ac295d63-b765-474f-b039-0ccd71f1fa6b_isrForCrosstalkSources_855390_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ac295d63-b765-474f-b039-0ccd71f1fa6b_isrForCrosstalkSources_855390_43.stderr parsl.dataflow.dflow DEBUG: Task 2067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2067 has memoization hash e91af3ef6db4b540af4b16b800f386e2 parsl.dataflow.memoization INFO: Task 2067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:52 proc_lsst.multi:170 [INFO] [multi] job_id 16964114 proc_lsst.multi INFO: [multi] job_id 16964114 2024-03-09 14:50:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-09 14:50:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964114 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964114 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16964114 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-09 14:50:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-09 14:50:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-09 14:50:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd3f44417537f24c73de7423f764bd1e with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e6928c95-26c9-4943-9fe2-cb2045d2b78e_isrForCrosstalkSources_855450_6.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e6928c95-26c9-4943-9fe2-cb2045d2b78e_isrForCrosstalkSources_855450_6.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c92e78be4198a7485c394ea573ad757 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f77a25e-3cd6-4b0a-a5b6-7212dc21aa01_isrForCrosstalkSources_855409_46.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f77a25e-3cd6-4b0a-a5b6-7212dc21aa01_isrForCrosstalkSources_855409_46.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecc156ea81fb568296dd3281239bbb56 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ec5ec741-aaf1-4f41-af89-bf88d52107ea_isrForCrosstalkSources_855400_6.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ec5ec741-aaf1-4f41-af89-bf88d52107ea_isrForCrosstalkSources_855400_6.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6c529506d8074def65ad4eb3e7df36b with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2e53abfc-2a33-4c1e-88cd-b8502deea102_isrForCrosstalkSources_855444_12.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2e53abfc-2a33-4c1e-88cd-b8502deea102_isrForCrosstalkSources_855444_12.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 283f9b8a6a337bf8a9ec70da15060eca with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e0b55214-6a28-47e9-aa43-e604f892428b_isrForCrosstalkSources_855443_36.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e0b55214-6a28-47e9-aa43-e604f892428b_isrForCrosstalkSources_855443_36.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e146ab17e7d0f054cdfbc5575a5642 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a8038f37-c6f7-45f1-bb0c-82ece3ea524b_isrForCrosstalkSources_855457_44.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a8038f37-c6f7-45f1-bb0c-82ece3ea524b_isrForCrosstalkSources_855457_44.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e85a72a9e82f6dcaa051d0f227ece183 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/efd3de75-89f6-4171-a6cf-a0fc3c40f669_isrForCrosstalkSources_855406_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/efd3de75-89f6-4171-a6cf-a0fc3c40f669_isrForCrosstalkSources_855406_2.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b06489eaf0887c26d823db045067ae6e with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/78e49f2c-c790-4d42-85a2-3bda14c2183c_isrForCrosstalkSources_855442_56.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/78e49f2c-c790-4d42-85a2-3bda14c2183c_isrForCrosstalkSources_855442_56.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f5315963947ae2b31f46799cbacd251 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/30b22380-6ecd-4c33-86db-caa72e7f20cb_isrForCrosstalkSources_855381_21.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/30b22380-6ecd-4c33-86db-caa72e7f20cb_isrForCrosstalkSources_855381_21.stderr parsl.dataflow.dflow INFO: Parsl task 2067 try 0 launched on executor multi with executor id 2068 parsl.dataflow.dflow INFO: Standard output for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/14626533-51a2-4964-8335-85b142cde9ef_isrForCrosstalkSources_855458_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/14626533-51a2-4964-8335-85b142cde9ef_isrForCrosstalkSources_855458_5.stderr parsl.dataflow.dflow DEBUG: Task 2068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2068 has memoization hash 20169b051239e9a92cc58a04545e3b09 parsl.dataflow.memoization INFO: Task 2068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b6df7b00a62d5c7c61408a7f6aeb843 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9f95f1ee-93c1-412d-90c2-2ec65b7578f7_isrForCrosstalkSources_855395_23.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9f95f1ee-93c1-412d-90c2-2ec65b7578f7_isrForCrosstalkSources_855395_23.stderr 2024-03-09 14:50:53 proc_lsst.multi:170 [INFO] [multi] job_id 16964115 proc_lsst.multi INFO: [multi] job_id 16964115 2024-03-09 14:50:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-09 14:50:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964115 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964115 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16964115 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-09 14:50:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-09 14:50:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-09 14:50:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cabe3788987602504f3388a736e7e02d with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4435830b-a058-470d-9459-09dcc2bdca13_isrForCrosstalkSources_855379_8.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4435830b-a058-470d-9459-09dcc2bdca13_isrForCrosstalkSources_855379_8.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcf421f4f842669c40c7fe45082aaddc with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8ca45d1b-b832-49b1-86e7-395da5ecb4be_isrForCrosstalkSources_855422_1.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8ca45d1b-b832-49b1-86e7-395da5ecb4be_isrForCrosstalkSources_855422_1.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a5d9748e83b2af65b6a3150c047e262 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2ada34f3-3c10-456a-aaa4-56b51246d719_isrForCrosstalkSources_855384_37.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/2ada34f3-3c10-456a-aaa4-56b51246d719_isrForCrosstalkSources_855384_37.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e628d83db5bdc2fd7a66445b6f4070 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d294132c-4929-41bc-b2b6-7f868f627409_isrForCrosstalkSources_855433_43.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d294132c-4929-41bc-b2b6-7f868f627409_isrForCrosstalkSources_855433_43.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db6cfbb94c1b3f015e7181ced3b9d5a7 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bba1f562-0ac8-4e8c-9f99-a88cbd130dc2_isrForCrosstalkSources_855450_48.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bba1f562-0ac8-4e8c-9f99-a88cbd130dc2_isrForCrosstalkSources_855450_48.stderr parsl.dataflow.dflow INFO: Parsl task 2068 try 0 launched on executor multi with executor id 2069 parsl.dataflow.dflow INFO: Standard output for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/533f88fa-62c0-47b1-9866-de08bf88777f_isrForCrosstalkSources_855418_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/533f88fa-62c0-47b1-9866-de08bf88777f_isrForCrosstalkSources_855418_59.stderr parsl.dataflow.dflow DEBUG: Task 2069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2069 has memoization hash 3d96bc47a7d0eabb294d42d720d3672b parsl.dataflow.memoization INFO: Task 2069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a75cef6ab6e51e7bd0e5696bac66d87b with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4f126955-6e46-4ec4-baea-2bac8a8d8c08_isrForCrosstalkSources_855421_61.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4f126955-6e46-4ec4-baea-2bac8a8d8c08_isrForCrosstalkSources_855421_61.stderr 2024-03-09 14:50:54 proc_lsst.multi:170 [INFO] [multi] job_id 16964116 proc_lsst.multi INFO: [multi] job_id 16964116 2024-03-09 14:50:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-09 14:50:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964116 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964116 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16964116 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-09 14:50:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-09 14:50:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-09 14:50:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8794e04b20985bff4c62ac4024e5b214 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/654c9797-6340-48de-b23e-dd1e696cbb3a_isrForCrosstalkSources_855379_23.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/654c9797-6340-48de-b23e-dd1e696cbb3a_isrForCrosstalkSources_855379_23.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd86a80eff622afba6731e9c0ea3a1e with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a7b154b2-51af-41f8-ab75-d5070c31723a_isrForCrosstalkSources_855448_31.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a7b154b2-51af-41f8-ab75-d5070c31723a_isrForCrosstalkSources_855448_31.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b26c28dc55b3ea9ffb7847e1f64c2f with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/65d887cc-b8a5-4910-b23b-7e765782b9a7_isrForCrosstalkSources_855446_44.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/65d887cc-b8a5-4910-b23b-7e765782b9a7_isrForCrosstalkSources_855446_44.stderr parsl.dataflow.dflow INFO: Parsl task 2069 try 0 launched on executor multi with executor id 2070 parsl.dataflow.dflow INFO: Standard output for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e22899e9-671d-4643-be6d-0b381a6a82d2_isrForCrosstalkSources_855391_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e22899e9-671d-4643-be6d-0b381a6a82d2_isrForCrosstalkSources_855391_27.stderr parsl.dataflow.dflow DEBUG: Task 2070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2070 has memoization hash 17b95e2a299bc50350bfd1033ddd73fc parsl.dataflow.memoization INFO: Task 2070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415ac16c7b4d47a7681fae994f022777 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c9d0a1c6-1ad7-4eef-bc79-207ab5490a4f_isrForCrosstalkSources_855385_49.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c9d0a1c6-1ad7-4eef-bc79-207ab5490a4f_isrForCrosstalkSources_855385_49.stderr 2024-03-09 14:50:55 proc_lsst.multi:170 [INFO] [multi] job_id 16964117 proc_lsst.multi INFO: [multi] job_id 16964117 2024-03-09 14:50:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-09 14:50:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964117 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964117 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 16964117 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-09 14:50:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-09 14:50:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-09 14:50:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e8ea2ad10ef1a4c04ae15eb44bcb8ec with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f76035d3-20ff-46f3-87ff-20ff80164a3e_isrForCrosstalkSources_855430_61.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f76035d3-20ff-46f3-87ff-20ff80164a3e_isrForCrosstalkSources_855430_61.stderr parsl.dataflow.dflow INFO: Parsl task 2070 try 0 launched on executor multi with executor id 2071 parsl.dataflow.dflow INFO: Standard output for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/62425bb3-91b1-40dd-a922-6f1d51c6ae2e_isrForCrosstalkSources_855415_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/62425bb3-91b1-40dd-a922-6f1d51c6ae2e_isrForCrosstalkSources_855415_43.stderr parsl.dataflow.dflow DEBUG: Task 2071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2071 has memoization hash a71410f76faac6a79206ef50639bdbfa parsl.dataflow.memoization INFO: Task 2071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2071 try 0 launched on executor multi with executor id 2072 parsl.dataflow.dflow INFO: Standard output for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c16f3700-e331-4f53-964e-0d8a604eba6d_isrForCrosstalkSources_855423_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c16f3700-e331-4f53-964e-0d8a604eba6d_isrForCrosstalkSources_855423_1.stderr parsl.dataflow.dflow DEBUG: Task 2072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2072 has memoization hash 3f0eeac64016c7701d4f84f074bfaafb parsl.dataflow.memoization INFO: Task 2072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 975f113b7057d8c2bff36b6087e292bb with result from task 74 parsl.dataflow.dflow INFO: Parsl task 2072 try 0 launched on executor multi with executor id 2073 parsl.dataflow.dflow INFO: Standard output for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/87027ef8-e8f9-4984-ace6-1aa69f70e8f7_isrForCrosstalkSources_855432_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/87027ef8-e8f9-4984-ace6-1aa69f70e8f7_isrForCrosstalkSources_855432_1.stderr parsl.dataflow.dflow DEBUG: Task 2073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2073 has memoization hash b2d73d26a7c6dd9249377a54679f9111 parsl.dataflow.memoization INFO: Task 2073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/2903731f-694e-4190-b37f-52c355faf5a1_isrForCrosstalkSources_855454_53.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/2903731f-694e-4190-b37f-52c355faf5a1_isrForCrosstalkSources_855454_53.stderr parsl.dataflow.dflow INFO: Parsl task 2073 try 0 launched on executor multi with executor id 2074 parsl.dataflow.dflow INFO: Standard output for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/af61e1ab-60d6-4f88-9da1-9aa39dc938d3_isrForCrosstalkSources_855438_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/af61e1ab-60d6-4f88-9da1-9aa39dc938d3_isrForCrosstalkSources_855438_13.stderr parsl.dataflow.dflow DEBUG: Task 2074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2074 has memoization hash 73be6d22febeb9cc0e23260c833dd686 parsl.dataflow.memoization INFO: Task 2074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2074 try 0 launched on executor multi with executor id 2075 parsl.dataflow.dflow INFO: Standard output for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c23d3c86-24ef-4225-8e6c-bf5bcedd23e7_isrForCrosstalkSources_855383_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c23d3c86-24ef-4225-8e6c-bf5bcedd23e7_isrForCrosstalkSources_855383_48.stderr parsl.dataflow.dflow DEBUG: Task 2075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2075 has memoization hash ac23bdeeb02edfb934463735282edcc5 parsl.dataflow.memoization INFO: Task 2075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2075 try 0 launched on executor multi with executor id 2076 parsl.dataflow.dflow INFO: Standard output for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b20cc6b7-984e-44f7-8e83-9b21cdaa8d62_isrForCrosstalkSources_855394_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b20cc6b7-984e-44f7-8e83-9b21cdaa8d62_isrForCrosstalkSources_855394_25.stderr parsl.dataflow.dflow DEBUG: Task 2076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2076 has memoization hash 58ee79c6c44eb8b905fa0ffcfc0af836 parsl.dataflow.memoization INFO: Task 2076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2076 try 0 launched on executor multi with executor id 2077 parsl.dataflow.dflow INFO: Standard output for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/343e3f8a-7485-4648-9b07-be90d4d00308_isrForCrosstalkSources_855385_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/343e3f8a-7485-4648-9b07-be90d4d00308_isrForCrosstalkSources_855385_4.stderr parsl.dataflow.dflow DEBUG: Task 2077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2077 has memoization hash d45b7311e4d73a1709b0694b0196b35b parsl.dataflow.memoization INFO: Task 2077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2077 try 0 launched on executor multi with executor id 2078 parsl.dataflow.dflow INFO: Standard output for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/08ba01d6-8b6a-4d3d-bdc6-d9c7f9a04b6a_isrForCrosstalkSources_855404_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/08ba01d6-8b6a-4d3d-bdc6-d9c7f9a04b6a_isrForCrosstalkSources_855404_14.stderr parsl.dataflow.dflow DEBUG: Task 2078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2078 has memoization hash 8c57f58314dc260c9502086da3ddeb8e parsl.dataflow.memoization INFO: Task 2078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2078 try 0 launched on executor multi with executor id 2079 parsl.dataflow.dflow INFO: Standard output for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/19f74548-f7b7-486d-913d-8b9d569971e8_isrForCrosstalkSources_855404_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/19f74548-f7b7-486d-913d-8b9d569971e8_isrForCrosstalkSources_855404_54.stderr parsl.dataflow.dflow DEBUG: Task 2079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2079 has memoization hash 16984164ee7ba7d7b2a2c167b555d362 parsl.dataflow.memoization INFO: Task 2079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2079 try 0 launched on executor multi with executor id 2080 parsl.dataflow.dflow INFO: Standard output for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/9e224c68-dbe2-479b-b916-33054328f857_isrForCrosstalkSources_855390_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/9e224c68-dbe2-479b-b916-33054328f857_isrForCrosstalkSources_855390_14.stderr parsl.dataflow.dflow DEBUG: Task 2080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2080 has memoization hash b7973840e21a04c12bfa666179a3bd19 parsl.dataflow.memoization INFO: Task 2080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2080 try 0 launched on executor multi with executor id 2081 parsl.dataflow.dflow INFO: Standard output for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/384c3f45-c087-425c-84fb-306d1f0ea26c_isrForCrosstalkSources_855428_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/384c3f45-c087-425c-84fb-306d1f0ea26c_isrForCrosstalkSources_855428_58.stderr parsl.dataflow.dflow DEBUG: Task 2081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2081 has memoization hash ee536651437ea88a6edfcdaff94d4173 parsl.dataflow.memoization INFO: Task 2081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2081 try 0 launched on executor multi with executor id 2082 parsl.dataflow.dflow INFO: Standard output for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4894d996-f2d6-440f-8d52-8599122314d0_isrForCrosstalkSources_855400_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4894d996-f2d6-440f-8d52-8599122314d0_isrForCrosstalkSources_855400_37.stderr parsl.dataflow.dflow DEBUG: Task 2082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2082 has memoization hash eab516c911cef3c1298e071a1f8be68a parsl.dataflow.memoization INFO: Task 2082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964118 proc_lsst.multi INFO: [multi] job_id 16964118 2024-03-09 14:50:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-09 14:50:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964118 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964118 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16964118 parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-09 14:50:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2082 try 0 launched on executor multi with executor id 2083 parsl.dataflow.dflow INFO: Standard output for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68a997c1-b08a-4ce4-8a2c-433765f04e2e_isrForCrosstalkSources_855455_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68a997c1-b08a-4ce4-8a2c-433765f04e2e_isrForCrosstalkSources_855455_42.stderr parsl.dataflow.dflow DEBUG: Task 2083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2083 has memoization hash 8dc090749f1762c69bf4d0bc45d681ed parsl.dataflow.memoization INFO: Task 2083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2083 try 0 launched on executor multi with executor id 2084 parsl.dataflow.dflow INFO: Standard output for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8e520ba5-323e-4282-9261-d74d467e2150_isrForCrosstalkSources_855445_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8e520ba5-323e-4282-9261-d74d467e2150_isrForCrosstalkSources_855445_49.stderr parsl.dataflow.dflow DEBUG: Task 2084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2084 has memoization hash 6066f2cad231c0b0f21794ecc00b5302 parsl.dataflow.memoization INFO: Task 2084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2084 try 0 launched on executor multi with executor id 2085 parsl.dataflow.dflow INFO: Standard output for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5bff9dd8-301a-427d-b435-3aac42e6a186_isrForCrosstalkSources_855391_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5bff9dd8-301a-427d-b435-3aac42e6a186_isrForCrosstalkSources_855391_10.stderr parsl.dataflow.dflow DEBUG: Task 2085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2085 has memoization hash 92e66890898663ec85521ad91a42d8d7 parsl.dataflow.memoization INFO: Task 2085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2085 try 0 launched on executor multi with executor id 2086 parsl.dataflow.dflow INFO: Standard output for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c447a757-845c-4105-9b78-e0ca27532d34_isrForCrosstalkSources_855458_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c447a757-845c-4105-9b78-e0ca27532d34_isrForCrosstalkSources_855458_9.stderr parsl.dataflow.dflow DEBUG: Task 2086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2086 has memoization hash 7cd2566fc0d1259d9576708337bfd9fb parsl.dataflow.memoization INFO: Task 2086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2086 try 0 launched on executor multi with executor id 2087 parsl.dataflow.dflow INFO: Standard output for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24ba6116-5896-4344-92da-805cd08e9b66_isrForCrosstalkSources_855444_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24ba6116-5896-4344-92da-805cd08e9b66_isrForCrosstalkSources_855444_46.stderr parsl.dataflow.dflow DEBUG: Task 2087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2087 has memoization hash 49e0539168124d1165f5709ca0341aa4 parsl.dataflow.memoization INFO: Task 2087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964119 proc_lsst.multi INFO: [multi] job_id 16964119 2024-03-09 14:50:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-09 14:50:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964119 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964119 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 16964119 parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-09 14:50:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2087 try 0 launched on executor multi with executor id 2088 parsl.dataflow.dflow INFO: Standard output for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cc7299b6-5f81-4cfc-a811-b8b10562cd7e_isrForCrosstalkSources_855412_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cc7299b6-5f81-4cfc-a811-b8b10562cd7e_isrForCrosstalkSources_855412_23.stderr parsl.dataflow.dflow DEBUG: Task 2088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2088 has memoization hash e126ec4c6be1aa192ea8ff2fe8090e97 parsl.dataflow.memoization INFO: Task 2088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48279bb21fc60bdb6f4c5ece5423454e with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2d77db6f-b069-496e-a0cf-a2ae5b2a36cf_isrForCrosstalkSources_855404_52.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2d77db6f-b069-496e-a0cf-a2ae5b2a36cf_isrForCrosstalkSources_855404_52.stderr 2024-03-09 14:50:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964120 proc_lsst.multi INFO: [multi] job_id 16964120 2024-03-09 14:50:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-09 14:50:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964120 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964120 parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 16964120 parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-09 14:50:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a37b75a39b353fe60ba907160fe010 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/94890e15-b2c9-4d76-aad0-eaf87a691673_isrForCrosstalkSources_855379_54.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/94890e15-b2c9-4d76-aad0-eaf87a691673_isrForCrosstalkSources_855379_54.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2088 try 0 launched on executor multi with executor id 2089 parsl.dataflow.dflow INFO: Standard output for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d869b044-07a7-485b-9947-42d6436972a7_isrForCrosstalkSources_855457_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d869b044-07a7-485b-9947-42d6436972a7_isrForCrosstalkSources_855457_19.stderr parsl.dataflow.dflow DEBUG: Task 2089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2089 has memoization hash 443e23d6f97ee6b4ad9ab522b5f204f4 parsl.dataflow.memoization INFO: Task 2089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac4ba10e95c824de60ab63dc7363598a with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d6b6318-fe1a-4a0f-b255-f7a53b61b1c3_isrForCrosstalkSources_855400_54.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d6b6318-fe1a-4a0f-b255-f7a53b61b1c3_isrForCrosstalkSources_855400_54.stderr parsl.dataflow.dflow INFO: Parsl task 2089 try 0 launched on executor multi with executor id 2090 parsl.dataflow.dflow INFO: Standard output for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d79eb5fb-bf83-47af-bc42-87357c0af806_isrForCrosstalkSources_855415_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d79eb5fb-bf83-47af-bc42-87357c0af806_isrForCrosstalkSources_855415_46.stderr parsl.dataflow.dflow DEBUG: Task 2090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2090 has memoization hash 0763553b745a4a98e33a2126aca147bf parsl.dataflow.memoization INFO: Task 2090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2090 try 0 launched on executor multi with executor id 2091 parsl.dataflow.dflow INFO: Standard output for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f2e1c7f5-2f69-4a1c-b036-58060b61e4a7_isrForCrosstalkSources_855419_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f2e1c7f5-2f69-4a1c-b036-58060b61e4a7_isrForCrosstalkSources_855419_9.stderr parsl.dataflow.dflow DEBUG: Task 2091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2091 has memoization hash 69f3eff892c9e87c6d8b663f1d3e5953 parsl.dataflow.memoization INFO: Task 2091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2091 try 0 launched on executor multi with executor id 2092 parsl.dataflow.dflow INFO: Standard output for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/68d113d0-cdca-45c3-b799-ce2281e48795_isrForCrosstalkSources_855435_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/68d113d0-cdca-45c3-b799-ce2281e48795_isrForCrosstalkSources_855435_41.stderr parsl.dataflow.dflow DEBUG: Task 2092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2092 has memoization hash 919ad2a5fbbb1d9db19277504f6628b2 parsl.dataflow.memoization INFO: Task 2092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964121 proc_lsst.multi INFO: [multi] job_id 16964121 2024-03-09 14:50:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-09 14:50:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964121 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964121 parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 16964121 parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-09 14:50:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-09 14:50:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2092 try 0 launched on executor multi with executor id 2093 parsl.dataflow.dflow INFO: Standard output for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/466e163a-5b8e-47c7-a4a0-912f4f1d7780_isrForCrosstalkSources_855405_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/466e163a-5b8e-47c7-a4a0-912f4f1d7780_isrForCrosstalkSources_855405_60.stderr parsl.dataflow.dflow DEBUG: Task 2093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2093 has memoization hash edf9fca314f92200fb85026486f16c2b parsl.dataflow.memoization INFO: Task 2093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2093 try 0 launched on executor multi with executor id 2094 parsl.dataflow.dflow INFO: Standard output for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e7070891-ee00-4ee7-a254-142e9f8d3e59_isrForCrosstalkSources_855436_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e7070891-ee00-4ee7-a254-142e9f8d3e59_isrForCrosstalkSources_855436_53.stderr parsl.dataflow.dflow DEBUG: Task 2094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2094 has memoization hash 2ae4f436f20ebb272c070c025f235f6d parsl.dataflow.memoization INFO: Task 2094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2094 try 0 launched on executor multi with executor id 2095 parsl.dataflow.dflow INFO: Standard output for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4bef055c-15cf-47f9-8c71-1211adaff5f5_isrForCrosstalkSources_855401_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4bef055c-15cf-47f9-8c71-1211adaff5f5_isrForCrosstalkSources_855401_49.stderr parsl.dataflow.dflow DEBUG: Task 2095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2095 has memoization hash f3fa4bd094ed2dd1ce14d1ca140ef974 parsl.dataflow.memoization INFO: Task 2095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2095 try 0 launched on executor multi with executor id 2096 parsl.dataflow.dflow INFO: Standard output for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/58e73ece-3098-4d39-9c64-35883c2b4bfe_isrForCrosstalkSources_855382_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/58e73ece-3098-4d39-9c64-35883c2b4bfe_isrForCrosstalkSources_855382_14.stderr parsl.dataflow.dflow DEBUG: Task 2096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2096 has memoization hash d5fcd81b9c9dec7e23c4ea6466750032 parsl.dataflow.memoization INFO: Task 2096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2096 try 0 launched on executor multi with executor id 2097 parsl.dataflow.dflow INFO: Standard output for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/798bd937-5adc-4f99-b5ab-ffffbb03b8f8_isrForCrosstalkSources_855414_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/798bd937-5adc-4f99-b5ab-ffffbb03b8f8_isrForCrosstalkSources_855414_55.stderr parsl.dataflow.dflow DEBUG: Task 2097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2097 has memoization hash 4cbe530143c1e8ef3e009de64b291e1f parsl.dataflow.memoization INFO: Task 2097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2097 try 0 launched on executor multi with executor id 2098 parsl.dataflow.dflow INFO: Standard output for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0996d724-f46b-4a1a-af2d-40964ab2e514_isrForCrosstalkSources_855387_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0996d724-f46b-4a1a-af2d-40964ab2e514_isrForCrosstalkSources_855387_5.stderr parsl.dataflow.dflow DEBUG: Task 2098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2098 has memoization hash fb7370bd8a16c28276f5772e6e7c32d0 parsl.dataflow.memoization INFO: Task 2098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2098 try 0 launched on executor multi with executor id 2099 parsl.dataflow.dflow INFO: Standard output for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/34ff6d7e-27e7-468d-b69b-e06461b18172_isrForCrosstalkSources_855402_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/34ff6d7e-27e7-468d-b69b-e06461b18172_isrForCrosstalkSources_855402_57.stderr parsl.dataflow.dflow DEBUG: Task 2099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2099 has memoization hash 2b32b94b46b0a39ed67ae249afce2dbe parsl.dataflow.memoization INFO: Task 2099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964122 proc_lsst.multi INFO: [multi] job_id 16964122 2024-03-09 14:50:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-09 14:50:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964122 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964122 parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 16964122 parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-09 14:50:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2099 try 0 launched on executor multi with executor id 2100 parsl.dataflow.dflow INFO: Standard output for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/04664b55-bb29-4346-889a-584fb9cb1bb9_isrForCrosstalkSources_855448_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/04664b55-bb29-4346-889a-584fb9cb1bb9_isrForCrosstalkSources_855448_44.stderr parsl.dataflow.dflow DEBUG: Task 2100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2100 has memoization hash 50eb3aecdc4b050229ddd596ede6dbac parsl.dataflow.memoization INFO: Task 2100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2100 try 0 launched on executor multi with executor id 2101 parsl.dataflow.dflow INFO: Standard output for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3b2a73c4-13ec-454b-8c88-92af2f508b8a_isrForCrosstalkSources_855452_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3b2a73c4-13ec-454b-8c88-92af2f508b8a_isrForCrosstalkSources_855452_47.stderr parsl.dataflow.dflow DEBUG: Task 2101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2101 has memoization hash 4a5e3cf50e7f3ab1c3fc8f79ba0c99de parsl.dataflow.memoization INFO: Task 2101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2101 try 0 launched on executor multi with executor id 2102 parsl.dataflow.dflow INFO: Standard output for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2df00e72-094c-4b5d-b60c-491be948dafd_isrForCrosstalkSources_855443_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2df00e72-094c-4b5d-b60c-491be948dafd_isrForCrosstalkSources_855443_4.stderr parsl.dataflow.dflow DEBUG: Task 2102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2102 has memoization hash 28dfc5fcc29fafa13393dba21920728e parsl.dataflow.memoization INFO: Task 2102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2102 try 0 launched on executor multi with executor id 2103 parsl.dataflow.dflow INFO: Standard output for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/d797de75-16c3-45cc-8160-dbf0b865075f_isrForCrosstalkSources_855453_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/d797de75-16c3-45cc-8160-dbf0b865075f_isrForCrosstalkSources_855453_13.stderr parsl.dataflow.dflow DEBUG: Task 2103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2103 has memoization hash 669ed5968d19f4e155dcd640e392ee08 parsl.dataflow.memoization INFO: Task 2103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2103 try 0 launched on executor multi with executor id 2104 parsl.dataflow.dflow INFO: Standard output for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/41b341fc-dd52-4d81-a5ee-6af936aac9e1_isrForCrosstalkSources_855416_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/41b341fc-dd52-4d81-a5ee-6af936aac9e1_isrForCrosstalkSources_855416_33.stderr parsl.dataflow.dflow DEBUG: Task 2104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2104 has memoization hash 8a902e716dbed34676963347171e373f parsl.dataflow.memoization INFO: Task 2104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2104 try 0 launched on executor multi with executor id 2105 parsl.dataflow.dflow INFO: Standard output for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ef2d0b7-5ce1-41a7-aa8f-93787674a8ac_isrForCrosstalkSources_855427_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ef2d0b7-5ce1-41a7-aa8f-93787674a8ac_isrForCrosstalkSources_855427_7.stderr parsl.dataflow.dflow DEBUG: Task 2105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2105 has memoization hash 6c6e8db1abb51fa686bbda291c68006c parsl.dataflow.memoization INFO: Task 2105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2105 try 0 launched on executor multi with executor id 2106 parsl.dataflow.dflow INFO: Standard output for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/82955c78-fa1a-409c-9306-18a7a537b10b_isrForCrosstalkSources_855436_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/82955c78-fa1a-409c-9306-18a7a537b10b_isrForCrosstalkSources_855436_45.stderr parsl.dataflow.dflow DEBUG: Task 2106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2106 has memoization hash e83cbd46c3c33fb3d6c43e517d19e9b3 parsl.dataflow.memoization INFO: Task 2106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2106 try 0 launched on executor multi with executor id 2107 parsl.dataflow.dflow INFO: Standard output for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f606330f-c16c-4bea-95d6-8b1201a05685_isrForCrosstalkSources_855388_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f606330f-c16c-4bea-95d6-8b1201a05685_isrForCrosstalkSources_855388_36.stderr parsl.dataflow.dflow DEBUG: Task 2107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2107 has memoization hash ad0cd4f623aabd37ffd0a99d22f9b811 parsl.dataflow.memoization INFO: Task 2107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2107 try 0 launched on executor multi with executor id 2108 parsl.dataflow.dflow INFO: Standard output for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a2500e5-c52b-4085-9496-ec8100dab1d5_isrForCrosstalkSources_855400_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a2500e5-c52b-4085-9496-ec8100dab1d5_isrForCrosstalkSources_855400_2.stderr parsl.dataflow.dflow DEBUG: Task 2108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2108 has memoization hash 43eb4e1ddfff7628296a3bb2831a0eb8 parsl.dataflow.memoization INFO: Task 2108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:50:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964123 proc_lsst.multi INFO: [multi] job_id 16964123 2024-03-09 14:50:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-09 14:50:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964123 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964123 parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 16964123 parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-09 14:50:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:50:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:50:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-09 14:50:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:50:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2108 try 0 launched on executor multi with executor id 2109 parsl.dataflow.dflow INFO: Standard output for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5843f953-649c-4709-aee3-8b3a8c4ffdc2_isrForCrosstalkSources_855436_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5843f953-649c-4709-aee3-8b3a8c4ffdc2_isrForCrosstalkSources_855436_38.stderr parsl.dataflow.dflow DEBUG: Task 2109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2109 has memoization hash bba360c61833907b2fa0aab37e2b3333 parsl.dataflow.memoization INFO: Task 2109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2109 try 0 launched on executor multi with executor id 2110 parsl.dataflow.dflow INFO: Standard output for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/0fa0bb6c-26b5-4343-8ef2-7f91f1141c0b_isrForCrosstalkSources_855453_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/0fa0bb6c-26b5-4343-8ef2-7f91f1141c0b_isrForCrosstalkSources_855453_30.stderr parsl.dataflow.dflow DEBUG: Task 2110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2110 has memoization hash 5f501260affaea8a726e10213938343a parsl.dataflow.memoization INFO: Task 2110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2110 try 0 launched on executor multi with executor id 2111 parsl.dataflow.dflow INFO: Standard output for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c4524680-65b3-462f-9e1c-144029330489_isrForCrosstalkSources_855441_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c4524680-65b3-462f-9e1c-144029330489_isrForCrosstalkSources_855441_37.stderr parsl.dataflow.dflow DEBUG: Task 2111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2111 has memoization hash 52dfa9b3b9b165e1aa5ff799bcd3cdc7 parsl.dataflow.memoization INFO: Task 2111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2111 try 0 launched on executor multi with executor id 2112 parsl.dataflow.dflow INFO: Standard output for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/58a0562d-3b18-4dea-9e10-6c33a07c46ce_isrForCrosstalkSources_855400_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/58a0562d-3b18-4dea-9e10-6c33a07c46ce_isrForCrosstalkSources_855400_60.stderr parsl.dataflow.dflow DEBUG: Task 2112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2112 has memoization hash f581e6d2ae1595ac0964fd242076de0f parsl.dataflow.memoization INFO: Task 2112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2112 try 0 launched on executor multi with executor id 2113 parsl.dataflow.dflow INFO: Standard output for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a50f6e6a-3be2-4e23-a7ae-834a507144fa_isrForCrosstalkSources_855409_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a50f6e6a-3be2-4e23-a7ae-834a507144fa_isrForCrosstalkSources_855409_61.stderr parsl.dataflow.dflow DEBUG: Task 2113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2113 has memoization hash 30bd97355db82dc92a9deafe670b064a parsl.dataflow.memoization INFO: Task 2113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2113 try 0 launched on executor multi with executor id 2114 parsl.dataflow.dflow INFO: Standard output for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f6f118f6-52f9-42dc-b366-0202ecff9f1d_isrForCrosstalkSources_855399_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f6f118f6-52f9-42dc-b366-0202ecff9f1d_isrForCrosstalkSources_855399_16.stderr parsl.dataflow.dflow DEBUG: Task 2114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2114 has memoization hash 467db57c8e9f6397535674d2ec9cbb32 parsl.dataflow.memoization INFO: Task 2114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2114 try 0 launched on executor multi with executor id 2115 parsl.dataflow.dflow INFO: Standard output for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/42d05cec-0a1f-4176-a2fc-6cb9bc15aa83_isrForCrosstalkSources_855415_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/42d05cec-0a1f-4176-a2fc-6cb9bc15aa83_isrForCrosstalkSources_855415_28.stderr parsl.dataflow.dflow DEBUG: Task 2115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2115 has memoization hash 6889898b1ad71137cf247d0808155940 parsl.dataflow.memoization INFO: Task 2115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2115 try 0 launched on executor multi with executor id 2116 parsl.dataflow.dflow INFO: Standard output for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/892f3446-5c5e-4325-9301-208614979cff_isrForCrosstalkSources_855425_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/892f3446-5c5e-4325-9301-208614979cff_isrForCrosstalkSources_855425_1.stderr parsl.dataflow.dflow DEBUG: Task 2116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2116 has memoization hash 87b9a98adf1840572b7d04768ffd49ed parsl.dataflow.memoization INFO: Task 2116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2116 try 0 launched on executor multi with executor id 2117 parsl.dataflow.dflow INFO: Standard output for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5802e5fc-6368-4444-abb5-7220750a9414_isrForCrosstalkSources_855444_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5802e5fc-6368-4444-abb5-7220750a9414_isrForCrosstalkSources_855444_20.stderr parsl.dataflow.dflow DEBUG: Task 2117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2117 has memoization hash 4abe1227f69d50b122c23833bd4afa90 parsl.dataflow.memoization INFO: Task 2117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2117 try 0 launched on executor multi with executor id 2118 parsl.dataflow.dflow INFO: Standard output for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/38058cb9-fe08-458c-8ff2-afab29e609e6_isrForCrosstalkSources_855429_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/38058cb9-fe08-458c-8ff2-afab29e609e6_isrForCrosstalkSources_855429_42.stderr parsl.dataflow.dflow DEBUG: Task 2118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2118 has memoization hash 975b76bb72b95e534fef00a3d0d702e0 parsl.dataflow.memoization INFO: Task 2118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2118 try 0 launched on executor multi with executor id 2119 parsl.dataflow.dflow INFO: Standard output for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed6c8424-28ae-4c82-94c6-fdcc81bed214_isrForCrosstalkSources_855427_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed6c8424-28ae-4c82-94c6-fdcc81bed214_isrForCrosstalkSources_855427_3.stderr parsl.dataflow.dflow DEBUG: Task 2119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2119 has memoization hash 63778f1181776553b66ccb9e8180f7f7 parsl.dataflow.memoization INFO: Task 2119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2119 try 0 launched on executor multi with executor id 2120 parsl.dataflow.dflow INFO: Standard output for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1a79629f-03ee-49fa-9984-45434c30668f_isrForCrosstalkSources_855447_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1a79629f-03ee-49fa-9984-45434c30668f_isrForCrosstalkSources_855447_21.stderr parsl.dataflow.dflow DEBUG: Task 2120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2120 has memoization hash 92f00cc6e07f03933e9df02e48d65410 parsl.dataflow.memoization INFO: Task 2120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2120 try 0 launched on executor multi with executor id 2121 parsl.dataflow.dflow INFO: Standard output for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/12b64628-cc59-4c13-b63a-a7da76d755cd_isrForCrosstalkSources_855420_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/12b64628-cc59-4c13-b63a-a7da76d755cd_isrForCrosstalkSources_855420_22.stderr parsl.dataflow.dflow DEBUG: Task 2121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2121 has memoization hash 36ec56242dce2b442348a6e69a2096a4 parsl.dataflow.memoization INFO: Task 2121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2121 try 0 launched on executor multi with executor id 2122 parsl.dataflow.dflow INFO: Standard output for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/51163d5b-4f7a-48d7-abf7-e5d1acf06f9e_isrForCrosstalkSources_855434_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/51163d5b-4f7a-48d7-abf7-e5d1acf06f9e_isrForCrosstalkSources_855434_8.stderr parsl.dataflow.dflow DEBUG: Task 2122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2122 has memoization hash 9773386965c411c3b369b02ac4f7e822 parsl.dataflow.memoization INFO: Task 2122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2122 try 0 launched on executor multi with executor id 2123 parsl.dataflow.dflow INFO: Standard output for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/add05326-0f4d-478b-b234-a34592b3f9ee_isrForCrosstalkSources_855380_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/add05326-0f4d-478b-b234-a34592b3f9ee_isrForCrosstalkSources_855380_38.stderr parsl.dataflow.dflow DEBUG: Task 2123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2123 has memoization hash 4d8c6d52c7617ac253736ed8d168a42c parsl.dataflow.memoization INFO: Task 2123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2123 try 0 launched on executor multi with executor id 2124 parsl.dataflow.dflow INFO: Standard output for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cbfdf39d-b2ce-482e-bdb5-4937ed1f74d2_isrForCrosstalkSources_855445_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cbfdf39d-b2ce-482e-bdb5-4937ed1f74d2_isrForCrosstalkSources_855445_27.stderr parsl.dataflow.dflow DEBUG: Task 2124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2124 has memoization hash 9263ea19e65a88a8f1dc0ccd60297e10 parsl.dataflow.memoization INFO: Task 2124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2124 try 0 launched on executor multi with executor id 2125 parsl.dataflow.dflow INFO: Standard output for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ead198c-f371-4cbc-a5c5-8f1f4da1d082_isrForCrosstalkSources_855445_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ead198c-f371-4cbc-a5c5-8f1f4da1d082_isrForCrosstalkSources_855445_62.stderr parsl.dataflow.dflow DEBUG: Task 2125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2125 has memoization hash effcedcc0a7fd2e0c849f37de2e52b4e parsl.dataflow.memoization INFO: Task 2125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2125 try 0 launched on executor multi with executor id 2126 parsl.dataflow.dflow INFO: Standard output for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6182ed0c-a33d-4ad1-b55d-a9b0ceb8dfd9_isrForCrosstalkSources_855425_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6182ed0c-a33d-4ad1-b55d-a9b0ceb8dfd9_isrForCrosstalkSources_855425_7.stderr parsl.dataflow.dflow DEBUG: Task 2126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2126 has memoization hash 79eef9ded790c15540a80846f4c5e748 parsl.dataflow.memoization INFO: Task 2126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2126 try 0 launched on executor multi with executor id 2127 parsl.dataflow.dflow INFO: Standard output for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/b82ace8d-f11b-49b8-8e40-090b288ae8a1_isrForCrosstalkSources_855417_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/b82ace8d-f11b-49b8-8e40-090b288ae8a1_isrForCrosstalkSources_855417_18.stderr parsl.dataflow.dflow DEBUG: Task 2127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2127 has memoization hash 97c90f751ab6e153f86f938637e2cc3f parsl.dataflow.memoization INFO: Task 2127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2127 try 0 launched on executor multi with executor id 2128 parsl.dataflow.dflow INFO: Standard output for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/83cf2b45-3431-4bd9-a790-217c41d957b2_isrForCrosstalkSources_855436_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/83cf2b45-3431-4bd9-a790-217c41d957b2_isrForCrosstalkSources_855436_7.stderr parsl.dataflow.dflow DEBUG: Task 2128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2128 has memoization hash 781f97ddfa3dcaa3f730356de7823fb0 parsl.dataflow.memoization INFO: Task 2128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2128 try 0 launched on executor multi with executor id 2129 parsl.dataflow.dflow INFO: Standard output for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5174f3f5-39f8-4669-990d-133c31470342_isrForCrosstalkSources_855436_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5174f3f5-39f8-4669-990d-133c31470342_isrForCrosstalkSources_855436_16.stderr parsl.dataflow.dflow DEBUG: Task 2129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2129 has memoization hash 07be29bd091113ca1ed2deeb9f0e2090 parsl.dataflow.memoization INFO: Task 2129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2129 try 0 launched on executor multi with executor id 2130 parsl.dataflow.dflow INFO: Standard output for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04cb6fc9-33c7-4457-96bd-cd068b6e13ee_isrForCrosstalkSources_855440_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04cb6fc9-33c7-4457-96bd-cd068b6e13ee_isrForCrosstalkSources_855440_21.stderr parsl.dataflow.dflow DEBUG: Task 2130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2130 has memoization hash 3afb9febb6b3126711d6d69f7135cccd parsl.dataflow.memoization INFO: Task 2130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2130 try 0 launched on executor multi with executor id 2131 parsl.dataflow.dflow INFO: Standard output for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/54674d2f-fc09-467c-bced-7258793982bb_isrForCrosstalkSources_855422_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/54674d2f-fc09-467c-bced-7258793982bb_isrForCrosstalkSources_855422_10.stderr parsl.dataflow.dflow DEBUG: Task 2131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2131 has memoization hash 344d31ee4610e65d5f6d96bc338de2f6 parsl.dataflow.memoization INFO: Task 2131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2131 try 0 launched on executor multi with executor id 2132 parsl.dataflow.dflow INFO: Standard output for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e494d765-b3b5-4cd3-9463-e9d749c2cfed_isrForCrosstalkSources_855418_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e494d765-b3b5-4cd3-9463-e9d749c2cfed_isrForCrosstalkSources_855418_14.stderr parsl.dataflow.dflow DEBUG: Task 2132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2132 has memoization hash a8916a6c5ab3b4113b9daeb3b626b6a9 parsl.dataflow.memoization INFO: Task 2132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2132 try 0 launched on executor multi with executor id 2133 parsl.dataflow.dflow INFO: Standard output for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/162f738d-3a9b-44bf-8978-e08f62aa10c8_isrForCrosstalkSources_855424_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/162f738d-3a9b-44bf-8978-e08f62aa10c8_isrForCrosstalkSources_855424_54.stderr parsl.dataflow.dflow DEBUG: Task 2133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2133 has memoization hash 06c62fc21e78ce19e85ba90d6cc4b919 parsl.dataflow.memoization INFO: Task 2133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2133 try 0 launched on executor multi with executor id 2134 parsl.dataflow.dflow INFO: Standard output for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/bd204bf0-6e2d-448c-be95-8a922e401250_isrForCrosstalkSources_855425_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/bd204bf0-6e2d-448c-be95-8a922e401250_isrForCrosstalkSources_855425_62.stderr parsl.dataflow.dflow DEBUG: Task 2134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2134 has memoization hash b1e133ecd669b54b86502e051deeacd1 parsl.dataflow.memoization INFO: Task 2134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2134 try 0 launched on executor multi with executor id 2135 parsl.dataflow.dflow INFO: Standard output for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5c2ee636-3d0a-4399-ba70-90774a7a1a08_isrForCrosstalkSources_855406_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5c2ee636-3d0a-4399-ba70-90774a7a1a08_isrForCrosstalkSources_855406_46.stderr parsl.dataflow.dflow DEBUG: Task 2135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2135 has memoization hash eb5c74e2587274f78c19d85fd42fb22b parsl.dataflow.memoization INFO: Task 2135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2135 try 0 launched on executor multi with executor id 2136 parsl.dataflow.dflow INFO: Standard output for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/b68caddc-c4b6-4004-bc8b-97de5f14369c_isrForCrosstalkSources_855431_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/b68caddc-c4b6-4004-bc8b-97de5f14369c_isrForCrosstalkSources_855431_14.stderr parsl.dataflow.dflow DEBUG: Task 2136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2136 has memoization hash f4942778b2948bfd5d4d26ec2352e14d parsl.dataflow.memoization INFO: Task 2136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2136 try 0 launched on executor multi with executor id 2137 parsl.dataflow.dflow INFO: Standard output for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6d0e94f9-8354-4683-9438-dc9d3b8102bc_isrForCrosstalkSources_855408_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6d0e94f9-8354-4683-9438-dc9d3b8102bc_isrForCrosstalkSources_855408_45.stderr parsl.dataflow.dflow DEBUG: Task 2137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2137 has memoization hash d78c4f1d90257e66e4b4a04df6c378df parsl.dataflow.memoization INFO: Task 2137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2137 try 0 launched on executor multi with executor id 2138 parsl.dataflow.dflow INFO: Standard output for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7aca9415-8032-45fd-8fc6-e77318ab86ea_isrForCrosstalkSources_855381_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7aca9415-8032-45fd-8fc6-e77318ab86ea_isrForCrosstalkSources_855381_11.stderr parsl.dataflow.dflow DEBUG: Task 2138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2138 has memoization hash aa9f924fb5512b8e19d0e471140c0af8 parsl.dataflow.memoization INFO: Task 2138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2138 try 0 launched on executor multi with executor id 2139 parsl.dataflow.dflow INFO: Standard output for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bfeb9968-9c61-4a74-86f3-869c5d90e325_isrForCrosstalkSources_855386_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bfeb9968-9c61-4a74-86f3-869c5d90e325_isrForCrosstalkSources_855386_32.stderr parsl.dataflow.dflow DEBUG: Task 2139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2139 has memoization hash 0654a6f2823d83a259a9971e34619a42 parsl.dataflow.memoization INFO: Task 2139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2139 try 0 launched on executor multi with executor id 2140 parsl.dataflow.dflow INFO: Standard output for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/63ce73e6-90ba-481e-ba00-4ec48e5ef5f0_isrForCrosstalkSources_855385_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/63ce73e6-90ba-481e-ba00-4ec48e5ef5f0_isrForCrosstalkSources_855385_16.stderr parsl.dataflow.dflow DEBUG: Task 2140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2140 has memoization hash ef04ba1594dcecf19ad0d20f22fe202f parsl.dataflow.memoization INFO: Task 2140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2140 try 0 launched on executor multi with executor id 2141 parsl.dataflow.dflow INFO: Standard output for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/524683ad-04da-4d11-bb13-693d4127ce61_isrForCrosstalkSources_855408_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/524683ad-04da-4d11-bb13-693d4127ce61_isrForCrosstalkSources_855408_59.stderr parsl.dataflow.dflow DEBUG: Task 2141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2141 has memoization hash 50618d649b3fe8c5649fc2353e408a87 parsl.dataflow.memoization INFO: Task 2141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2141 try 0 launched on executor multi with executor id 2142 parsl.dataflow.dflow INFO: Standard output for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/aedda8e7-bbae-4be6-8780-31dfcd3b57a9_isrForCrosstalkSources_855412_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/aedda8e7-bbae-4be6-8780-31dfcd3b57a9_isrForCrosstalkSources_855412_30.stderr parsl.dataflow.dflow DEBUG: Task 2142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2142 has memoization hash 244aa1cd8d191b3d1f39c9aee6a0e407 parsl.dataflow.memoization INFO: Task 2142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2142 try 0 launched on executor multi with executor id 2143 parsl.dataflow.dflow INFO: Standard output for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fbe313e0-5187-4e9c-8786-03c703bc41ed_isrForCrosstalkSources_855442_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fbe313e0-5187-4e9c-8786-03c703bc41ed_isrForCrosstalkSources_855442_31.stderr parsl.dataflow.dflow DEBUG: Task 2143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2143 has memoization hash 2370cc75ee4487b437f97ed2d4aae669 parsl.dataflow.memoization INFO: Task 2143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2143 try 0 launched on executor multi with executor id 2144 parsl.dataflow.dflow INFO: Standard output for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7af609e0-9a3c-4e1d-bfc8-9d6b5824c607_isrForCrosstalkSources_855422_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7af609e0-9a3c-4e1d-bfc8-9d6b5824c607_isrForCrosstalkSources_855422_21.stderr parsl.dataflow.dflow DEBUG: Task 2144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2144 has memoization hash 94b9e717e0bc20d65d6720bdc196d4ad parsl.dataflow.memoization INFO: Task 2144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2144 try 0 launched on executor multi with executor id 2145 parsl.dataflow.dflow INFO: Standard output for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8f3d366e-613c-4610-ad5f-cf21321c8678_isrForCrosstalkSources_855450_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8f3d366e-613c-4610-ad5f-cf21321c8678_isrForCrosstalkSources_855450_55.stderr parsl.dataflow.dflow DEBUG: Task 2145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2145 has memoization hash 36f98afe7fb0f09936b7cf8c4c07ec02 parsl.dataflow.memoization INFO: Task 2145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2145 try 0 launched on executor multi with executor id 2146 parsl.dataflow.dflow INFO: Standard output for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e5979b25-587a-4302-ba58-104f34ac9ed6_isrForCrosstalkSources_855456_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e5979b25-587a-4302-ba58-104f34ac9ed6_isrForCrosstalkSources_855456_21.stderr parsl.dataflow.dflow DEBUG: Task 2146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2146 has memoization hash 117ddd224ff0e9d72f6ad46ef1931079 parsl.dataflow.memoization INFO: Task 2146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2146 try 0 launched on executor multi with executor id 2147 parsl.dataflow.dflow INFO: Standard output for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/4ea6cc67-fe22-40cc-b85f-6ba47067b0b6_isrForCrosstalkSources_855396_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/4ea6cc67-fe22-40cc-b85f-6ba47067b0b6_isrForCrosstalkSources_855396_22.stderr parsl.dataflow.dflow DEBUG: Task 2147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2147 has memoization hash 25ce78e5419872784a600d62c844b09c parsl.dataflow.memoization INFO: Task 2147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2147 try 0 launched on executor multi with executor id 2148 parsl.dataflow.dflow INFO: Standard output for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bc1604f0-5276-42de-befd-b4acc8f42011_isrForCrosstalkSources_855397_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bc1604f0-5276-42de-befd-b4acc8f42011_isrForCrosstalkSources_855397_14.stderr parsl.dataflow.dflow DEBUG: Task 2148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2148 has memoization hash 10635ad7de62ac74bb5179b35f4ba5d8 parsl.dataflow.memoization INFO: Task 2148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2148 try 0 launched on executor multi with executor id 2149 parsl.dataflow.dflow INFO: Standard output for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/33fdadc6-1463-465a-bd88-961655bf045b_isrForCrosstalkSources_855411_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/33fdadc6-1463-465a-bd88-961655bf045b_isrForCrosstalkSources_855411_14.stderr parsl.dataflow.dflow DEBUG: Task 2149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2149 has memoization hash 0f42ecf45eb8f7aaad9a6986d048dc9d parsl.dataflow.memoization INFO: Task 2149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2149 try 0 launched on executor multi with executor id 2150 parsl.dataflow.dflow INFO: Standard output for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/95c51693-86ee-4210-bcf3-d34c93302693_isrForCrosstalkSources_855433_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/95c51693-86ee-4210-bcf3-d34c93302693_isrForCrosstalkSources_855433_42.stderr parsl.dataflow.dflow DEBUG: Task 2150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2150 has memoization hash 96cc86f66a615412b2c2f46184b8b2b5 parsl.dataflow.memoization INFO: Task 2150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2150 try 0 launched on executor multi with executor id 2151 parsl.dataflow.dflow INFO: Standard output for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/da9c9462-dff2-4330-893b-9d98f42f9d85_isrForCrosstalkSources_855439_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/da9c9462-dff2-4330-893b-9d98f42f9d85_isrForCrosstalkSources_855439_24.stderr parsl.dataflow.dflow DEBUG: Task 2151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2151 has memoization hash 0c48c47ecfb686831d8b118aba08a70e parsl.dataflow.memoization INFO: Task 2151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2151 try 0 launched on executor multi with executor id 2152 parsl.dataflow.dflow INFO: Standard output for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7885c3c6-3d32-47a5-b7d4-8fa94525f9f7_isrForCrosstalkSources_855436_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7885c3c6-3d32-47a5-b7d4-8fa94525f9f7_isrForCrosstalkSources_855436_13.stderr parsl.dataflow.dflow DEBUG: Task 2152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2152 has memoization hash 7b2f20a5f6499b619f8db332e95908db parsl.dataflow.memoization INFO: Task 2152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2152 try 0 launched on executor multi with executor id 2153 parsl.dataflow.dflow INFO: Standard output for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/08ab1ede-2be7-48fa-bc45-2a84d7b1ef6f_isrForCrosstalkSources_855393_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/08ab1ede-2be7-48fa-bc45-2a84d7b1ef6f_isrForCrosstalkSources_855393_33.stderr parsl.dataflow.dflow DEBUG: Task 2153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2153 has memoization hash 50f308212660a770a7942bb4b71fdc22 parsl.dataflow.memoization INFO: Task 2153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2153 try 0 launched on executor multi with executor id 2154 parsl.dataflow.dflow INFO: Standard output for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/342d3c40-463a-4d66-9676-5698ff60cc15_isrForCrosstalkSources_855436_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/342d3c40-463a-4d66-9676-5698ff60cc15_isrForCrosstalkSources_855436_37.stderr parsl.dataflow.dflow DEBUG: Task 2154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2154 has memoization hash 769bcd040da8aa75ca4177d72b6a5e75 parsl.dataflow.memoization INFO: Task 2154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2154 try 0 launched on executor multi with executor id 2155 parsl.dataflow.dflow INFO: Standard output for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bdf01289-64f1-490c-84a0-fe0d9692eecf_isrForCrosstalkSources_855410_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bdf01289-64f1-490c-84a0-fe0d9692eecf_isrForCrosstalkSources_855410_5.stderr parsl.dataflow.dflow DEBUG: Task 2155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2155 has memoization hash ed38a014d39c491c4510ba375708e327 parsl.dataflow.memoization INFO: Task 2155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2155 try 0 launched on executor multi with executor id 2156 parsl.dataflow.dflow INFO: Standard output for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/87141871-97b7-4c07-b1ef-8aad7c17d70e_isrForCrosstalkSources_855457_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/87141871-97b7-4c07-b1ef-8aad7c17d70e_isrForCrosstalkSources_855457_59.stderr parsl.dataflow.dflow DEBUG: Task 2156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2156 has memoization hash 02c68160da5d6145b2040eb31922bad8 parsl.dataflow.memoization INFO: Task 2156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2156 try 0 launched on executor multi with executor id 2157 parsl.dataflow.dflow INFO: Standard output for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3af54afd-0fc7-4fa9-8dcc-738b444aba12_isrForCrosstalkSources_855386_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3af54afd-0fc7-4fa9-8dcc-738b444aba12_isrForCrosstalkSources_855386_38.stderr parsl.dataflow.dflow DEBUG: Task 2157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2157 has memoization hash 81c607f13e19771bf533f48e9af3edf8 parsl.dataflow.memoization INFO: Task 2157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2157 try 0 launched on executor multi with executor id 2158 parsl.dataflow.dflow INFO: Standard output for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/664b9362-dc40-4600-9b25-88560ed90756_isrForCrosstalkSources_855401_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/664b9362-dc40-4600-9b25-88560ed90756_isrForCrosstalkSources_855401_29.stderr parsl.dataflow.dflow DEBUG: Task 2158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2158 has memoization hash 4cac8102bc88c04e14d28a3da21f9c87 parsl.dataflow.memoization INFO: Task 2158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2158 try 0 launched on executor multi with executor id 2159 parsl.dataflow.dflow INFO: Standard output for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/70a9f4e0-e2f2-4bab-8f3e-4efc3129da0c_isrForCrosstalkSources_855399_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/70a9f4e0-e2f2-4bab-8f3e-4efc3129da0c_isrForCrosstalkSources_855399_56.stderr parsl.dataflow.dflow DEBUG: Task 2159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2159 has memoization hash ffa21a1c4e23747fca89c90e8d87ed29 parsl.dataflow.memoization INFO: Task 2159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2159 try 0 launched on executor multi with executor id 2160 parsl.dataflow.dflow INFO: Standard output for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/04e3e035-6c1c-477a-bf64-ebf78ec77bb9_isrForCrosstalkSources_855425_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/04e3e035-6c1c-477a-bf64-ebf78ec77bb9_isrForCrosstalkSources_855425_17.stderr parsl.dataflow.dflow DEBUG: Task 2160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2160 has memoization hash 78e63dad178ea1245315844f283527d9 parsl.dataflow.memoization INFO: Task 2160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2160 try 0 launched on executor multi with executor id 2161 parsl.dataflow.dflow INFO: Standard output for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3aac6332-eb7c-4d09-b92a-bbdd3d4dcf44_isrForCrosstalkSources_855420_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3aac6332-eb7c-4d09-b92a-bbdd3d4dcf44_isrForCrosstalkSources_855420_42.stderr parsl.dataflow.dflow DEBUG: Task 2161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2161 has memoization hash 0ff1a73d38eeda8fcf5f7bd6ae574658 parsl.dataflow.memoization INFO: Task 2161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2161 try 0 launched on executor multi with executor id 2162 parsl.dataflow.dflow INFO: Standard output for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35119359-d300-4283-b9b8-f0e94cd1a376_isrForCrosstalkSources_855409_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35119359-d300-4283-b9b8-f0e94cd1a376_isrForCrosstalkSources_855409_5.stderr parsl.dataflow.dflow DEBUG: Task 2162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2162 has memoization hash c58208bfbbe4425f3b40fb69fc7c8f67 parsl.dataflow.memoization INFO: Task 2162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2162 try 0 launched on executor multi with executor id 2163 parsl.dataflow.dflow INFO: Standard output for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/807abd61-00e7-4437-bfe1-f9930bef95ce_isrForCrosstalkSources_855431_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/807abd61-00e7-4437-bfe1-f9930bef95ce_isrForCrosstalkSources_855431_20.stderr parsl.dataflow.dflow DEBUG: Task 2163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2163 has memoization hash 5763c87aa4acaecba155f382b208dfa3 parsl.dataflow.memoization INFO: Task 2163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2163 try 0 launched on executor multi with executor id 2164 parsl.dataflow.dflow INFO: Standard output for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2bf82d1c-a2fc-45da-a0f0-9530180c88d5_isrForCrosstalkSources_855400_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2bf82d1c-a2fc-45da-a0f0-9530180c88d5_isrForCrosstalkSources_855400_26.stderr parsl.dataflow.dflow DEBUG: Task 2164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2164 has memoization hash 8f9d8fc728a6a2e0428fac1097a459b1 parsl.dataflow.memoization INFO: Task 2164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2164 try 0 launched on executor multi with executor id 2165 parsl.dataflow.dflow INFO: Standard output for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/363a2cee-6061-4100-88fb-e1a06b242fa8_isrForCrosstalkSources_855435_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/363a2cee-6061-4100-88fb-e1a06b242fa8_isrForCrosstalkSources_855435_22.stderr parsl.dataflow.dflow DEBUG: Task 2165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2165 has memoization hash 2e3e1b54136acace8954ae400ddfe80b parsl.dataflow.memoization INFO: Task 2165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f968c0df8ae2e989b0865c97b7806fb8 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f19f2bc5-f453-46bb-9529-3df745494d3a_isrForCrosstalkSources_855395_19.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f19f2bc5-f453-46bb-9529-3df745494d3a_isrForCrosstalkSources_855395_19.stderr parsl.dataflow.dflow INFO: Parsl task 2165 try 0 launched on executor multi with executor id 2166 parsl.dataflow.dflow INFO: Standard output for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c9fe0283-9c7f-471b-b751-42677cd2febc_isrForCrosstalkSources_855439_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c9fe0283-9c7f-471b-b751-42677cd2febc_isrForCrosstalkSources_855439_58.stderr parsl.dataflow.dflow DEBUG: Task 2166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2166 has memoization hash fc486888e3d953507267a90d2c1d17ac parsl.dataflow.memoization INFO: Task 2166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2166 try 0 launched on executor multi with executor id 2167 parsl.dataflow.dflow INFO: Standard output for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e43ad5bd-d650-4862-a7e3-8d5fa7b1f487_isrForCrosstalkSources_855382_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e43ad5bd-d650-4862-a7e3-8d5fa7b1f487_isrForCrosstalkSources_855382_25.stderr parsl.dataflow.dflow DEBUG: Task 2167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2167 has memoization hash 062e26e0f220684706d058e3caa06e54 parsl.dataflow.memoization INFO: Task 2167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2167 try 0 launched on executor multi with executor id 2168 parsl.dataflow.dflow INFO: Standard output for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/90cb8e99-7c3e-476a-8d8b-6214edd07c64_isrForCrosstalkSources_855392_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/90cb8e99-7c3e-476a-8d8b-6214edd07c64_isrForCrosstalkSources_855392_2.stderr parsl.dataflow.dflow DEBUG: Task 2168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2168 has memoization hash fbb3e7dc389c1b860f5f55152895b717 parsl.dataflow.memoization INFO: Task 2168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2168 try 0 launched on executor multi with executor id 2169 parsl.dataflow.dflow INFO: Standard output for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/347a1464-336a-451d-8772-50ee6f540ef5_isrForCrosstalkSources_855422_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/347a1464-336a-451d-8772-50ee6f540ef5_isrForCrosstalkSources_855422_2.stderr parsl.dataflow.dflow DEBUG: Task 2169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2169 has memoization hash 3b95d28af1f383ec57f5fa92ab6faea9 parsl.dataflow.memoization INFO: Task 2169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2169 try 0 launched on executor multi with executor id 2170 parsl.dataflow.dflow INFO: Standard output for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b9329f7b-85a8-4711-8c68-f9b586957f3e_isrForCrosstalkSources_855399_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b9329f7b-85a8-4711-8c68-f9b586957f3e_isrForCrosstalkSources_855399_36.stderr parsl.dataflow.dflow DEBUG: Task 2170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2170 has memoization hash d11ef85c37383c37e9e9b9fa018518c1 parsl.dataflow.memoization INFO: Task 2170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6522c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2170 try 0 launched on executor multi with executor id 2171 parsl.dataflow.dflow INFO: Standard output for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/8f88c479-9371-4ab8-8f11-657b19756223_isrForCrosstalkSources_855438_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/8f88c479-9371-4ab8-8f11-657b19756223_isrForCrosstalkSources_855438_19.stderr parsl.dataflow.dflow DEBUG: Task 2171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2171 has memoization hash 1c3c9145ae85b4d63ef4dff2b1f32d75 parsl.dataflow.memoization INFO: Task 2171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2171 try 0 launched on executor multi with executor id 2172 parsl.dataflow.dflow INFO: Standard output for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e58045cf-c247-454b-b2b3-142c7e860d76_isrForCrosstalkSources_855458_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e58045cf-c247-454b-b2b3-142c7e860d76_isrForCrosstalkSources_855458_55.stderr parsl.dataflow.dflow DEBUG: Task 2172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2172 has memoization hash 008488c7c5275a6771ecd738f54c7cb2 parsl.dataflow.memoization INFO: Task 2172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2172 try 0 launched on executor multi with executor id 2173 parsl.dataflow.dflow INFO: Standard output for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/664637fb-bbbe-4a50-a742-b88fd35d6571_isrForCrosstalkSources_855404_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/664637fb-bbbe-4a50-a742-b88fd35d6571_isrForCrosstalkSources_855404_44.stderr parsl.dataflow.dflow DEBUG: Task 2173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2173 has memoization hash ea7c70d20d276a6330ef0ed6acc8cb38 parsl.dataflow.memoization INFO: Task 2173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2173 try 0 launched on executor multi with executor id 2174 parsl.dataflow.dflow INFO: Standard output for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3576c736-0e40-4ec2-bf88-aa7d01e341ec_isrForCrosstalkSources_855413_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3576c736-0e40-4ec2-bf88-aa7d01e341ec_isrForCrosstalkSources_855413_22.stderr parsl.dataflow.dflow DEBUG: Task 2174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2174 has memoization hash cfd9e5e95e3005352a7e841ee110ca8f parsl.dataflow.memoization INFO: Task 2174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2174 try 0 launched on executor multi with executor id 2175 parsl.dataflow.dflow INFO: Standard output for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e837e086-91f3-4b86-a9cd-a2ca43e75baf_isrForCrosstalkSources_855453_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e837e086-91f3-4b86-a9cd-a2ca43e75baf_isrForCrosstalkSources_855453_36.stderr parsl.dataflow.dflow DEBUG: Task 2175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2175 has memoization hash 566cbf68ee180ab7596ae0a3fb4c55b3 parsl.dataflow.memoization INFO: Task 2175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2175 try 0 launched on executor multi with executor id 2176 parsl.dataflow.dflow INFO: Standard output for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/63ff24bb-a27b-409e-ace5-f9b42391331e_isrForCrosstalkSources_855425_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/63ff24bb-a27b-409e-ace5-f9b42391331e_isrForCrosstalkSources_855425_34.stderr parsl.dataflow.dflow DEBUG: Task 2176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2176 has memoization hash 88dd9b40ed02337b383c770cfdc4be17 parsl.dataflow.memoization INFO: Task 2176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2176 try 0 launched on executor multi with executor id 2177 parsl.dataflow.dflow INFO: Standard output for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/30cbd2d7-1afb-44a6-9b46-59a5894492fb_isrForCrosstalkSources_855431_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/30cbd2d7-1afb-44a6-9b46-59a5894492fb_isrForCrosstalkSources_855431_43.stderr parsl.dataflow.dflow DEBUG: Task 2177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2177 has memoization hash 3deb2ee4ad906e82d47d91d45d6777fc parsl.dataflow.memoization INFO: Task 2177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6520fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2177 try 0 launched on executor multi with executor id 2178 parsl.dataflow.dflow INFO: Standard output for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2da315fc-47b7-4d48-a299-754e0ae0cded_isrForCrosstalkSources_855419_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2da315fc-47b7-4d48-a299-754e0ae0cded_isrForCrosstalkSources_855419_59.stderr parsl.dataflow.dflow DEBUG: Task 2178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2178 has memoization hash 6599b195a00d84ad41d72b8a07d0707c parsl.dataflow.memoization INFO: Task 2178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964124 proc_lsst.multi INFO: [multi] job_id 16964124 2024-03-09 14:51:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-09 14:51:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964124 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964124 parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 16964124 parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-09 14:51:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-09 14:51:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-09 14:51:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b37082f97105925d5776f83826097f9 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1caed2e4-b98c-4b09-945e-94009ee52b3b_isrForCrosstalkSources_855450_32.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1caed2e4-b98c-4b09-945e-94009ee52b3b_isrForCrosstalkSources_855450_32.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54dbef74b877970e62c6e009a0954921 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/00884cfb-f47c-4adb-a764-69569df32c9c_isrForCrosstalkSources_855382_56.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/00884cfb-f47c-4adb-a764-69569df32c9c_isrForCrosstalkSources_855382_56.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e80d41d1023d5fe86ff68f74b40f433 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/72725a73-e6a9-4f9d-8915-899d94c00649_isrForCrosstalkSources_855387_61.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/72725a73-e6a9-4f9d-8915-899d94c00649_isrForCrosstalkSources_855387_61.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59349211e8f905c32d531ab94161c4e8 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3aa504f5-6954-4f3e-ae97-5a36f335097e_isrForCrosstalkSources_855407_16.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3aa504f5-6954-4f3e-ae97-5a36f335097e_isrForCrosstalkSources_855407_16.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60e761d03f73deff01e2ef5101d157fc with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/de1df16e-7580-404b-aba1-7cc2069d00f1_isrForCrosstalkSources_855416_56.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/de1df16e-7580-404b-aba1-7cc2069d00f1_isrForCrosstalkSources_855416_56.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e21e9a4a3556c84ee88ec78a478e821 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e8ae782f-351d-48fb-aa0d-5a6bafad3b63_isrForCrosstalkSources_855451_37.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e8ae782f-351d-48fb-aa0d-5a6bafad3b63_isrForCrosstalkSources_855451_37.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574c6494627f1ae8892c394ce13acf04 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6cf79e8c-ff5f-473f-ad0f-00d027cfc95e_isrForCrosstalkSources_855458_54.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6cf79e8c-ff5f-473f-ad0f-00d027cfc95e_isrForCrosstalkSources_855458_54.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70a731926700baa1e94306bfb8ab3de with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/da3e6e76-fdda-4547-87c1-d9001a475908_isrForCrosstalkSources_855446_58.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/da3e6e76-fdda-4547-87c1-d9001a475908_isrForCrosstalkSources_855446_58.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f4be53e7db643e1ae10452dd867866a with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/12e671f0-ea70-4f90-afb0-7a460381860c_isrForCrosstalkSources_855434_27.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/12e671f0-ea70-4f90-afb0-7a460381860c_isrForCrosstalkSources_855434_27.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7b0fe4d05f2b8f495f90ad8220d6e3 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/05ca043e-ca98-4d7c-9f3b-0eb65c9796dc_isrForCrosstalkSources_855388_26.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/05ca043e-ca98-4d7c-9f3b-0eb65c9796dc_isrForCrosstalkSources_855388_26.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c19bc66b4dc6c0802f87b79a8059dc5 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2cdf767f-a379-4ce7-9fa8-d1284f449b90_isrForCrosstalkSources_855414_33.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2cdf767f-a379-4ce7-9fa8-d1284f449b90_isrForCrosstalkSources_855414_33.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cac43c6c0ddb18e8139f454e5356c63 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2520b55b-5326-471e-a882-cdd2eeea49d0_isrForCrosstalkSources_855422_13.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2520b55b-5326-471e-a882-cdd2eeea49d0_isrForCrosstalkSources_855422_13.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c14b7962587db55cfa7c848b729b79c with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3f0bb869-b42e-48ac-bf38-a1e079813d89_isrForCrosstalkSources_855407_45.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3f0bb869-b42e-48ac-bf38-a1e079813d89_isrForCrosstalkSources_855407_45.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a49da6315192775ad99d8468e52c943 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bdb11fcb-1aca-432e-9401-b4dbea4fe933_isrForCrosstalkSources_855426_39.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bdb11fcb-1aca-432e-9401-b4dbea4fe933_isrForCrosstalkSources_855426_39.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eef7980a82e47bcb509160154b8727b with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e38c929a-fcef-4f87-aee8-2440a92d0636_isrForCrosstalkSources_855390_51.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e38c929a-fcef-4f87-aee8-2440a92d0636_isrForCrosstalkSources_855390_51.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 609e46bacb71111b9e3e20fc16c63a56 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/a436dd41-b62b-49e8-a549-035f274b3365_isrForCrosstalkSources_855402_51.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/a436dd41-b62b-49e8-a549-035f274b3365_isrForCrosstalkSources_855402_51.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceac414a4e8524609238a634c79f6ccc with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a206943f-07aa-44e3-86b8-6cf3f7805baf_isrForCrosstalkSources_855396_39.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a206943f-07aa-44e3-86b8-6cf3f7805baf_isrForCrosstalkSources_855396_39.stderr parsl.dataflow.dflow INFO: Parsl task 2178 try 0 launched on executor multi with executor id 2179 parsl.dataflow.dflow INFO: Standard output for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/62584125-5889-46e3-a278-1ed1ef0e010e_isrForCrosstalkSources_855427_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/62584125-5889-46e3-a278-1ed1ef0e010e_isrForCrosstalkSources_855427_31.stderr parsl.dataflow.dflow DEBUG: Task 2179 will be sent to executor multi 2024-03-09 14:51:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964125 proc_lsst.multi INFO: [multi] job_id 16964125 2024-03-09 14:51:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-09 14:51:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964125 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964125 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2179 has memoization hash 5cd2f3b3b4e688a477718faf8ebde384 parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 16964125 parsl.dataflow.memoization INFO: Task 2179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-09 14:51:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-09 14:51:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-09 14:51:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e0b71a5ef0e33261972d93b475c30b1 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3a1fb120-6d62-48e1-89ba-0dc3519bbbdb_isrForCrosstalkSources_855381_23.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3a1fb120-6d62-48e1-89ba-0dc3519bbbdb_isrForCrosstalkSources_855381_23.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc354ca22f8c32ff533c813a99f18c48 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f45ff313-5be1-4231-864f-78306c92f152_isrForCrosstalkSources_855453_12.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f45ff313-5be1-4231-864f-78306c92f152_isrForCrosstalkSources_855453_12.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50066869a46f4748424864434f68713a with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d8a47d61-5578-47e3-902b-029fdfd14362_isrForCrosstalkSources_855383_11.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d8a47d61-5578-47e3-902b-029fdfd14362_isrForCrosstalkSources_855383_11.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbf7715212f53535dbbbe02930e81643 with result from task 114 parsl.dataflow.dflow INFO: Parsl task 2179 try 0 launched on executor multi with executor id 2180 parsl.dataflow.dflow INFO: Standard output for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/d69df90c-03ae-464b-b9a7-55862d1b05b7_isrForCrosstalkSources_855458_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/d69df90c-03ae-464b-b9a7-55862d1b05b7_isrForCrosstalkSources_855458_20.stderr parsl.dataflow.dflow DEBUG: Task 2180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2180 has memoization hash 9d9e674d6de3c879085d960e380003c0 parsl.dataflow.memoization INFO: Task 2180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6677ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a20e8c6d-5502-4bc4-a979-dd1b447d24b9_isrForCrosstalkSources_855435_12.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a20e8c6d-5502-4bc4-a979-dd1b447d24b9_isrForCrosstalkSources_855435_12.stderr parsl.dataflow.dflow INFO: Parsl task 2180 try 0 launched on executor multi with executor id 2181 parsl.dataflow.dflow INFO: Standard output for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea4989c1-6813-465a-a583-69eb525c053c_isrForCrosstalkSources_855390_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea4989c1-6813-465a-a583-69eb525c053c_isrForCrosstalkSources_855390_21.stderr parsl.dataflow.dflow DEBUG: Task 2181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2181 has memoization hash 6a6a2afc9ba866b2659f251c17086a4a parsl.dataflow.memoization INFO: Task 2181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2181 try 0 launched on executor multi with executor id 2182 parsl.dataflow.dflow INFO: Standard output for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ad4a6dbc-d0c7-4832-ad45-e7ce3a58f69b_isrForCrosstalkSources_855441_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ad4a6dbc-d0c7-4832-ad45-e7ce3a58f69b_isrForCrosstalkSources_855441_61.stderr parsl.dataflow.dflow DEBUG: Task 2182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2182 has memoization hash 8bfe460e55c3303ae51e8b11a9d049a9 parsl.dataflow.memoization INFO: Task 2182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2182 try 0 launched on executor multi with executor id 2183 parsl.dataflow.dflow INFO: Standard output for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/33e42e57-9065-446e-801b-58681d3594df_isrForCrosstalkSources_855433_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/33e42e57-9065-446e-801b-58681d3594df_isrForCrosstalkSources_855433_6.stderr parsl.dataflow.dflow DEBUG: Task 2183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2183 has memoization hash d69ec9b8a1a7219f4ab33cc74389408c parsl.dataflow.memoization INFO: Task 2183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964126 proc_lsst.multi INFO: [multi] job_id 16964126 2024-03-09 14:51:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-09 14:51:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964126 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964126 parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 16964126 parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-09 14:51:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b9faf945d4628db21da8151a1f6a29 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f23539ef-f7bb-499a-a675-717f48a6b68a_isrForCrosstalkSources_855436_31.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f23539ef-f7bb-499a-a675-717f48a6b68a_isrForCrosstalkSources_855436_31.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4c94a823aa4ee3034c7041d6b558ad8 with result from task 115 parsl.dataflow.dflow INFO: Parsl task 2183 try 0 launched on executor multi with executor id 2184 parsl.dataflow.dflow INFO: Standard output for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c43c8c7f-7171-4f39-833f-fd909359be6e_isrForCrosstalkSources_855419_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c43c8c7f-7171-4f39-833f-fd909359be6e_isrForCrosstalkSources_855419_35.stderr parsl.dataflow.dflow DEBUG: Task 2184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2184 has memoization hash a66a4a17c3fdb304fe4bddc8f02d6ead parsl.dataflow.memoization INFO: Task 2184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d9da83-fa3c-4bc5-b63a-033c0b1f24bc_isrForCrosstalkSources_855426_48.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d9da83-fa3c-4bc5-b63a-033c0b1f24bc_isrForCrosstalkSources_855426_48.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1379f3cd1f83c55e1153701b3a8b9d13 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/90342eb7-944d-4590-a07f-05f38937d614_isrForCrosstalkSources_855453_29.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/90342eb7-944d-4590-a07f-05f38937d614_isrForCrosstalkSources_855453_29.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50cb983923d2db41c1183758d5c9493a with result from task 110 parsl.dataflow.dflow INFO: Parsl task 2184 try 0 launched on executor multi with executor id 2185 parsl.dataflow.dflow INFO: Standard output for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7c1603b7-6007-48a5-b2a5-a7b85c47f321_isrForCrosstalkSources_855443_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7c1603b7-6007-48a5-b2a5-a7b85c47f321_isrForCrosstalkSources_855443_33.stderr parsl.dataflow.dflow DEBUG: Task 2185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2185 has memoization hash 68f9d88a3cc5441f08d66e418c9b979b parsl.dataflow.memoization INFO: Task 2185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/678ae8e1-b153-4867-b973-c2c5f1ad3fa5_isrForCrosstalkSources_855408_30.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/678ae8e1-b153-4867-b973-c2c5f1ad3fa5_isrForCrosstalkSources_855408_30.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc73d24009ab5a2efe89f43d4c03ec35 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ee457bf2-8e07-494a-ba70-93c9da0cb863_isrForCrosstalkSources_855440_29.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ee457bf2-8e07-494a-ba70-93c9da0cb863_isrForCrosstalkSources_855440_29.stderr parsl.dataflow.dflow INFO: Parsl task 2185 try 0 launched on executor multi with executor id 2186 parsl.dataflow.dflow INFO: Standard output for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/780227a6-056c-421c-ad36-1389ac85a770_isrForCrosstalkSources_855386_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/780227a6-056c-421c-ad36-1389ac85a770_isrForCrosstalkSources_855386_46.stderr parsl.dataflow.dflow DEBUG: Task 2186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2186 has memoization hash 3c874ea4b3d30aa97b7ff2e5ba9f8e56 parsl.dataflow.memoization INFO: Task 2186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2186 try 0 launched on executor multi with executor id 2187 parsl.dataflow.dflow INFO: Standard output for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b2abcb4b-bc18-4304-aae0-8da58382202d_isrForCrosstalkSources_855428_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b2abcb4b-bc18-4304-aae0-8da58382202d_isrForCrosstalkSources_855428_31.stderr parsl.dataflow.dflow DEBUG: Task 2187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2187 has memoization hash 9d22b213a38abdaa604c0d45c44767a4 parsl.dataflow.memoization INFO: Task 2187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2187 try 0 launched on executor multi with executor id 2188 parsl.dataflow.dflow INFO: Standard output for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6fdc2b77-532c-498f-b8ea-49ef8510ebe7_isrForCrosstalkSources_855380_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6fdc2b77-532c-498f-b8ea-49ef8510ebe7_isrForCrosstalkSources_855380_19.stderr parsl.dataflow.dflow DEBUG: Task 2188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2188 has memoization hash 74b20abb8e89ccb2afb9349f11385fac parsl.dataflow.memoization INFO: Task 2188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c50bcc00cc702ee763917221cec9aea6 with result from task 144 parsl.dataflow.dflow INFO: Parsl task 2188 try 0 launched on executor multi with executor id 2189 parsl.dataflow.dflow INFO: Standard output for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a16b9cde-afa6-411f-88c2-b110e61b2f73_isrForCrosstalkSources_855380_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a16b9cde-afa6-411f-88c2-b110e61b2f73_isrForCrosstalkSources_855380_47.stderr parsl.dataflow.dflow DEBUG: Task 2189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2189 has memoization hash 7e978e90bc28663eb31bbae8b3a003da parsl.dataflow.memoization INFO: Task 2189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/75fe732a-57fb-4594-9c77-863ba215d6f6_isrForCrosstalkSources_855399_22.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/75fe732a-57fb-4594-9c77-863ba215d6f6_isrForCrosstalkSources_855399_22.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2c133d699617b1d79cc360d330d563 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/8e3fa845-c8a1-409d-865a-4bc7c7cd7e5b_isrForCrosstalkSources_855390_23.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/8e3fa845-c8a1-409d-865a-4bc7c7cd7e5b_isrForCrosstalkSources_855390_23.stderr parsl.dataflow.dflow INFO: Parsl task 2189 try 0 launched on executor multi with executor id 2190 parsl.dataflow.dflow INFO: Standard output for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c9e0bf92-bae3-4cb0-8053-b1af1d98d2e3_isrForCrosstalkSources_855450_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c9e0bf92-bae3-4cb0-8053-b1af1d98d2e3_isrForCrosstalkSources_855450_56.stderr parsl.dataflow.dflow DEBUG: Task 2190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2190 has memoization hash 6be75777e8aad9e2e43867377cb1b79b parsl.dataflow.memoization INFO: Task 2190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2190 try 0 launched on executor multi with executor id 2191 parsl.dataflow.dflow INFO: Standard output for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1caaf1cc-679e-436a-95c1-322b4fb09d48_isrForCrosstalkSources_855454_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1caaf1cc-679e-436a-95c1-322b4fb09d48_isrForCrosstalkSources_855454_26.stderr parsl.dataflow.dflow DEBUG: Task 2191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2191 has memoization hash b9ce762825b0d1a08c4507efafbdc8fd parsl.dataflow.memoization INFO: Task 2191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964127 proc_lsst.multi INFO: [multi] job_id 16964127 2024-03-09 14:51:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-09 14:51:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964127 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964127 parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 16964127 parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-09 14:51:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 820243d04d1c0c510ecbdb0bd4c6a73e with result from task 111 parsl.dataflow.dflow INFO: Parsl task 2191 try 0 launched on executor multi with executor id 2192 parsl.dataflow.dflow INFO: Standard output for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/091cb321-d3ff-437b-9ff7-b375342a1cc6_isrForCrosstalkSources_855411_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/091cb321-d3ff-437b-9ff7-b375342a1cc6_isrForCrosstalkSources_855411_45.stderr parsl.dataflow.dflow DEBUG: Task 2192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2192 has memoization hash 5d77c3c7a08afd7aefa0147c57925812 parsl.dataflow.memoization INFO: Task 2192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/97c76f00-e7e6-450b-911e-2b1f96995a7b_isrForCrosstalkSources_855392_44.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/97c76f00-e7e6-450b-911e-2b1f96995a7b_isrForCrosstalkSources_855392_44.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28d0f7322c36f92f9ae931298cb282f9 with result from task 113 parsl.dataflow.dflow INFO: Parsl task 2192 try 0 launched on executor multi with executor id 2193 parsl.dataflow.dflow INFO: Standard output for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/79cfa3ab-b197-4a11-a294-3aac68736802_isrForCrosstalkSources_855393_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/79cfa3ab-b197-4a11-a294-3aac68736802_isrForCrosstalkSources_855393_61.stderr parsl.dataflow.dflow DEBUG: Task 2193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2193 has memoization hash 5fe08f8bfcbfd17c36e360f64fa2396a parsl.dataflow.memoization INFO: Task 2193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2193 try 0 launched on executor multi with executor id 2194 parsl.dataflow.dflow INFO: Standard output for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c947b04e-baef-4c96-bd90-968558c853c7_isrForCrosstalkSources_855393_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c947b04e-baef-4c96-bd90-968558c853c7_isrForCrosstalkSources_855393_15.stderr parsl.dataflow.dflow DEBUG: Task 2194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2194 has memoization hash 413de9da3fd15d417341481f3a50e678 parsl.dataflow.memoization INFO: Task 2194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/64182dd3-9340-4086-8242-0bf78e117eb4_isrForCrosstalkSources_855392_53.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/64182dd3-9340-4086-8242-0bf78e117eb4_isrForCrosstalkSources_855392_53.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 888a04031b8b481f0da47d89c17c4fe5 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2347db10-d335-4336-b6ce-3f552480e9b6_isrForCrosstalkSources_855429_50.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2347db10-d335-4336-b6ce-3f552480e9b6_isrForCrosstalkSources_855429_50.stderr parsl.dataflow.dflow INFO: Parsl task 2194 try 0 launched on executor multi with executor id 2195 parsl.dataflow.dflow INFO: Standard output for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e87ffe98-44c8-4e9a-b14c-eed52eec43a3_isrForCrosstalkSources_855424_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e87ffe98-44c8-4e9a-b14c-eed52eec43a3_isrForCrosstalkSources_855424_59.stderr parsl.dataflow.dflow DEBUG: Task 2195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2195 has memoization hash e7fd1eba183f3c143a0ec60a106c0d4b parsl.dataflow.memoization INFO: Task 2195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2195 try 0 launched on executor multi with executor id 2196 parsl.dataflow.dflow INFO: Standard output for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c545b8da-4fa6-420b-b196-d701340da825_isrForCrosstalkSources_855403_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c545b8da-4fa6-420b-b196-d701340da825_isrForCrosstalkSources_855403_9.stderr parsl.dataflow.dflow DEBUG: Task 2196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2196 has memoization hash 72c94013f5cfb1d4a25f735008e1835b parsl.dataflow.memoization INFO: Task 2196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2196 try 0 launched on executor multi with executor id 2197 parsl.dataflow.dflow INFO: Standard output for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f37a00a2-441d-43a2-b5c8-879d48a91b6d_isrForCrosstalkSources_855428_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f37a00a2-441d-43a2-b5c8-879d48a91b6d_isrForCrosstalkSources_855428_40.stderr parsl.dataflow.dflow DEBUG: Task 2197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2197 has memoization hash 3cb72c99053812d12d46f9b38f2985e5 parsl.dataflow.memoization INFO: Task 2197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2197 try 0 launched on executor multi with executor id 2198 parsl.dataflow.dflow INFO: Standard output for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/ed191250-5e4b-407f-95c4-df8bf3d6348a_isrForCrosstalkSources_855408_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/ed191250-5e4b-407f-95c4-df8bf3d6348a_isrForCrosstalkSources_855408_61.stderr parsl.dataflow.dflow DEBUG: Task 2198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2198 has memoization hash 17eb2ffa1497443cc72dc59cdedb78e6 parsl.dataflow.memoization INFO: Task 2198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2198 try 0 launched on executor multi with executor id 2199 parsl.dataflow.dflow INFO: Standard output for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df101f5c-5cb9-4676-ac98-b695c27104d5_isrForCrosstalkSources_855422_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df101f5c-5cb9-4676-ac98-b695c27104d5_isrForCrosstalkSources_855422_16.stderr parsl.dataflow.dflow DEBUG: Task 2199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2199 has memoization hash 1b2c01d48f28297f60c8b0b0bd5bdb04 parsl.dataflow.memoization INFO: Task 2199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65232e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2199 try 0 launched on executor multi with executor id 2200 parsl.dataflow.dflow INFO: Standard output for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/293b0fea-4bae-4aa1-b059-78b0e67c63b0_isrForCrosstalkSources_855453_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/293b0fea-4bae-4aa1-b059-78b0e67c63b0_isrForCrosstalkSources_855453_48.stderr parsl.dataflow.dflow DEBUG: Task 2200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2200 has memoization hash c2ce1506be07fbb3f4a8085c796e6951 parsl.dataflow.memoization INFO: Task 2200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b7d150660f28eceda89a0dea98a3d8b with result from task 116 parsl.dataflow.dflow INFO: Parsl task 2200 try 0 launched on executor multi with executor id 2201 parsl.dataflow.dflow INFO: Standard output for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/149aa58c-c0bd-47d8-ad0c-dbbc6c317a14_isrForCrosstalkSources_855406_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/149aa58c-c0bd-47d8-ad0c-dbbc6c317a14_isrForCrosstalkSources_855406_20.stderr parsl.dataflow.dflow DEBUG: Task 2201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2201 has memoization hash 2253e63d276476b51d698a7cf395a344 parsl.dataflow.memoization INFO: Task 2201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65237e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b5abb156-f24a-4a91-9ada-c25403942e58_isrForCrosstalkSources_855397_30.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b5abb156-f24a-4a91-9ada-c25403942e58_isrForCrosstalkSources_855397_30.stderr parsl.dataflow.dflow INFO: Parsl task 2201 try 0 launched on executor multi with executor id 2202 parsl.dataflow.dflow INFO: Standard output for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7339f5af-9527-44c9-ace4-a34863b3927a_isrForCrosstalkSources_855386_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7339f5af-9527-44c9-ace4-a34863b3927a_isrForCrosstalkSources_855386_42.stderr parsl.dataflow.dflow DEBUG: Task 2202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2202 has memoization hash b46b2fa38689b5a61f2bc294ce502cf2 parsl.dataflow.memoization INFO: Task 2202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2202 try 0 launched on executor multi with executor id 2203 parsl.dataflow.dflow INFO: Standard output for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/38e44ff3-3552-458d-be70-e71f713d0c6c_isrForCrosstalkSources_855403_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/38e44ff3-3552-458d-be70-e71f713d0c6c_isrForCrosstalkSources_855403_45.stderr parsl.dataflow.dflow DEBUG: Task 2203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2203 has memoization hash 5727bbf580fe971581f2230b923c28c1 parsl.dataflow.memoization INFO: Task 2203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2203 try 0 launched on executor multi with executor id 2204 parsl.dataflow.dflow INFO: Standard output for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/675c34f2-491b-4199-9daf-3866758bac0c_isrForCrosstalkSources_855409_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/675c34f2-491b-4199-9daf-3866758bac0c_isrForCrosstalkSources_855409_53.stderr parsl.dataflow.dflow DEBUG: Task 2204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2204 has memoization hash f045f8ac7a41ddb8bb79d81e876ede50 parsl.dataflow.memoization INFO: Task 2204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2204 try 0 launched on executor multi with executor id 2205 parsl.dataflow.dflow INFO: Standard output for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3b453b68-9046-41a3-b49c-eabc3e4babcc_isrForCrosstalkSources_855407_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3b453b68-9046-41a3-b49c-eabc3e4babcc_isrForCrosstalkSources_855407_31.stderr parsl.dataflow.dflow DEBUG: Task 2205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2205 has memoization hash ed4187d3f05e14af68f5fa0fc8b916c4 parsl.dataflow.memoization INFO: Task 2205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2205 try 0 launched on executor multi with executor id 2206 parsl.dataflow.dflow INFO: Standard output for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6eb4bf60-5ec7-4c6d-97df-67a5b7e85573_isrForCrosstalkSources_855456_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6eb4bf60-5ec7-4c6d-97df-67a5b7e85573_isrForCrosstalkSources_855456_16.stderr parsl.dataflow.dflow DEBUG: Task 2206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2206 has memoization hash d631dfe9e9677ee20b45484dcc23ee66 parsl.dataflow.memoization INFO: Task 2206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9f45d61a0f369d7d55e34eb540f1e5d with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed5b9281-bf42-4ea0-824f-b8bafde70dbc_isrForCrosstalkSources_855427_50.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed5b9281-bf42-4ea0-824f-b8bafde70dbc_isrForCrosstalkSources_855427_50.stderr parsl.dataflow.dflow INFO: Parsl task 2206 try 0 launched on executor multi with executor id 2207 parsl.dataflow.dflow INFO: Standard output for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/478a21d7-3b95-40c6-aaf5-7cf86cc932b9_isrForCrosstalkSources_855422_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/478a21d7-3b95-40c6-aaf5-7cf86cc932b9_isrForCrosstalkSources_855422_36.stderr parsl.dataflow.dflow DEBUG: Task 2207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2207 has memoization hash de441fa0be02154e399eece55507e3bc parsl.dataflow.memoization INFO: Task 2207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2207 try 0 launched on executor multi with executor id 2208 parsl.dataflow.dflow INFO: Standard output for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/36b8c5d7-2d0f-4795-85d5-cdc9842d7c55_isrForCrosstalkSources_855434_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/36b8c5d7-2d0f-4795-85d5-cdc9842d7c55_isrForCrosstalkSources_855434_52.stderr parsl.dataflow.dflow DEBUG: Task 2208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2208 has memoization hash c058b9dde0b487b88dd3ad1c6a9afc82 parsl.dataflow.memoization INFO: Task 2208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85dd6c62f60e209e8642fdc8662b5cfd with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ded99ff1-adf4-4ae1-bf63-0a51ad24abd8_isrForCrosstalkSources_855424_29.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ded99ff1-adf4-4ae1-bf63-0a51ad24abd8_isrForCrosstalkSources_855424_29.stderr parsl.dataflow.dflow INFO: Parsl task 2208 try 0 launched on executor multi with executor id 2209 parsl.dataflow.dflow INFO: Standard output for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/77fe158a-e3c4-483f-b4b0-3ba282f9f704_isrForCrosstalkSources_855441_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/77fe158a-e3c4-483f-b4b0-3ba282f9f704_isrForCrosstalkSources_855441_18.stderr parsl.dataflow.dflow DEBUG: Task 2209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2209 has memoization hash c862f54a86002b575ebc288c76955698 parsl.dataflow.memoization INFO: Task 2209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964128 proc_lsst.multi INFO: [multi] job_id 16964128 2024-03-09 14:51:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-09 14:51:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964128 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964128 parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 16964128 parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-09 14:51:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-09 14:51:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2209 try 0 launched on executor multi with executor id 2210 parsl.dataflow.dflow INFO: Standard output for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f3c5ebaf-535f-464d-89fd-db04f3be8677_isrForCrosstalkSources_855392_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f3c5ebaf-535f-464d-89fd-db04f3be8677_isrForCrosstalkSources_855392_22.stderr parsl.dataflow.dflow DEBUG: Task 2210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2210 has memoization hash 9eadb57ae329d64cbfac486d92abec3d parsl.dataflow.memoization INFO: Task 2210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2210 try 0 launched on executor multi with executor id 2211 parsl.dataflow.dflow INFO: Standard output for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5074cfc3-f6c3-44f9-a655-af1fab12913f_isrForCrosstalkSources_855384_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5074cfc3-f6c3-44f9-a655-af1fab12913f_isrForCrosstalkSources_855384_24.stderr parsl.dataflow.dflow DEBUG: Task 2211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2211 has memoization hash 5de2bcdcc2d95d1194670926777235ea parsl.dataflow.memoization INFO: Task 2211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2211 try 0 launched on executor multi with executor id 2212 parsl.dataflow.dflow INFO: Standard output for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec521dd7-6b2d-4f01-b767-be95ff5ec0dc_isrForCrosstalkSources_855413_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec521dd7-6b2d-4f01-b767-be95ff5ec0dc_isrForCrosstalkSources_855413_5.stderr parsl.dataflow.dflow DEBUG: Task 2212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2212 has memoization hash 60f52c6d1ea8789c8baa825dca2fe803 parsl.dataflow.memoization INFO: Task 2212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2212 try 0 launched on executor multi with executor id 2213 parsl.dataflow.dflow INFO: Standard output for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/320e01f0-d24c-4dff-8c94-c6f08c5cc92d_isrForCrosstalkSources_855459_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/320e01f0-d24c-4dff-8c94-c6f08c5cc92d_isrForCrosstalkSources_855459_54.stderr parsl.dataflow.dflow DEBUG: Task 2213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2213 has memoization hash 34e60009b39e1c2685c725e032b54245 parsl.dataflow.memoization INFO: Task 2213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2213 try 0 launched on executor multi with executor id 2214 parsl.dataflow.dflow INFO: Standard output for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5eff4b7f-bc69-4a32-9cef-71fd542d678e_isrForCrosstalkSources_855449_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5eff4b7f-bc69-4a32-9cef-71fd542d678e_isrForCrosstalkSources_855449_28.stderr parsl.dataflow.dflow DEBUG: Task 2214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2214 has memoization hash ba0522cfd2537064e80dc5ed90213070 parsl.dataflow.memoization INFO: Task 2214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2214 try 0 launched on executor multi with executor id 2215 parsl.dataflow.dflow INFO: Standard output for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/65fd9145-3a70-407f-907b-6a10d020b524_isrForCrosstalkSources_855425_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/65fd9145-3a70-407f-907b-6a10d020b524_isrForCrosstalkSources_855425_55.stderr parsl.dataflow.dflow DEBUG: Task 2215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2215 has memoization hash f505fd64a30645e4ce120e978ef57803 parsl.dataflow.memoization INFO: Task 2215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2215 try 0 launched on executor multi with executor id 2216 parsl.dataflow.dflow INFO: Standard output for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b0ec1c55-4019-435e-92af-d59dbf589e1a_isrForCrosstalkSources_855439_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b0ec1c55-4019-435e-92af-d59dbf589e1a_isrForCrosstalkSources_855439_8.stderr parsl.dataflow.dflow DEBUG: Task 2216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2216 has memoization hash 5d3b06c7565a77f82de68e6378dced69 parsl.dataflow.memoization INFO: Task 2216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2216 try 0 launched on executor multi with executor id 2217 parsl.dataflow.dflow INFO: Standard output for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eca3d4c1-1dc4-44c4-b08b-e958a70392dd_isrForCrosstalkSources_855381_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eca3d4c1-1dc4-44c4-b08b-e958a70392dd_isrForCrosstalkSources_855381_26.stderr parsl.dataflow.dflow DEBUG: Task 2217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2217 has memoization hash 7d87a9934a79539475ffa4e8e144b906 parsl.dataflow.memoization INFO: Task 2217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2217 try 0 launched on executor multi with executor id 2218 parsl.dataflow.dflow INFO: Standard output for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/40e4df54-4f6a-4c92-b66e-c7220300d49c_isrForCrosstalkSources_855446_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/40e4df54-4f6a-4c92-b66e-c7220300d49c_isrForCrosstalkSources_855446_4.stderr parsl.dataflow.dflow DEBUG: Task 2218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2218 has memoization hash 56b8e0496e4dcd6fd7091a1a4813e485 parsl.dataflow.memoization INFO: Task 2218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6521260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2218 try 0 launched on executor multi with executor id 2219 parsl.dataflow.dflow INFO: Standard output for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/10e42b24-e996-4756-9d65-f35469fafe8a_isrForCrosstalkSources_855387_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/10e42b24-e996-4756-9d65-f35469fafe8a_isrForCrosstalkSources_855387_8.stderr parsl.dataflow.dflow DEBUG: Task 2219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2219 has memoization hash 9190bec2fffb87c166a25f6e1b957f39 parsl.dataflow.memoization INFO: Task 2219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2219 try 0 launched on executor multi with executor id 2220 parsl.dataflow.dflow INFO: Standard output for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d093dd3-3abd-44b4-b2cf-90bfbc49939d_isrForCrosstalkSources_855388_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d093dd3-3abd-44b4-b2cf-90bfbc49939d_isrForCrosstalkSources_855388_43.stderr parsl.dataflow.dflow DEBUG: Task 2220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2220 has memoization hash 0bb2c30c76dfb584b066ee3f7b594014 parsl.dataflow.memoization INFO: Task 2220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2220 try 0 launched on executor multi with executor id 2221 parsl.dataflow.dflow INFO: Standard output for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/70f1ceec-460e-44e4-bed6-9183f023120d_isrForCrosstalkSources_855430_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/70f1ceec-460e-44e4-bed6-9183f023120d_isrForCrosstalkSources_855430_44.stderr parsl.dataflow.dflow DEBUG: Task 2221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2221 has memoization hash 0d42cd4fc8650b13a62dc1b55c13b1c4 parsl.dataflow.memoization INFO: Task 2221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6523600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2221 try 0 launched on executor multi with executor id 2222 parsl.dataflow.dflow INFO: Standard output for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/38cfd2a8-e61e-470f-96a8-e67116c300c8_isrForCrosstalkSources_855402_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/38cfd2a8-e61e-470f-96a8-e67116c300c8_isrForCrosstalkSources_855402_35.stderr parsl.dataflow.dflow DEBUG: Task 2222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2222 has memoization hash 55790f6c5437482f591a9debe43f6fe5 parsl.dataflow.memoization INFO: Task 2222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2222 try 0 launched on executor multi with executor id 2223 parsl.dataflow.dflow INFO: Standard output for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/c72df4ec-5657-44dd-ad1a-2c3ee302924e_isrForCrosstalkSources_855447_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/c72df4ec-5657-44dd-ad1a-2c3ee302924e_isrForCrosstalkSources_855447_52.stderr parsl.dataflow.dflow DEBUG: Task 2223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2223 has memoization hash b1da6d0df94b7c6792ff0e4eafda57e1 parsl.dataflow.memoization INFO: Task 2223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2223 try 0 launched on executor multi with executor id 2224 parsl.dataflow.dflow INFO: Standard output for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c3b8ca0d-a181-4c20-8181-4dda739e5841_isrForCrosstalkSources_855421_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c3b8ca0d-a181-4c20-8181-4dda739e5841_isrForCrosstalkSources_855421_30.stderr parsl.dataflow.dflow DEBUG: Task 2224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2224 has memoization hash ba89465c459049b8b04b595aae9abe01 parsl.dataflow.memoization INFO: Task 2224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2224 try 0 launched on executor multi with executor id 2225 parsl.dataflow.dflow INFO: Standard output for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91f50958-895d-4517-abda-80e39161909f_isrForCrosstalkSources_855413_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91f50958-895d-4517-abda-80e39161909f_isrForCrosstalkSources_855413_47.stderr parsl.dataflow.dflow DEBUG: Task 2225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2225 has memoization hash 2b554a77f01e30a691456eb64e9adb73 parsl.dataflow.memoization INFO: Task 2225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43586e16767f1a54bcddfda0f37bd3a2 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d092a12d-eb32-4f69-a2f0-86cfd7f687c2_isrForCrosstalkSources_855386_22.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d092a12d-eb32-4f69-a2f0-86cfd7f687c2_isrForCrosstalkSources_855386_22.stderr parsl.dataflow.dflow INFO: Parsl task 2225 try 0 launched on executor multi with executor id 2226 parsl.dataflow.dflow INFO: Standard output for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/91e08b9e-057b-4707-b392-b2334920011f_isrForCrosstalkSources_855430_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/91e08b9e-057b-4707-b392-b2334920011f_isrForCrosstalkSources_855430_9.stderr parsl.dataflow.dflow DEBUG: Task 2226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2226 has memoization hash 4370c344b263ee40cee256c0aea75320 parsl.dataflow.memoization INFO: Task 2226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2226 try 0 launched on executor multi with executor id 2227 parsl.dataflow.dflow INFO: Standard output for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d7299c2c-5faa-4e9b-a670-62aa2bfe4475_isrForCrosstalkSources_855389_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d7299c2c-5faa-4e9b-a670-62aa2bfe4475_isrForCrosstalkSources_855389_18.stderr parsl.dataflow.dflow DEBUG: Task 2227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2227 has memoization hash 23805fe21f975f8221486f5f24d61292 parsl.dataflow.memoization INFO: Task 2227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 597cbe14f2e5c9d3dd335470d92e5738 with result from task 93 parsl.dataflow.dflow INFO: Parsl task 2227 try 0 launched on executor multi with executor id 2228 parsl.dataflow.dflow INFO: Standard output for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7b032554-e377-4419-bbed-ca13b7c4989c_isrForCrosstalkSources_855445_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7b032554-e377-4419-bbed-ca13b7c4989c_isrForCrosstalkSources_855445_39.stderr parsl.dataflow.dflow DEBUG: Task 2228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2228 has memoization hash 18c1bbf932f4348438fcba7816354f84 parsl.dataflow.memoization INFO: Task 2228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6d3e9d06-6f73-4f12-bb2d-3c1486d789f0_isrForCrosstalkSources_855410_48.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6d3e9d06-6f73-4f12-bb2d-3c1486d789f0_isrForCrosstalkSources_855410_48.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fec3e05117d8a0eec97b80f9d6b3283a with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b3a156e5-db54-470c-9f6e-20c7c624ca38_isrForCrosstalkSources_855415_54.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b3a156e5-db54-470c-9f6e-20c7c624ca38_isrForCrosstalkSources_855415_54.stderr parsl.dataflow.dflow INFO: Parsl task 2228 try 0 launched on executor multi with executor id 2229 parsl.dataflow.dflow INFO: Standard output for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/82122e32-1885-4816-8034-4b18414578b8_isrForCrosstalkSources_855416_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/82122e32-1885-4816-8034-4b18414578b8_isrForCrosstalkSources_855416_8.stderr parsl.dataflow.dflow DEBUG: Task 2229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2229 has memoization hash e4eb912c1cca531813a347ebb58999a2 parsl.dataflow.memoization INFO: Task 2229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2229 try 0 launched on executor multi with executor id 2230 parsl.dataflow.dflow INFO: Standard output for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d0ffd2cf-6baa-411a-8373-f6ed9c2a3088_isrForCrosstalkSources_855433_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d0ffd2cf-6baa-411a-8373-f6ed9c2a3088_isrForCrosstalkSources_855433_32.stderr parsl.dataflow.dflow DEBUG: Task 2230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2230 has memoization hash b25ae3f106bfc144d7f39fa85afd72ff parsl.dataflow.memoization INFO: Task 2230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2230 try 0 launched on executor multi with executor id 2231 parsl.dataflow.dflow INFO: Standard output for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/013f87dc-f15c-4658-b4e2-00a692dc86d3_isrForCrosstalkSources_855385_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/013f87dc-f15c-4658-b4e2-00a692dc86d3_isrForCrosstalkSources_855385_53.stderr parsl.dataflow.dflow DEBUG: Task 2231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2231 has memoization hash b01bb69c46300cd7f201c6658b666870 parsl.dataflow.memoization INFO: Task 2231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2231 try 0 launched on executor multi with executor id 2232 parsl.dataflow.dflow INFO: Standard output for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72d448e7-6db5-4a6b-8016-9f8e7324e621_isrForCrosstalkSources_855437_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72d448e7-6db5-4a6b-8016-9f8e7324e621_isrForCrosstalkSources_855437_15.stderr parsl.dataflow.dflow DEBUG: Task 2232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2232 has memoization hash 6c8a20548767274abe864c082c95dcea parsl.dataflow.memoization INFO: Task 2232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964129 proc_lsst.multi INFO: [multi] job_id 16964129 2024-03-09 14:51:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-09 14:51:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964129 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964129 parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 16964129 parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-09 14:51:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2232 try 0 launched on executor multi with executor id 2233 parsl.dataflow.dflow INFO: Standard output for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32f9d826-11b0-4989-9833-b4a738daed43_isrForCrosstalkSources_855380_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32f9d826-11b0-4989-9833-b4a738daed43_isrForCrosstalkSources_855380_60.stderr parsl.dataflow.dflow DEBUG: Task 2233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2233 has memoization hash 513e372b409bcb2f5fa9208f6e2c4695 parsl.dataflow.memoization INFO: Task 2233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2233 try 0 launched on executor multi with executor id 2234 parsl.dataflow.dflow INFO: Standard output for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6643d875-685d-455b-8a8e-7468dca90ea7_isrForCrosstalkSources_855404_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6643d875-685d-455b-8a8e-7468dca90ea7_isrForCrosstalkSources_855404_15.stderr parsl.dataflow.dflow DEBUG: Task 2234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2234 has memoization hash 7216daa81a1564c6757e9ee24796a3e9 parsl.dataflow.memoization INFO: Task 2234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2234 try 0 launched on executor multi with executor id 2235 parsl.dataflow.dflow INFO: Standard output for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ea36501b-ed1d-41a8-925d-7b151b99ac2a_isrForCrosstalkSources_855404_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ea36501b-ed1d-41a8-925d-7b151b99ac2a_isrForCrosstalkSources_855404_18.stderr parsl.dataflow.dflow DEBUG: Task 2235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2235 has memoization hash c502f1e016015e5a2ada163b34dd6114 parsl.dataflow.memoization INFO: Task 2235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2235 try 0 launched on executor multi with executor id 2236 parsl.dataflow.dflow INFO: Standard output for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8bbea5fa-a479-4df0-982f-662700304a06_isrForCrosstalkSources_855431_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8bbea5fa-a479-4df0-982f-662700304a06_isrForCrosstalkSources_855431_8.stderr parsl.dataflow.dflow DEBUG: Task 2236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2236 has memoization hash 09c3c5ae3bf8ab84624919c24906412f parsl.dataflow.memoization INFO: Task 2236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2236 try 0 launched on executor multi with executor id 2237 parsl.dataflow.dflow INFO: Standard output for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3b6206a4-c8f2-4398-9fb6-e9ec9ac29ce4_isrForCrosstalkSources_855405_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3b6206a4-c8f2-4398-9fb6-e9ec9ac29ce4_isrForCrosstalkSources_855405_36.stderr parsl.dataflow.dflow DEBUG: Task 2237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2237 has memoization hash 19ab40f91024f26e027a5fb893b19ce9 parsl.dataflow.memoization INFO: Task 2237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2237 try 0 launched on executor multi with executor id 2238 parsl.dataflow.dflow INFO: Standard output for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f490dff2-dcab-4246-b94c-0a96d9c0a74c_isrForCrosstalkSources_855451_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f490dff2-dcab-4246-b94c-0a96d9c0a74c_isrForCrosstalkSources_855451_62.stderr parsl.dataflow.dflow DEBUG: Task 2238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2238 has memoization hash 651be80f14b5afdf91442ba5070ab96f parsl.dataflow.memoization INFO: Task 2238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2238 try 0 launched on executor multi with executor id 2239 parsl.dataflow.dflow INFO: Standard output for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31d33b0c-edef-4299-86db-60e4cf5424ec_isrForCrosstalkSources_855409_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31d33b0c-edef-4299-86db-60e4cf5424ec_isrForCrosstalkSources_855409_31.stderr parsl.dataflow.dflow DEBUG: Task 2239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2239 has memoization hash 2946fb0fa322516c2aa3749fc02cb439 parsl.dataflow.memoization INFO: Task 2239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2239 try 0 launched on executor multi with executor id 2240 parsl.dataflow.dflow INFO: Standard output for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/69389098-2ebd-4047-b277-1a9bc2c1b778_isrForCrosstalkSources_855420_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/69389098-2ebd-4047-b277-1a9bc2c1b778_isrForCrosstalkSources_855420_19.stderr parsl.dataflow.dflow DEBUG: Task 2240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2240 has memoization hash 57cea68d927c7c6eedd81b96798139b2 parsl.dataflow.memoization INFO: Task 2240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2240 try 0 launched on executor multi with executor id 2241 parsl.dataflow.dflow INFO: Standard output for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/881f6011-e669-44f1-a996-5b8fea089224_isrForCrosstalkSources_855415_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/881f6011-e669-44f1-a996-5b8fea089224_isrForCrosstalkSources_855415_58.stderr parsl.dataflow.dflow DEBUG: Task 2241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2241 has memoization hash 12edc420053bc86d9acebc81d987fdf9 parsl.dataflow.memoization INFO: Task 2241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2241 try 0 launched on executor multi with executor id 2242 parsl.dataflow.dflow INFO: Standard output for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/15dce3a0-be8c-45ca-8828-715afcdd0b80_isrForCrosstalkSources_855424_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/15dce3a0-be8c-45ca-8828-715afcdd0b80_isrForCrosstalkSources_855424_18.stderr parsl.dataflow.dflow DEBUG: Task 2242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2242 has memoization hash 99aae585fdd2053702983883ac459039 parsl.dataflow.memoization INFO: Task 2242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2242 try 0 launched on executor multi with executor id 2243 parsl.dataflow.dflow INFO: Standard output for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81709fa0-8bc8-47a7-afbd-999d90dffc84_isrForCrosstalkSources_855443_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81709fa0-8bc8-47a7-afbd-999d90dffc84_isrForCrosstalkSources_855443_55.stderr parsl.dataflow.dflow DEBUG: Task 2243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2243 has memoization hash b67ef1f1a5e883744da137f7355f52c4 parsl.dataflow.memoization INFO: Task 2243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2243 try 0 launched on executor multi with executor id 2244 parsl.dataflow.dflow INFO: Standard output for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ced89f93-1660-446d-b3d1-d039048d1645_isrForCrosstalkSources_855457_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ced89f93-1660-446d-b3d1-d039048d1645_isrForCrosstalkSources_855457_60.stderr parsl.dataflow.dflow DEBUG: Task 2244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2244 has memoization hash 4536f2f3245c25a4d1cc9c05b3ed07a6 parsl.dataflow.memoization INFO: Task 2244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2244 try 0 launched on executor multi with executor id 2245 parsl.dataflow.dflow INFO: Standard output for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/367bb177-9512-434f-a7f2-d668daac1f5c_isrForCrosstalkSources_855428_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/367bb177-9512-434f-a7f2-d668daac1f5c_isrForCrosstalkSources_855428_28.stderr parsl.dataflow.dflow DEBUG: Task 2245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2245 has memoization hash cc0b0f4a6dac1abf0b29eaa475cceabe parsl.dataflow.memoization INFO: Task 2245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2245 try 0 launched on executor multi with executor id 2246 parsl.dataflow.dflow INFO: Standard output for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/00bfc4ad-2fe7-45cc-9085-f0be19f40b8f_isrForCrosstalkSources_855407_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/00bfc4ad-2fe7-45cc-9085-f0be19f40b8f_isrForCrosstalkSources_855407_9.stderr parsl.dataflow.dflow DEBUG: Task 2246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2246 has memoization hash 3ff798d7c32797f37e6af92bdce9c8d5 parsl.dataflow.memoization INFO: Task 2246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2246 try 0 launched on executor multi with executor id 2247 parsl.dataflow.dflow INFO: Standard output for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5d43df9c-39d8-437f-9b1b-b7234fb49de0_isrForCrosstalkSources_855440_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5d43df9c-39d8-437f-9b1b-b7234fb49de0_isrForCrosstalkSources_855440_9.stderr parsl.dataflow.dflow DEBUG: Task 2247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2247 has memoization hash 0bd32dff7c1976de5761662b99b4a9ad parsl.dataflow.memoization INFO: Task 2247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2247 try 0 launched on executor multi with executor id 2248 parsl.dataflow.dflow INFO: Standard output for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/6fa0e290-1379-42c4-9b16-158f9426a3b6_isrForCrosstalkSources_855438_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/6fa0e290-1379-42c4-9b16-158f9426a3b6_isrForCrosstalkSources_855438_14.stderr parsl.dataflow.dflow DEBUG: Task 2248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2248 has memoization hash b896bb7a002d94eb5718d4ea95cf23cd parsl.dataflow.memoization INFO: Task 2248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964130 proc_lsst.multi INFO: [multi] job_id 16964130 2024-03-09 14:51:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-09 14:51:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964130 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964130 parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 16964130 parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-09 14:51:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-09 14:51:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2248 try 0 launched on executor multi with executor id 2249 parsl.dataflow.dflow INFO: Standard output for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c8005905-f0d4-4bab-9aec-8af67af05c6f_isrForCrosstalkSources_855438_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c8005905-f0d4-4bab-9aec-8af67af05c6f_isrForCrosstalkSources_855438_16.stderr parsl.dataflow.dflow DEBUG: Task 2249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2249 has memoization hash ee895acb3fae3796ce25dd169c8b7aa2 parsl.dataflow.memoization INFO: Task 2249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2249 try 0 launched on executor multi with executor id 2250 parsl.dataflow.dflow INFO: Standard output for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/53b76f67-7f29-4657-8913-45d20bdd073b_isrForCrosstalkSources_855434_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/53b76f67-7f29-4657-8913-45d20bdd073b_isrForCrosstalkSources_855434_23.stderr parsl.dataflow.dflow DEBUG: Task 2250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2250 has memoization hash 9d58a69be5ee43579fc37f1cb244e38a parsl.dataflow.memoization INFO: Task 2250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2250 try 0 launched on executor multi with executor id 2251 parsl.dataflow.dflow INFO: Standard output for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/65479ed6-3a43-48b9-abe0-0fd5a891f427_isrForCrosstalkSources_855441_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/65479ed6-3a43-48b9-abe0-0fd5a891f427_isrForCrosstalkSources_855441_49.stderr parsl.dataflow.dflow DEBUG: Task 2251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2251 has memoization hash 2baa66d9be46895a505cebe1fb40a8f5 parsl.dataflow.memoization INFO: Task 2251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2251 try 0 launched on executor multi with executor id 2252 parsl.dataflow.dflow INFO: Standard output for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9bb318d3-bc92-4031-823d-7c168a97b85d_isrForCrosstalkSources_855383_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9bb318d3-bc92-4031-823d-7c168a97b85d_isrForCrosstalkSources_855383_9.stderr parsl.dataflow.dflow DEBUG: Task 2252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2252 has memoization hash 6e343b0244fab152f83f919d6fde700f parsl.dataflow.memoization INFO: Task 2252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2252 try 0 launched on executor multi with executor id 2253 parsl.dataflow.dflow INFO: Standard output for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a1870018-368b-493c-acc5-7ca18bc3700d_isrForCrosstalkSources_855427_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a1870018-368b-493c-acc5-7ca18bc3700d_isrForCrosstalkSources_855427_8.stderr parsl.dataflow.dflow DEBUG: Task 2253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2253 has memoization hash f516d9acea88958457882e939e1baa1c parsl.dataflow.memoization INFO: Task 2253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb20e1302004d73e13095262a61e70c with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bd269dbd-8a6d-4b6a-bc2c-b025639b2755_isrForCrosstalkSources_855431_61.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bd269dbd-8a6d-4b6a-bc2c-b025639b2755_isrForCrosstalkSources_855431_61.stderr parsl.dataflow.dflow INFO: Parsl task 2253 try 0 launched on executor multi with executor id 2254 parsl.dataflow.dflow INFO: Standard output for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/62871ec3-1c9f-485f-828e-8e9b5f317719_isrForCrosstalkSources_855426_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/62871ec3-1c9f-485f-828e-8e9b5f317719_isrForCrosstalkSources_855426_6.stderr parsl.dataflow.dflow DEBUG: Task 2254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2254 has memoization hash 1d9953d6ddf13155ef72a03eb20d911c parsl.dataflow.memoization INFO: Task 2254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2254 try 0 launched on executor multi with executor id 2255 parsl.dataflow.dflow INFO: Standard output for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/528002b4-5226-45e7-bbda-1ec4b6258ade_isrForCrosstalkSources_855419_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/528002b4-5226-45e7-bbda-1ec4b6258ade_isrForCrosstalkSources_855419_18.stderr parsl.dataflow.dflow DEBUG: Task 2255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2255 has memoization hash 36bdc47874cb5c0ed5b3409d70c46c22 parsl.dataflow.memoization INFO: Task 2255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2255 try 0 launched on executor multi with executor id 2256 parsl.dataflow.dflow INFO: Standard output for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b0374f13-3a89-4a4f-9d67-c41e37ace78f_isrForCrosstalkSources_855399_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b0374f13-3a89-4a4f-9d67-c41e37ace78f_isrForCrosstalkSources_855399_55.stderr parsl.dataflow.dflow DEBUG: Task 2256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2256 has memoization hash 4f1f105965416034f76bd7a6b225b83d parsl.dataflow.memoization INFO: Task 2256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2256 try 0 launched on executor multi with executor id 2257 parsl.dataflow.dflow INFO: Standard output for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f617a65a-7beb-4b33-98c7-65923b2e6063_isrForCrosstalkSources_855452_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f617a65a-7beb-4b33-98c7-65923b2e6063_isrForCrosstalkSources_855452_26.stderr parsl.dataflow.dflow DEBUG: Task 2257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2257 has memoization hash 1adac9899b9d118aca14ec954ac45308 parsl.dataflow.memoization INFO: Task 2257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2257 try 0 launched on executor multi with executor id 2258 parsl.dataflow.dflow INFO: Standard output for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b5e5b0c4-6101-4050-be8b-55d0c94db78b_isrForCrosstalkSources_855427_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b5e5b0c4-6101-4050-be8b-55d0c94db78b_isrForCrosstalkSources_855427_28.stderr parsl.dataflow.dflow DEBUG: Task 2258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2258 has memoization hash 469f3c90b148c7f55e19233438997bc6 parsl.dataflow.memoization INFO: Task 2258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2258 try 0 launched on executor multi with executor id 2259 parsl.dataflow.dflow INFO: Standard output for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b00ecd27-77f6-4b46-816b-d25fec42dfe0_isrForCrosstalkSources_855427_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b00ecd27-77f6-4b46-816b-d25fec42dfe0_isrForCrosstalkSources_855427_10.stderr parsl.dataflow.dflow DEBUG: Task 2259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2259 has memoization hash 79a964039603136545746f4a5b5954e4 parsl.dataflow.memoization INFO: Task 2259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2259 try 0 launched on executor multi with executor id 2260 parsl.dataflow.dflow INFO: Standard output for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f8d60b2-8948-4a0b-9bee-5c75a5cde9ee_isrForCrosstalkSources_855409_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f8d60b2-8948-4a0b-9bee-5c75a5cde9ee_isrForCrosstalkSources_855409_20.stderr parsl.dataflow.dflow DEBUG: Task 2260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2260 has memoization hash d3ff686cba2cdd4dc485dc12a98ebf13 parsl.dataflow.memoization INFO: Task 2260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2260 try 0 launched on executor multi with executor id 2261 parsl.dataflow.dflow INFO: Standard output for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/fdc804dc-2387-4ef9-aa27-77e831cfc25c_isrForCrosstalkSources_855397_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/fdc804dc-2387-4ef9-aa27-77e831cfc25c_isrForCrosstalkSources_855397_10.stderr parsl.dataflow.dflow DEBUG: Task 2261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2261 has memoization hash c3874f4dffd33cc0a8fcee3fc9ef1306 parsl.dataflow.memoization INFO: Task 2261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2261 try 0 launched on executor multi with executor id 2262 parsl.dataflow.dflow INFO: Standard output for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5e95f288-f475-4fc7-abc5-67b6de053c20_isrForCrosstalkSources_855413_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5e95f288-f475-4fc7-abc5-67b6de053c20_isrForCrosstalkSources_855413_10.stderr parsl.dataflow.dflow DEBUG: Task 2262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2262 has memoization hash 0bd483f98d8f541b994c7aa75cedd7d3 parsl.dataflow.memoization INFO: Task 2262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2262 try 0 launched on executor multi with executor id 2263 parsl.dataflow.dflow INFO: Standard output for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0c654d09-4e78-4284-8f30-563540cba64f_isrForCrosstalkSources_855408_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0c654d09-4e78-4284-8f30-563540cba64f_isrForCrosstalkSources_855408_20.stderr parsl.dataflow.dflow DEBUG: Task 2263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2263 has memoization hash 0ffc75cac8d87c32970dcc71e4c93582 parsl.dataflow.memoization INFO: Task 2263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2263 try 0 launched on executor multi with executor id 2264 parsl.dataflow.dflow INFO: Standard output for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a94700fc-f45f-4a97-b3b8-1175aa11e590_isrForCrosstalkSources_855429_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a94700fc-f45f-4a97-b3b8-1175aa11e590_isrForCrosstalkSources_855429_45.stderr parsl.dataflow.dflow DEBUG: Task 2264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2264 has memoization hash 4dee0ac057efb6f4276a39456ee83b37 parsl.dataflow.memoization INFO: Task 2264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2264 try 0 launched on executor multi with executor id 2265 parsl.dataflow.dflow INFO: Standard output for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0692b536-0ef7-4951-9510-a58a9045eff3_isrForCrosstalkSources_855395_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0692b536-0ef7-4951-9510-a58a9045eff3_isrForCrosstalkSources_855395_9.stderr parsl.dataflow.dflow DEBUG: Task 2265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2265 has memoization hash 70289b270f9154b6eec67da7b4fdb047 parsl.dataflow.memoization INFO: Task 2265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2265 try 0 launched on executor multi with executor id 2266 parsl.dataflow.dflow INFO: Standard output for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed1ca0fd-8667-4c0e-b710-1ba00aeb59d5_isrForCrosstalkSources_855453_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed1ca0fd-8667-4c0e-b710-1ba00aeb59d5_isrForCrosstalkSources_855453_45.stderr parsl.dataflow.dflow DEBUG: Task 2266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2266 has memoization hash 571ef8288800675d54fb60e5efb95eb7 parsl.dataflow.memoization INFO: Task 2266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2266 try 0 launched on executor multi with executor id 2267 parsl.dataflow.dflow INFO: Standard output for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/166c15ac-6f73-4da2-912d-f6d1f3a8cd6c_isrForCrosstalkSources_855453_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/166c15ac-6f73-4da2-912d-f6d1f3a8cd6c_isrForCrosstalkSources_855453_4.stderr parsl.dataflow.dflow DEBUG: Task 2267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2267 has memoization hash fc62885015d6ed98b7cd85e8425fd6b5 parsl.dataflow.memoization INFO: Task 2267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2267 try 0 launched on executor multi with executor id 2268 parsl.dataflow.dflow INFO: Standard output for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/99c54937-1a43-457e-b345-ad51b9b68e6b_isrForCrosstalkSources_855447_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/99c54937-1a43-457e-b345-ad51b9b68e6b_isrForCrosstalkSources_855447_60.stderr parsl.dataflow.dflow DEBUG: Task 2268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2268 has memoization hash aa4db97a41d592bb13fbfed5368157f7 parsl.dataflow.memoization INFO: Task 2268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2268 try 0 launched on executor multi with executor id 2269 parsl.dataflow.dflow INFO: Standard output for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0d9f88ff-7435-43dc-8dee-c7fdd9761ba9_isrForCrosstalkSources_855418_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0d9f88ff-7435-43dc-8dee-c7fdd9761ba9_isrForCrosstalkSources_855418_33.stderr parsl.dataflow.dflow DEBUG: Task 2269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2269 has memoization hash bd81910d6c35be31760f00d2fb2071fb parsl.dataflow.memoization INFO: Task 2269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05c9889dd602dcb35abff08000f91427 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/63a24297-926d-419e-ae0a-2287b2a6f6af_isrForCrosstalkSources_855413_20.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/63a24297-926d-419e-ae0a-2287b2a6f6af_isrForCrosstalkSources_855413_20.stderr parsl.dataflow.dflow INFO: Parsl task 2269 try 0 launched on executor multi with executor id 2270 parsl.dataflow.dflow INFO: Standard output for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72f33fd1-2050-4c9f-8dbd-5251e22e557f_isrForCrosstalkSources_855417_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72f33fd1-2050-4c9f-8dbd-5251e22e557f_isrForCrosstalkSources_855417_62.stderr parsl.dataflow.dflow DEBUG: Task 2270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2270 has memoization hash 457404b32b4e1c5144104d546517db71 parsl.dataflow.memoization INFO: Task 2270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2270 try 0 launched on executor multi with executor id 2271 parsl.dataflow.dflow INFO: Standard output for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9c715dde-cf14-4043-9766-8cd7c2603856_isrForCrosstalkSources_855392_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9c715dde-cf14-4043-9766-8cd7c2603856_isrForCrosstalkSources_855392_49.stderr parsl.dataflow.dflow DEBUG: Task 2271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2271 has memoization hash 7f87db6fbdc94f01fad78f203c8498dd parsl.dataflow.memoization INFO: Task 2271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2271 try 0 launched on executor multi with executor id 2272 parsl.dataflow.dflow INFO: Standard output for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/68e90117-fc46-4599-978e-8d7aec200db6_isrForCrosstalkSources_855430_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/68e90117-fc46-4599-978e-8d7aec200db6_isrForCrosstalkSources_855430_58.stderr parsl.dataflow.dflow DEBUG: Task 2272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2272 has memoization hash d1a0475a9ef88c6b825939ab28a1a65f parsl.dataflow.memoization INFO: Task 2272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2272 try 0 launched on executor multi with executor id 2273 parsl.dataflow.dflow INFO: Standard output for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e03790f-bee2-4b54-adcd-88fda00bdffb_isrForCrosstalkSources_855380_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e03790f-bee2-4b54-adcd-88fda00bdffb_isrForCrosstalkSources_855380_16.stderr parsl.dataflow.dflow DEBUG: Task 2273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2273 has memoization hash 97233c537c13abb9f241be3d71ae4f52 parsl.dataflow.memoization INFO: Task 2273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2273 try 0 launched on executor multi with executor id 2274 parsl.dataflow.dflow INFO: Standard output for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/187d73bd-c05f-4f96-af65-d4876ecbbdea_isrForCrosstalkSources_855435_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/187d73bd-c05f-4f96-af65-d4876ecbbdea_isrForCrosstalkSources_855435_34.stderr parsl.dataflow.dflow DEBUG: Task 2274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2274 has memoization hash ad06044013facbc2a7c3022a9a668d4d parsl.dataflow.memoization INFO: Task 2274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:09 proc_lsst.multi:170 [INFO] [multi] job_id 16964131 proc_lsst.multi INFO: [multi] job_id 16964131 2024-03-09 14:51:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-09 14:51:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964131 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964131 parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 16964131 parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-09 14:51:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-09 14:51:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-09 14:51:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2274 try 0 launched on executor multi with executor id 2275 parsl.dataflow.dflow INFO: Standard output for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1161d7b3-e471-4450-903d-74eb9ef71828_isrForCrosstalkSources_855435_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1161d7b3-e471-4450-903d-74eb9ef71828_isrForCrosstalkSources_855435_47.stderr parsl.dataflow.dflow DEBUG: Task 2275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2275 has memoization hash 92f4ba3b931dce232403daaa63a993b1 parsl.dataflow.memoization INFO: Task 2275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2275 try 0 launched on executor multi with executor id 2276 parsl.dataflow.dflow INFO: Standard output for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/15dbcb47-16fb-412e-8bd5-5abc5e2bb13c_isrForCrosstalkSources_855390_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/15dbcb47-16fb-412e-8bd5-5abc5e2bb13c_isrForCrosstalkSources_855390_32.stderr parsl.dataflow.dflow DEBUG: Task 2276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2276 has memoization hash fd309e4077ec1447bdda219cdf856e24 parsl.dataflow.memoization INFO: Task 2276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0594deafae0aad3b111687d2b766bc2 with result from task 141 parsl.dataflow.dflow INFO: Parsl task 2276 try 0 launched on executor multi with executor id 2277 parsl.dataflow.dflow INFO: Standard output for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/90d887cf-8d23-4919-bb97-8994563ff1ca_isrForCrosstalkSources_855433_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/90d887cf-8d23-4919-bb97-8994563ff1ca_isrForCrosstalkSources_855433_36.stderr parsl.dataflow.dflow DEBUG: Task 2277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2277 has memoization hash 3a33f51f5084bc536d641cfaa168a516 parsl.dataflow.memoization INFO: Task 2277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0bcdbeb7-86de-4898-b485-183f79bc3ed8_isrForCrosstalkSources_855395_1.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0bcdbeb7-86de-4898-b485-183f79bc3ed8_isrForCrosstalkSources_855395_1.stderr parsl.dataflow.dflow INFO: Parsl task 2277 try 0 launched on executor multi with executor id 2278 parsl.dataflow.dflow INFO: Standard output for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b4c3f855-b2ae-4615-be5b-fa1d4a4409e2_isrForCrosstalkSources_855423_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b4c3f855-b2ae-4615-be5b-fa1d4a4409e2_isrForCrosstalkSources_855423_31.stderr parsl.dataflow.dflow DEBUG: Task 2278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2278 has memoization hash 4e530899d17e53a76a26e79f8a35eaa3 parsl.dataflow.memoization INFO: Task 2278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2278 try 0 launched on executor multi with executor id 2279 parsl.dataflow.dflow INFO: Standard output for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e81a3ca3-9ef1-4f17-a17a-1ebf688a41c7_isrForCrosstalkSources_855436_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e81a3ca3-9ef1-4f17-a17a-1ebf688a41c7_isrForCrosstalkSources_855436_5.stderr parsl.dataflow.dflow DEBUG: Task 2279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2279 has memoization hash 798724dc3287ac6dd7dadac01ccac16f parsl.dataflow.memoization INFO: Task 2279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2279 try 0 launched on executor multi with executor id 2280 parsl.dataflow.dflow INFO: Standard output for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dddb7ade-b0c7-4887-a6d1-3e86ebce764c_isrForCrosstalkSources_855458_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dddb7ade-b0c7-4887-a6d1-3e86ebce764c_isrForCrosstalkSources_855458_30.stderr parsl.dataflow.dflow DEBUG: Task 2280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2280 has memoization hash cb5caf7c7ab43be5ef009abd23451359 parsl.dataflow.memoization INFO: Task 2280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59534fd95e65e7f677d36630fe01cce0 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1cadedca-87f2-4800-99d3-26ba45ed42bd_isrForCrosstalkSources_855442_62.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1cadedca-87f2-4800-99d3-26ba45ed42bd_isrForCrosstalkSources_855442_62.stderr parsl.dataflow.dflow INFO: Parsl task 2280 try 0 launched on executor multi with executor id 2281 parsl.dataflow.dflow INFO: Standard output for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/63662c64-8044-40fb-936b-0219fc2032d3_isrForCrosstalkSources_855458_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/63662c64-8044-40fb-936b-0219fc2032d3_isrForCrosstalkSources_855458_62.stderr parsl.dataflow.dflow DEBUG: Task 2281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2281 has memoization hash 6f23227e5c9ae378ba1c998ce6344ade parsl.dataflow.memoization INFO: Task 2281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2281 try 0 launched on executor multi with executor id 2282 parsl.dataflow.dflow INFO: Standard output for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/1ed9e1f0-513c-4128-96d9-f7117ae9b17d_isrForCrosstalkSources_855411_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/1ed9e1f0-513c-4128-96d9-f7117ae9b17d_isrForCrosstalkSources_855411_54.stderr parsl.dataflow.dflow DEBUG: Task 2282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2282 has memoization hash 48cf5dff5ff163542e5d1f4ec57a011a parsl.dataflow.memoization INFO: Task 2282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f835085fad3a4e74572055f309bb642 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e79aeb1c-11b1-4716-9c68-089c1ef2c9c7_isrForCrosstalkSources_855438_62.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e79aeb1c-11b1-4716-9c68-089c1ef2c9c7_isrForCrosstalkSources_855438_62.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e84ca320771f73ba286e77f0f073b992 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/e40f36d6-805c-4582-8817-3c40a2a3d714_isrForCrosstalkSources_855426_60.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/e40f36d6-805c-4582-8817-3c40a2a3d714_isrForCrosstalkSources_855426_60.stderr parsl.dataflow.dflow INFO: Parsl task 2282 try 0 launched on executor multi with executor id 2283 parsl.dataflow.dflow INFO: Standard output for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4e0979a2-e84b-4acb-ac01-bb0baaa6e461_isrForCrosstalkSources_855383_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4e0979a2-e84b-4acb-ac01-bb0baaa6e461_isrForCrosstalkSources_855383_28.stderr parsl.dataflow.dflow DEBUG: Task 2283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2283 has memoization hash b862a836b2d48fdfc83c40d41115344f parsl.dataflow.memoization INFO: Task 2283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2283 try 0 launched on executor multi with executor id 2284 parsl.dataflow.dflow INFO: Standard output for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1303cfd8-7714-4ab9-8403-256704e5e503_isrForCrosstalkSources_855391_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1303cfd8-7714-4ab9-8403-256704e5e503_isrForCrosstalkSources_855391_4.stderr parsl.dataflow.dflow DEBUG: Task 2284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2284 has memoization hash fd35ba3586954e6101a85e136b9d09e3 parsl.dataflow.memoization INFO: Task 2284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2284 try 0 launched on executor multi with executor id 2285 parsl.dataflow.dflow INFO: Standard output for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9bc96fe8-21e3-4841-bf15-61c71379f8bf_isrForCrosstalkSources_855395_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9bc96fe8-21e3-4841-bf15-61c71379f8bf_isrForCrosstalkSources_855395_46.stderr parsl.dataflow.dflow DEBUG: Task 2285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2285 has memoization hash 891ec19f143fced8f1fc16fc160375da parsl.dataflow.memoization INFO: Task 2285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2285 try 0 launched on executor multi with executor id 2286 parsl.dataflow.dflow INFO: Standard output for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f1980b97-b23b-4385-9a19-60a78e909e8f_isrForCrosstalkSources_855382_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f1980b97-b23b-4385-9a19-60a78e909e8f_isrForCrosstalkSources_855382_4.stderr parsl.dataflow.dflow DEBUG: Task 2286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2286 has memoization hash d00b97954ae352116a63232b6957f9ac parsl.dataflow.memoization INFO: Task 2286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2286 try 0 launched on executor multi with executor id 2287 parsl.dataflow.dflow INFO: Standard output for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e7c5f42-9b96-4bbb-b8bf-a762e6c0dd14_isrForCrosstalkSources_855429_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e7c5f42-9b96-4bbb-b8bf-a762e6c0dd14_isrForCrosstalkSources_855429_28.stderr parsl.dataflow.dflow DEBUG: Task 2287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2287 has memoization hash 7bfcd6d933ff2883a593d9945cb779cb parsl.dataflow.memoization INFO: Task 2287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2287 try 0 launched on executor multi with executor id 2288 parsl.dataflow.dflow INFO: Standard output for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f2f06b3b-fa93-41d8-840d-310c900fabf8_isrForCrosstalkSources_855396_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f2f06b3b-fa93-41d8-840d-310c900fabf8_isrForCrosstalkSources_855396_21.stderr parsl.dataflow.dflow DEBUG: Task 2288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2288 has memoization hash aa3c6b070941f72ebebaffb9c3486c4b parsl.dataflow.memoization INFO: Task 2288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2288 try 0 launched on executor multi with executor id 2289 parsl.dataflow.dflow INFO: Standard output for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e3266562-35ca-4930-8aa1-cc077a227152_isrForCrosstalkSources_855404_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e3266562-35ca-4930-8aa1-cc077a227152_isrForCrosstalkSources_855404_43.stderr parsl.dataflow.dflow DEBUG: Task 2289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2289 has memoization hash 4bac71f3323b77c8292a6a58a673422a parsl.dataflow.memoization INFO: Task 2289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2289 try 0 launched on executor multi with executor id 2290 parsl.dataflow.dflow INFO: Standard output for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/74ae3be6-e371-4a4f-88ac-46795a1c6876_isrForCrosstalkSources_855446_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/74ae3be6-e371-4a4f-88ac-46795a1c6876_isrForCrosstalkSources_855446_46.stderr parsl.dataflow.dflow DEBUG: Task 2290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2290 has memoization hash 8dabdcf9c9e7d8311abb118a6f765366 parsl.dataflow.memoization INFO: Task 2290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2290 try 0 launched on executor multi with executor id 2291 parsl.dataflow.dflow INFO: Standard output for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9f93f760-2440-4a49-a041-1ba81d33e8a0_isrForCrosstalkSources_855386_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9f93f760-2440-4a49-a041-1ba81d33e8a0_isrForCrosstalkSources_855386_18.stderr parsl.dataflow.dflow DEBUG: Task 2291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2291 has memoization hash 83ceec98812113e79019c81f2eabd790 parsl.dataflow.memoization INFO: Task 2291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2291 try 0 launched on executor multi with executor id 2292 parsl.dataflow.dflow INFO: Standard output for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/db86bdf4-e115-4986-a69b-a49f3a2deeac_isrForCrosstalkSources_855404_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/db86bdf4-e115-4986-a69b-a49f3a2deeac_isrForCrosstalkSources_855404_5.stderr parsl.dataflow.dflow DEBUG: Task 2292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2292 has memoization hash acb60f0bb771421cf7bc0ca17892c125 parsl.dataflow.memoization INFO: Task 2292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee6dbdfb20c93d97d20d4eba616ea88d with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7a87aa4c-f5c8-4066-a9b3-3d83bb61004a_isrForCrosstalkSources_855392_10.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7a87aa4c-f5c8-4066-a9b3-3d83bb61004a_isrForCrosstalkSources_855392_10.stderr parsl.dataflow.dflow INFO: Parsl task 2292 try 0 launched on executor multi with executor id 2293 parsl.dataflow.dflow INFO: Standard output for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5583e85c-de81-4fc2-bcad-28c701b8bcf0_isrForCrosstalkSources_855398_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5583e85c-de81-4fc2-bcad-28c701b8bcf0_isrForCrosstalkSources_855398_60.stderr parsl.dataflow.dflow DEBUG: Task 2293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2293 has memoization hash 77563abd113b5bbd6598132736d237aa parsl.dataflow.memoization INFO: Task 2293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2293 try 0 launched on executor multi with executor id 2294 parsl.dataflow.dflow INFO: Standard output for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/aabad0a7-467c-454e-9928-af64b949907d_isrForCrosstalkSources_855448_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/aabad0a7-467c-454e-9928-af64b949907d_isrForCrosstalkSources_855448_45.stderr parsl.dataflow.dflow DEBUG: Task 2294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2294 has memoization hash b5f11434c91503638459bd203768ee10 parsl.dataflow.memoization INFO: Task 2294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2294 try 0 launched on executor multi with executor id 2295 parsl.dataflow.dflow INFO: Standard output for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31656eec-9e22-4845-a86f-4b9ffa1978e4_isrForCrosstalkSources_855409_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31656eec-9e22-4845-a86f-4b9ffa1978e4_isrForCrosstalkSources_855409_43.stderr parsl.dataflow.dflow DEBUG: Task 2295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2295 has memoization hash 9ef07de31d7ed2c6e3fb4cf55bc168f0 parsl.dataflow.memoization INFO: Task 2295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2295 try 0 launched on executor multi with executor id 2296 parsl.dataflow.dflow INFO: Standard output for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/81e62d03-193f-4d2b-affd-d70e030751a2_isrForCrosstalkSources_855442_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/81e62d03-193f-4d2b-affd-d70e030751a2_isrForCrosstalkSources_855442_17.stderr parsl.dataflow.dflow DEBUG: Task 2296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2296 has memoization hash 05c2d36e9e38ded0b2eb708524657bfd parsl.dataflow.memoization INFO: Task 2296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:10 proc_lsst.multi:170 [INFO] [multi] job_id 16964132 proc_lsst.multi INFO: [multi] job_id 16964132 2024-03-09 14:51:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-09 14:51:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964132 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964132 parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 16964132 parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-09 14:51:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2296 try 0 launched on executor multi with executor id 2297 parsl.dataflow.dflow INFO: Standard output for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/56dd1b58-893b-4add-bbd3-f0823a946421_isrForCrosstalkSources_855432_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/56dd1b58-893b-4add-bbd3-f0823a946421_isrForCrosstalkSources_855432_17.stderr parsl.dataflow.dflow DEBUG: Task 2297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2297 has memoization hash 359d75a459ef4f682d0b940414462a33 parsl.dataflow.memoization INFO: Task 2297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2297 try 0 launched on executor multi with executor id 2298 parsl.dataflow.dflow INFO: Standard output for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/52307f2e-d68b-4858-8097-69032fcaf850_isrForCrosstalkSources_855387_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/52307f2e-d68b-4858-8097-69032fcaf850_isrForCrosstalkSources_855387_14.stderr parsl.dataflow.dflow DEBUG: Task 2298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2298 has memoization hash d26d47ee6be7d9eb24b5957f5b58073e parsl.dataflow.memoization INFO: Task 2298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2298 try 0 launched on executor multi with executor id 2299 parsl.dataflow.dflow INFO: Standard output for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/645f91d1-98a9-462e-b209-34cb836c099c_isrForCrosstalkSources_855395_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/645f91d1-98a9-462e-b209-34cb836c099c_isrForCrosstalkSources_855395_14.stderr parsl.dataflow.dflow DEBUG: Task 2299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2299 has memoization hash 2c0e4ef72ab6072f2725423b97780ae1 parsl.dataflow.memoization INFO: Task 2299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2299 try 0 launched on executor multi with executor id 2300 parsl.dataflow.dflow INFO: Standard output for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/84b65478-d134-4b01-bc9b-0bc4bc82e301_isrForCrosstalkSources_855402_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/84b65478-d134-4b01-bc9b-0bc4bc82e301_isrForCrosstalkSources_855402_53.stderr parsl.dataflow.dflow DEBUG: Task 2300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2300 has memoization hash f347e51b63fc08bd5f0cfb18f518ceb3 parsl.dataflow.memoization INFO: Task 2300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2300 try 0 launched on executor multi with executor id 2301 parsl.dataflow.dflow INFO: Standard output for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e69a528c-f3dc-456e-83fa-95ef92d78f83_isrForCrosstalkSources_855421_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e69a528c-f3dc-456e-83fa-95ef92d78f83_isrForCrosstalkSources_855421_8.stderr parsl.dataflow.dflow DEBUG: Task 2301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2301 has memoization hash 7aeb67e23919fd86e737a23768e571c7 parsl.dataflow.memoization INFO: Task 2301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2301 try 0 launched on executor multi with executor id 2302 parsl.dataflow.dflow INFO: Standard output for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/11cafe19-4d3b-4252-8bd9-cee93b01a8a3_isrForCrosstalkSources_855412_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/11cafe19-4d3b-4252-8bd9-cee93b01a8a3_isrForCrosstalkSources_855412_49.stderr parsl.dataflow.dflow DEBUG: Task 2302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2302 has memoization hash 26d0cd92127fe19286583259ef99ec39 parsl.dataflow.memoization INFO: Task 2302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2302 try 0 launched on executor multi with executor id 2303 parsl.dataflow.dflow INFO: Standard output for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e9b1c043-10ef-43a9-b01c-61906ba71dbd_isrForCrosstalkSources_855433_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e9b1c043-10ef-43a9-b01c-61906ba71dbd_isrForCrosstalkSources_855433_20.stderr parsl.dataflow.dflow DEBUG: Task 2303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2303 has memoization hash 3f8f87a746dd62c4e1dc07bc4367c494 parsl.dataflow.memoization INFO: Task 2303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2303 try 0 launched on executor multi with executor id 2304 parsl.dataflow.dflow INFO: Standard output for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7917b0ce-ea61-40af-8c03-ab361cc2b5ad_isrForCrosstalkSources_855379_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7917b0ce-ea61-40af-8c03-ab361cc2b5ad_isrForCrosstalkSources_855379_22.stderr parsl.dataflow.dflow DEBUG: Task 2304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2304 has memoization hash 41a490ccdd567acf9982084bf4c922a8 parsl.dataflow.memoization INFO: Task 2304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97214ae8b979df536b62494ac6df1e2e with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/6cf600cf-3bed-45da-ad5a-464db0dc546b_isrForCrosstalkSources_855403_37.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/6cf600cf-3bed-45da-ad5a-464db0dc546b_isrForCrosstalkSources_855403_37.stderr parsl.dataflow.dflow INFO: Parsl task 2304 try 0 launched on executor multi with executor id 2305 parsl.dataflow.dflow INFO: Standard output for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6094b6f4-8078-4e4b-9e05-fdcf2a3cc8d4_isrForCrosstalkSources_855391_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6094b6f4-8078-4e4b-9e05-fdcf2a3cc8d4_isrForCrosstalkSources_855391_36.stderr parsl.dataflow.dflow DEBUG: Task 2305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2305 has memoization hash 8bc99acaa338b713728a741f1fe012f8 parsl.dataflow.memoization INFO: Task 2305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2305 try 0 launched on executor multi with executor id 2306 parsl.dataflow.dflow INFO: Standard output for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01905662-aedb-49de-9459-d4bf4cf76c59_isrForCrosstalkSources_855417_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01905662-aedb-49de-9459-d4bf4cf76c59_isrForCrosstalkSources_855417_3.stderr parsl.dataflow.dflow DEBUG: Task 2306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2306 has memoization hash 3eae07189e92b81da9a806cfcc6cc152 parsl.dataflow.memoization INFO: Task 2306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2306 try 0 launched on executor multi with executor id 2307 parsl.dataflow.dflow INFO: Standard output for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/12ea2219-55ae-4399-bc79-e250462d1e7c_isrForCrosstalkSources_855432_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/12ea2219-55ae-4399-bc79-e250462d1e7c_isrForCrosstalkSources_855432_22.stderr parsl.dataflow.dflow DEBUG: Task 2307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2307 has memoization hash e50471854ee028df781f8655feb20950 parsl.dataflow.memoization INFO: Task 2307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2307 try 0 launched on executor multi with executor id 2308 parsl.dataflow.dflow INFO: Standard output for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8c137b8d-b082-4582-a2d5-9555c3b90f1a_isrForCrosstalkSources_855422_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8c137b8d-b082-4582-a2d5-9555c3b90f1a_isrForCrosstalkSources_855422_27.stderr parsl.dataflow.dflow DEBUG: Task 2308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2308 has memoization hash 478f474d9f591912719b45e310e1b856 parsl.dataflow.memoization INFO: Task 2308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2308 try 0 launched on executor multi with executor id 2309 parsl.dataflow.dflow INFO: Standard output for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b7c73c2e-2c9e-484a-88a2-ec892987f8d1_isrForCrosstalkSources_855396_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b7c73c2e-2c9e-484a-88a2-ec892987f8d1_isrForCrosstalkSources_855396_36.stderr parsl.dataflow.dflow DEBUG: Task 2309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2309 has memoization hash 6b213b03968f44dd2e18bdd2494c274b parsl.dataflow.memoization INFO: Task 2309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2309 try 0 launched on executor multi with executor id 2310 parsl.dataflow.dflow INFO: Standard output for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8c55c8a-3b3e-44dc-83b0-b36df2ba9439_isrForCrosstalkSources_855441_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8c55c8a-3b3e-44dc-83b0-b36df2ba9439_isrForCrosstalkSources_855441_30.stderr parsl.dataflow.dflow DEBUG: Task 2310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2310 has memoization hash bb3c162c6f83463342e3caee6bfe2627 parsl.dataflow.memoization INFO: Task 2310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2310 try 0 launched on executor multi with executor id 2311 parsl.dataflow.dflow INFO: Standard output for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4a5d9da5-f99b-4753-abba-4f4d94cb80b1_isrForCrosstalkSources_855434_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4a5d9da5-f99b-4753-abba-4f4d94cb80b1_isrForCrosstalkSources_855434_50.stderr parsl.dataflow.dflow DEBUG: Task 2311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2311 has memoization hash c75b2818cca8e3e348463323d954df70 parsl.dataflow.memoization INFO: Task 2311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2311 try 0 launched on executor multi with executor id 2312 parsl.dataflow.dflow INFO: Standard output for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/034712d0-9eb0-4d20-9a98-0f638d04526e_isrForCrosstalkSources_855415_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/034712d0-9eb0-4d20-9a98-0f638d04526e_isrForCrosstalkSources_855415_38.stderr parsl.dataflow.dflow DEBUG: Task 2312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2312 has memoization hash c4d1f75ba249e4fe8743097ec8716ecb parsl.dataflow.memoization INFO: Task 2312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2312 try 0 launched on executor multi with executor id 2313 parsl.dataflow.dflow INFO: Standard output for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/96709b38-8441-495c-a093-4241bb95706e_isrForCrosstalkSources_855450_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/96709b38-8441-495c-a093-4241bb95706e_isrForCrosstalkSources_855450_45.stderr parsl.dataflow.dflow DEBUG: Task 2313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2313 has memoization hash c0b1146a0355a07ff5780076573f078e parsl.dataflow.memoization INFO: Task 2313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2313 try 0 launched on executor multi with executor id 2314 parsl.dataflow.dflow INFO: Standard output for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/06d18253-25bd-4e8e-8e7b-0c56e0724702_isrForCrosstalkSources_855440_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/06d18253-25bd-4e8e-8e7b-0c56e0724702_isrForCrosstalkSources_855440_40.stderr parsl.dataflow.dflow DEBUG: Task 2314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2314 has memoization hash 85a2269a90ba856a14d0ee87320cd1c5 parsl.dataflow.memoization INFO: Task 2314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2314 try 0 launched on executor multi with executor id 2315 parsl.dataflow.dflow INFO: Standard output for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/30dd5d45-cf70-471d-9fe4-207998930282_isrForCrosstalkSources_855433_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/30dd5d45-cf70-471d-9fe4-207998930282_isrForCrosstalkSources_855433_46.stderr parsl.dataflow.dflow DEBUG: Task 2315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2315 has memoization hash 393caa88c008694612922910b35d721b parsl.dataflow.memoization INFO: Task 2315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2315 try 0 launched on executor multi with executor id 2316 parsl.dataflow.dflow INFO: Standard output for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4b7ccec6-0eda-4995-8643-6816d7706ebd_isrForCrosstalkSources_855409_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4b7ccec6-0eda-4995-8643-6816d7706ebd_isrForCrosstalkSources_855409_56.stderr parsl.dataflow.dflow DEBUG: Task 2316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2316 has memoization hash 4fed90bb621643c6fdfeeed1a87fea83 parsl.dataflow.memoization INFO: Task 2316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2316 try 0 launched on executor multi with executor id 2317 parsl.dataflow.dflow INFO: Standard output for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7bb8ba3a-7160-4f1d-8fc3-9a5da8494272_isrForCrosstalkSources_855448_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7bb8ba3a-7160-4f1d-8fc3-9a5da8494272_isrForCrosstalkSources_855448_35.stderr parsl.dataflow.dflow DEBUG: Task 2317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2317 has memoization hash 6980c70488d3c1d1e0aaa95aa5079a98 parsl.dataflow.memoization INFO: Task 2317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2317 try 0 launched on executor multi with executor id 2318 parsl.dataflow.dflow INFO: Standard output for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/20be52d6-d698-402d-81c6-ddd5691f3a10_isrForCrosstalkSources_855406_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/20be52d6-d698-402d-81c6-ddd5691f3a10_isrForCrosstalkSources_855406_7.stderr parsl.dataflow.dflow DEBUG: Task 2318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2318 has memoization hash 39483f3cd871c3b068d42e45942670a4 parsl.dataflow.memoization INFO: Task 2318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:10 proc_lsst.multi:170 [INFO] [multi] job_id 16964133 proc_lsst.multi INFO: [multi] job_id 16964133 2024-03-09 14:51:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-09 14:51:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964133 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964133 parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 16964133 parsl.executors.status_handling INFO: Allocated block ID 108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-09 14:51:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-09 14:51:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2318 try 0 launched on executor multi with executor id 2319 parsl.dataflow.dflow INFO: Standard output for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/884c4e4d-107f-4897-a054-5d01c03c76b3_isrForCrosstalkSources_855427_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/884c4e4d-107f-4897-a054-5d01c03c76b3_isrForCrosstalkSources_855427_40.stderr parsl.dataflow.dflow DEBUG: Task 2319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2319 has memoization hash 109c5a5a596f2903b2099b6d4c0a6c01 parsl.dataflow.memoization INFO: Task 2319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2319 try 0 launched on executor multi with executor id 2320 parsl.dataflow.dflow INFO: Standard output for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/1c82362d-34cf-4667-bc1a-97dcfec1086e_isrForCrosstalkSources_855413_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/1c82362d-34cf-4667-bc1a-97dcfec1086e_isrForCrosstalkSources_855413_2.stderr parsl.dataflow.dflow DEBUG: Task 2320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2320 has memoization hash 369f12b0d6506d2fea9a3a8068ef6375 parsl.dataflow.memoization INFO: Task 2320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2320 try 0 launched on executor multi with executor id 2321 parsl.dataflow.dflow INFO: Standard output for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9fcdccba-e818-470a-9fa7-f7c219ee8344_isrForCrosstalkSources_855444_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9fcdccba-e818-470a-9fa7-f7c219ee8344_isrForCrosstalkSources_855444_43.stderr parsl.dataflow.dflow DEBUG: Task 2321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2321 has memoization hash c262439c90f4700fa87bf9d6b7cf9b80 parsl.dataflow.memoization INFO: Task 2321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2321 try 0 launched on executor multi with executor id 2322 parsl.dataflow.dflow INFO: Standard output for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/cabfc420-e525-490f-b444-b9f8788cf314_isrForCrosstalkSources_855439_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/cabfc420-e525-490f-b444-b9f8788cf314_isrForCrosstalkSources_855439_62.stderr parsl.dataflow.dflow DEBUG: Task 2322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2322 has memoization hash d5ec657275d85580475f4ad7a85a5f4f parsl.dataflow.memoization INFO: Task 2322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2322 try 0 launched on executor multi with executor id 2323 parsl.dataflow.dflow INFO: Standard output for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c0c7f2ee-fe43-4c24-a8ab-07db0a81da72_isrForCrosstalkSources_855406_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c0c7f2ee-fe43-4c24-a8ab-07db0a81da72_isrForCrosstalkSources_855406_40.stderr parsl.dataflow.dflow DEBUG: Task 2323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2323 has memoization hash 2dc7007d72c28b7a235d91e8f1dd1a6c parsl.dataflow.memoization INFO: Task 2323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a659ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c9a49f8f8ae4aef38883f3a3e20e66e with result from task 145 parsl.dataflow.dflow INFO: Parsl task 2323 try 0 launched on executor multi with executor id 2324 parsl.dataflow.dflow INFO: Standard output for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f3b7e1fb-49aa-4f0a-ba73-79e62f27c33b_isrForCrosstalkSources_855407_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f3b7e1fb-49aa-4f0a-ba73-79e62f27c33b_isrForCrosstalkSources_855407_43.stderr parsl.dataflow.dflow DEBUG: Task 2324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2324 has memoization hash 8fc54df362e42719bebd6e1e802f9f6b parsl.dataflow.memoization INFO: Task 2324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2324 try 0 launched on executor multi with executor id 2325 parsl.dataflow.dflow INFO: Standard output for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/787b1826-4ba6-4e53-961a-05ddc3e9c917_isrForCrosstalkSources_855454_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/787b1826-4ba6-4e53-961a-05ddc3e9c917_isrForCrosstalkSources_855454_7.stderr parsl.dataflow.dflow DEBUG: Task 2325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2325 has memoization hash 77c2758b2947cce888a8a6f5f77d3694 parsl.dataflow.memoization INFO: Task 2325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b7a0aede-7218-44f2-b832-202a79f05bf8_isrForCrosstalkSources_855442_36.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b7a0aede-7218-44f2-b832-202a79f05bf8_isrForCrosstalkSources_855442_36.stderr parsl.dataflow.dflow INFO: Parsl task 2325 try 0 launched on executor multi with executor id 2326 parsl.dataflow.dflow INFO: Standard output for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc7d9708-1510-4b3c-96f4-93ca422461c0_isrForCrosstalkSources_855403_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc7d9708-1510-4b3c-96f4-93ca422461c0_isrForCrosstalkSources_855403_52.stderr parsl.dataflow.dflow DEBUG: Task 2326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2326 has memoization hash 72bd093dcbef3fffe94e31c3270040c9 parsl.dataflow.memoization INFO: Task 2326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2326 try 0 launched on executor multi with executor id 2327 parsl.dataflow.dflow INFO: Standard output for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/bc341306-3e8b-4d9f-8763-d9c08a3b4ea9_isrForCrosstalkSources_855439_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/bc341306-3e8b-4d9f-8763-d9c08a3b4ea9_isrForCrosstalkSources_855439_12.stderr parsl.dataflow.dflow DEBUG: Task 2327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2327 has memoization hash da30fefd66b4cb98f87d93509dec68d1 parsl.dataflow.memoization INFO: Task 2327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2327 try 0 launched on executor multi with executor id 2328 parsl.dataflow.dflow INFO: Standard output for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5c25d440-6301-471d-b916-c261fc6b82d4_isrForCrosstalkSources_855419_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5c25d440-6301-471d-b916-c261fc6b82d4_isrForCrosstalkSources_855419_37.stderr parsl.dataflow.dflow DEBUG: Task 2328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2328 has memoization hash eafca3cb7cdb224143f1fa9eba73bffa parsl.dataflow.memoization INFO: Task 2328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2328 try 0 launched on executor multi with executor id 2329 parsl.dataflow.dflow INFO: Standard output for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/54a71b9b-d43a-4b7c-998c-0d604071dfe4_isrForCrosstalkSources_855431_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/54a71b9b-d43a-4b7c-998c-0d604071dfe4_isrForCrosstalkSources_855431_54.stderr parsl.dataflow.dflow DEBUG: Task 2329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2329 has memoization hash c81d2424de923439b327769815a36696 parsl.dataflow.memoization INFO: Task 2329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2329 try 0 launched on executor multi with executor id 2330 parsl.dataflow.dflow INFO: Standard output for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/344543a8-7806-441e-bf65-1a9d7e559ae2_isrForCrosstalkSources_855424_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/344543a8-7806-441e-bf65-1a9d7e559ae2_isrForCrosstalkSources_855424_17.stderr parsl.dataflow.dflow DEBUG: Task 2330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2330 has memoization hash f7a917519a119e914f4e57796628dd12 parsl.dataflow.memoization INFO: Task 2330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2330 try 0 launched on executor multi with executor id 2331 parsl.dataflow.dflow INFO: Standard output for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/689d4ece-bede-45e1-8c10-99d92bac0cc1_isrForCrosstalkSources_855428_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/689d4ece-bede-45e1-8c10-99d92bac0cc1_isrForCrosstalkSources_855428_15.stderr parsl.dataflow.dflow DEBUG: Task 2331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2331 has memoization hash ce66ed810ededc1bccc2cc1ae8e38bce parsl.dataflow.memoization INFO: Task 2331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2331 try 0 launched on executor multi with executor id 2332 parsl.dataflow.dflow INFO: Standard output for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0893d78e-8bf0-4874-acdb-cb5279b1f84a_isrForCrosstalkSources_855405_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0893d78e-8bf0-4874-acdb-cb5279b1f84a_isrForCrosstalkSources_855405_39.stderr parsl.dataflow.dflow DEBUG: Task 2332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2332 has memoization hash cf30b73ee027d65463fc76436468ff33 parsl.dataflow.memoization INFO: Task 2332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2332 try 0 launched on executor multi with executor id 2333 parsl.dataflow.dflow INFO: Standard output for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2cd5d636-08c1-4a6a-a747-86cbd0f7689d_isrForCrosstalkSources_855388_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2cd5d636-08c1-4a6a-a747-86cbd0f7689d_isrForCrosstalkSources_855388_37.stderr parsl.dataflow.dflow DEBUG: Task 2333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2333 has memoization hash 069ac4cefe83e614b0966cf46828b8ca parsl.dataflow.memoization INFO: Task 2333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2333 try 0 launched on executor multi with executor id 2334 parsl.dataflow.dflow INFO: Standard output for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3094d68c-8e0d-49cc-af0c-09b65ed6ed8d_isrForCrosstalkSources_855403_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3094d68c-8e0d-49cc-af0c-09b65ed6ed8d_isrForCrosstalkSources_855403_32.stderr parsl.dataflow.dflow DEBUG: Task 2334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2334 has memoization hash 76546bbbc53d4d717b62a2f5d25a8e44 parsl.dataflow.memoization INFO: Task 2334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2334 try 0 launched on executor multi with executor id 2335 parsl.dataflow.dflow INFO: Standard output for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b21f96c9-10c2-4598-b0a9-a1f50e780f10_isrForCrosstalkSources_855446_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b21f96c9-10c2-4598-b0a9-a1f50e780f10_isrForCrosstalkSources_855446_20.stderr parsl.dataflow.dflow DEBUG: Task 2335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2335 has memoization hash 323fe4ca46587cd1ea736b8252961932 parsl.dataflow.memoization INFO: Task 2335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2335 try 0 launched on executor multi with executor id 2336 parsl.dataflow.dflow INFO: Standard output for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5299c219-a7df-4d8f-9908-9ede91d8cf75_isrForCrosstalkSources_855406_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5299c219-a7df-4d8f-9908-9ede91d8cf75_isrForCrosstalkSources_855406_32.stderr parsl.dataflow.dflow DEBUG: Task 2336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2336 has memoization hash e39c3950ff7b1a36437937d15523b6dd parsl.dataflow.memoization INFO: Task 2336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2336 try 0 launched on executor multi with executor id 2337 parsl.dataflow.dflow INFO: Standard output for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/51ed1c51-12f7-4836-84b0-4cd2b5fb7a43_isrForCrosstalkSources_855437_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/51ed1c51-12f7-4836-84b0-4cd2b5fb7a43_isrForCrosstalkSources_855437_27.stderr parsl.dataflow.dflow DEBUG: Task 2337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2337 has memoization hash 3130b9631da7ba44fa9456614c147fd5 parsl.dataflow.memoization INFO: Task 2337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2337 try 0 launched on executor multi with executor id 2338 parsl.dataflow.dflow INFO: Standard output for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/cb33ef6d-c505-40df-bb7c-f42ba8b9b4df_isrForCrosstalkSources_855398_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/cb33ef6d-c505-40df-bb7c-f42ba8b9b4df_isrForCrosstalkSources_855398_4.stderr parsl.dataflow.dflow DEBUG: Task 2338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2338 has memoization hash 7c07bb5ad211166a83acc0a2152a6f31 parsl.dataflow.memoization INFO: Task 2338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2338 try 0 launched on executor multi with executor id 2339 parsl.dataflow.dflow INFO: Standard output for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8deccc42-8486-4da0-a9bb-51e46b38cef8_isrForCrosstalkSources_855398_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8deccc42-8486-4da0-a9bb-51e46b38cef8_isrForCrosstalkSources_855398_20.stderr parsl.dataflow.dflow DEBUG: Task 2339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2339 has memoization hash d68621b2614737d3d02ecbd3f2b5444e parsl.dataflow.memoization INFO: Task 2339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2339 try 0 launched on executor multi with executor id 2340 parsl.dataflow.dflow INFO: Standard output for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7011596e-0fda-4dce-a284-6f44e8aa4da4_isrForCrosstalkSources_855401_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7011596e-0fda-4dce-a284-6f44e8aa4da4_isrForCrosstalkSources_855401_51.stderr parsl.dataflow.dflow DEBUG: Task 2340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2340 has memoization hash 334d7b51e806c1123561668466013543 parsl.dataflow.memoization INFO: Task 2340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2340 try 0 launched on executor multi with executor id 2341 parsl.dataflow.dflow INFO: Standard output for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/904c2f1e-61cf-4668-8ba5-c6ec084388d4_isrForCrosstalkSources_855399_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/904c2f1e-61cf-4668-8ba5-c6ec084388d4_isrForCrosstalkSources_855399_9.stderr parsl.dataflow.dflow DEBUG: Task 2341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2341 has memoization hash f22a58d7530b8a513005ac222a59cefb parsl.dataflow.memoization INFO: Task 2341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2341 try 0 launched on executor multi with executor id 2342 parsl.dataflow.dflow INFO: Standard output for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/4999fcc3-33e2-4c33-a09e-0c122fc35a8e_isrForCrosstalkSources_855380_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/4999fcc3-33e2-4c33-a09e-0c122fc35a8e_isrForCrosstalkSources_855380_28.stderr parsl.dataflow.dflow DEBUG: Task 2342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2342 has memoization hash b9079409a6da34e8ad6021bbf24f80c8 parsl.dataflow.memoization INFO: Task 2342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2342 try 0 launched on executor multi with executor id 2343 parsl.dataflow.dflow INFO: Standard output for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/590825ca-fdca-4682-9250-ea22174a8814_isrForCrosstalkSources_855380_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/590825ca-fdca-4682-9250-ea22174a8814_isrForCrosstalkSources_855380_7.stderr parsl.dataflow.dflow DEBUG: Task 2343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2343 has memoization hash c1db4e841ccd04ca8e8fcb6a69de2d3b parsl.dataflow.memoization INFO: Task 2343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2343 try 0 launched on executor multi with executor id 2344 parsl.dataflow.dflow INFO: Standard output for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ac294c7d-323c-44ce-b94e-f4357d86159d_isrForCrosstalkSources_855403_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ac294c7d-323c-44ce-b94e-f4357d86159d_isrForCrosstalkSources_855403_46.stderr parsl.dataflow.dflow DEBUG: Task 2344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2344 has memoization hash deeee4a2bf89bff7b79352faf980d1af parsl.dataflow.memoization INFO: Task 2344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e8c77056bb36a9f5b146daa1b8cb26 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/95a92f3e-45c6-4083-945f-1a0381bcff8d_isrForCrosstalkSources_855447_33.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/95a92f3e-45c6-4083-945f-1a0381bcff8d_isrForCrosstalkSources_855447_33.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae7b0bb8d4e63283347c08ff4f79d009 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/7ec35235-e7ab-40ce-9f7d-73075ba7fd10_isrForCrosstalkSources_855454_58.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/7ec35235-e7ab-40ce-9f7d-73075ba7fd10_isrForCrosstalkSources_855454_58.stderr parsl.dataflow.dflow INFO: Parsl task 2344 try 0 launched on executor multi with executor id 2345 parsl.dataflow.dflow INFO: Standard output for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7b3bf4e9-a4d1-46af-9b0f-f04afd1386a9_isrForCrosstalkSources_855456_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7b3bf4e9-a4d1-46af-9b0f-f04afd1386a9_isrForCrosstalkSources_855456_20.stderr parsl.dataflow.dflow DEBUG: Task 2345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2345 has memoization hash e642375255c62b13d2f186f778298d92 parsl.dataflow.memoization INFO: Task 2345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2345 try 0 launched on executor multi with executor id 2346 parsl.dataflow.dflow INFO: Standard output for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/69cebed1-9e09-4616-88f2-fb811de48b7a_isrForCrosstalkSources_855455_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/69cebed1-9e09-4616-88f2-fb811de48b7a_isrForCrosstalkSources_855455_61.stderr parsl.dataflow.dflow DEBUG: Task 2346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2346 has memoization hash e195f91c14c32048451f92f723bc6277 parsl.dataflow.memoization INFO: Task 2346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2346 try 0 launched on executor multi with executor id 2347 parsl.dataflow.dflow INFO: Standard output for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/57256f53-ba40-47da-9e2d-0130ce4afd69_isrForCrosstalkSources_855446_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/57256f53-ba40-47da-9e2d-0130ce4afd69_isrForCrosstalkSources_855446_40.stderr parsl.dataflow.dflow DEBUG: Task 2347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2347 has memoization hash 375f962949a917040fc39e08396ea8b2 parsl.dataflow.memoization INFO: Task 2347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:11 proc_lsst.multi:170 [INFO] [multi] job_id 16964134 proc_lsst.multi INFO: [multi] job_id 16964134 2024-03-09 14:51:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-09 14:51:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964134 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964134 parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 16964134 parsl.executors.status_handling INFO: Allocated block ID 109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-109 2024-03-09 14:51:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb04471057474de3f7460b4ab9b3d59 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9119ade5-7faa-4cf0-92d1-9bd6c4c79bf7_isrForCrosstalkSources_855444_53.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9119ade5-7faa-4cf0-92d1-9bd6c4c79bf7_isrForCrosstalkSources_855444_53.stderr parsl.dataflow.dflow INFO: Parsl task 2347 try 0 launched on executor multi with executor id 2348 parsl.dataflow.dflow INFO: Standard output for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4cb0a221-d636-44ea-9652-300cc01714ec_isrForCrosstalkSources_855425_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4cb0a221-d636-44ea-9652-300cc01714ec_isrForCrosstalkSources_855425_52.stderr parsl.dataflow.dflow DEBUG: Task 2348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2348 has memoization hash 0496fa74cfb2a68d0526a1e6f12f5936 parsl.dataflow.memoization INFO: Task 2348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2348 try 0 launched on executor multi with executor id 2349 parsl.dataflow.dflow INFO: Standard output for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/8dd49191-1303-4de2-afd2-c79d96924d23_isrForCrosstalkSources_855434_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/8dd49191-1303-4de2-afd2-c79d96924d23_isrForCrosstalkSources_855434_37.stderr parsl.dataflow.dflow DEBUG: Task 2349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2349 has memoization hash a88344f9bace09fb1304ae0426a7df11 parsl.dataflow.memoization INFO: Task 2349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2349 try 0 launched on executor multi with executor id 2350 parsl.dataflow.dflow INFO: Standard output for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/fb7ed4fa-4989-4b96-bf7a-bc2cb4075f6f_isrForCrosstalkSources_855419_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/fb7ed4fa-4989-4b96-bf7a-bc2cb4075f6f_isrForCrosstalkSources_855419_11.stderr parsl.dataflow.dflow DEBUG: Task 2350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2350 has memoization hash d9546d04cdff2c9a6b30f96382875b6d parsl.dataflow.memoization INFO: Task 2350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2350 try 0 launched on executor multi with executor id 2351 parsl.dataflow.dflow INFO: Standard output for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/84ebbbfb-f2e8-4f6c-9b66-a2c4473040e2_isrForCrosstalkSources_855418_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/84ebbbfb-f2e8-4f6c-9b66-a2c4473040e2_isrForCrosstalkSources_855418_37.stderr parsl.dataflow.dflow DEBUG: Task 2351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2351 has memoization hash ed0d089e1232d4cde5f0348c94d43ef4 parsl.dataflow.memoization INFO: Task 2351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2351 try 0 launched on executor multi with executor id 2352 parsl.dataflow.dflow INFO: Standard output for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/23d78b1f-6a8a-45dc-92b1-2a089f1a0d8c_isrForCrosstalkSources_855456_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/23d78b1f-6a8a-45dc-92b1-2a089f1a0d8c_isrForCrosstalkSources_855456_44.stderr parsl.dataflow.dflow DEBUG: Task 2352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2352 has memoization hash b66d593e8ddf0e998167d537d008122d parsl.dataflow.memoization INFO: Task 2352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2352 try 0 launched on executor multi with executor id 2353 parsl.dataflow.dflow INFO: Standard output for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a9abf411-27d1-4a39-933a-1ab7b6c06a84_isrForCrosstalkSources_855391_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a9abf411-27d1-4a39-933a-1ab7b6c06a84_isrForCrosstalkSources_855391_28.stderr parsl.dataflow.dflow DEBUG: Task 2353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2353 has memoization hash 61bbad7d69c1360631876a76dd30d023 parsl.dataflow.memoization INFO: Task 2353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2353 try 0 launched on executor multi with executor id 2354 parsl.dataflow.dflow INFO: Standard output for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1bff1e60-a5c9-49f2-b9c2-863620b0ece3_isrForCrosstalkSources_855392_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1bff1e60-a5c9-49f2-b9c2-863620b0ece3_isrForCrosstalkSources_855392_39.stderr parsl.dataflow.dflow DEBUG: Task 2354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2354 has memoization hash fcf13e14b9aff847d15262df2efcde0e parsl.dataflow.memoization INFO: Task 2354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2354 try 0 launched on executor multi with executor id 2355 parsl.dataflow.dflow INFO: Standard output for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/b2a2b8e9-9c30-4fa6-8d57-9a13fd163f24_isrForCrosstalkSources_855430_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/b2a2b8e9-9c30-4fa6-8d57-9a13fd163f24_isrForCrosstalkSources_855430_19.stderr parsl.dataflow.dflow DEBUG: Task 2355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2355 has memoization hash 00df4396fcecb8e3744ea52aed20bcd6 parsl.dataflow.memoization INFO: Task 2355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2355 try 0 launched on executor multi with executor id 2356 parsl.dataflow.dflow INFO: Standard output for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8091a086-4f7b-4324-8e57-c4b7ecb1a348_isrForCrosstalkSources_855453_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8091a086-4f7b-4324-8e57-c4b7ecb1a348_isrForCrosstalkSources_855453_50.stderr parsl.dataflow.dflow DEBUG: Task 2356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2356 has memoization hash 69bae0a3ec2c7a1fa33301063084cf0e parsl.dataflow.memoization INFO: Task 2356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0d180a65032dd52d6dcb4bac3833ea with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e448432d-30c1-4794-a41c-8f9ca313c6ea_isrForCrosstalkSources_855392_56.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e448432d-30c1-4794-a41c-8f9ca313c6ea_isrForCrosstalkSources_855392_56.stderr parsl.dataflow.dflow INFO: Parsl task 2356 try 0 launched on executor multi with executor id 2357 parsl.dataflow.dflow INFO: Standard output for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cbfe18ed-3651-476d-9f74-0292e14100ee_isrForCrosstalkSources_855407_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cbfe18ed-3651-476d-9f74-0292e14100ee_isrForCrosstalkSources_855407_6.stderr parsl.dataflow.dflow DEBUG: Task 2357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2357 has memoization hash 792bc1a16e5298a794afe92d93187b3e parsl.dataflow.memoization INFO: Task 2357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2357 try 0 launched on executor multi with executor id 2358 parsl.dataflow.dflow INFO: Standard output for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/db2747b5-b320-4364-888b-97b7a88c213f_isrForCrosstalkSources_855449_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/db2747b5-b320-4364-888b-97b7a88c213f_isrForCrosstalkSources_855449_12.stderr parsl.dataflow.dflow DEBUG: Task 2358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2358 has memoization hash fad1c7ed2dfccd15edbd6b428f7539c0 parsl.dataflow.memoization INFO: Task 2358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2358 try 0 launched on executor multi with executor id 2359 parsl.dataflow.dflow INFO: Standard output for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f0a3f829-d0e0-4a57-b445-3b998ff4e9aa_isrForCrosstalkSources_855437_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f0a3f829-d0e0-4a57-b445-3b998ff4e9aa_isrForCrosstalkSources_855437_13.stderr parsl.dataflow.dflow DEBUG: Task 2359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2359 has memoization hash 80321bf7dbc3f0b6a35bcfd62381d41e parsl.dataflow.memoization INFO: Task 2359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2359 try 0 launched on executor multi with executor id 2360 parsl.dataflow.dflow INFO: Standard output for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ff396927-fabd-47aa-9247-41293c89ff1e_isrForCrosstalkSources_855453_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ff396927-fabd-47aa-9247-41293c89ff1e_isrForCrosstalkSources_855453_58.stderr parsl.dataflow.dflow DEBUG: Task 2360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2360 has memoization hash a318d150d62986f052cb2feb161a4a98 parsl.dataflow.memoization INFO: Task 2360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2360 try 0 launched on executor multi with executor id 2361 parsl.dataflow.dflow INFO: Standard output for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/90f08de3-4d23-4f7d-b773-d8b0b70ec0a2_isrForCrosstalkSources_855387_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/90f08de3-4d23-4f7d-b773-d8b0b70ec0a2_isrForCrosstalkSources_855387_15.stderr parsl.dataflow.dflow DEBUG: Task 2361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2361 has memoization hash 0b068660fb2a9ae16930325e13ee1425 parsl.dataflow.memoization INFO: Task 2361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2361 try 0 launched on executor multi with executor id 2362 parsl.dataflow.dflow INFO: Standard output for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/a7be44d0-3d1f-459a-a153-da0fff078fc3_isrForCrosstalkSources_855424_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/a7be44d0-3d1f-459a-a153-da0fff078fc3_isrForCrosstalkSources_855424_2.stderr parsl.dataflow.dflow DEBUG: Task 2362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2362 has memoization hash 7e1d49e3fa22755de2cab4ac971d6064 parsl.dataflow.memoization INFO: Task 2362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2362 try 0 launched on executor multi with executor id 2363 parsl.dataflow.dflow INFO: Standard output for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/446a21b3-aeff-468e-a691-3ae15ac6083c_isrForCrosstalkSources_855380_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/446a21b3-aeff-468e-a691-3ae15ac6083c_isrForCrosstalkSources_855380_35.stderr parsl.dataflow.dflow DEBUG: Task 2363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2363 has memoization hash e1601410de90a6ef55a4eabe417b76c7 parsl.dataflow.memoization INFO: Task 2363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2363 try 0 launched on executor multi with executor id 2364 parsl.dataflow.dflow INFO: Standard output for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/88b612f0-ae8b-4980-8ad8-7e93aa6e709d_isrForCrosstalkSources_855399_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/88b612f0-ae8b-4980-8ad8-7e93aa6e709d_isrForCrosstalkSources_855399_32.stderr parsl.dataflow.dflow DEBUG: Task 2364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2364 has memoization hash d26be314f357a25f7f17af88ad2b8ec7 parsl.dataflow.memoization INFO: Task 2364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2364 try 0 launched on executor multi with executor id 2365 parsl.dataflow.dflow INFO: Standard output for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/bbca41a8-8bc4-4ab5-ab6b-4561ffd7e6ea_isrForCrosstalkSources_855448_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/bbca41a8-8bc4-4ab5-ab6b-4561ffd7e6ea_isrForCrosstalkSources_855448_60.stderr parsl.dataflow.dflow DEBUG: Task 2365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2365 has memoization hash de62a001ad1e38f96b835dd4b78383ec parsl.dataflow.memoization INFO: Task 2365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2365 try 0 launched on executor multi with executor id 2366 parsl.dataflow.dflow INFO: Standard output for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/7e627552-aa91-4242-bf87-03e0793e2a92_isrForCrosstalkSources_855426_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/7e627552-aa91-4242-bf87-03e0793e2a92_isrForCrosstalkSources_855426_16.stderr parsl.dataflow.dflow DEBUG: Task 2366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2366 has memoization hash e762ec81ff4d7cf215600ee315099802 parsl.dataflow.memoization INFO: Task 2366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2366 try 0 launched on executor multi with executor id 2367 parsl.dataflow.dflow INFO: Standard output for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2e4362cf-9e09-48ae-ad85-d80f8ff4f75b_isrForCrosstalkSources_855410_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2e4362cf-9e09-48ae-ad85-d80f8ff4f75b_isrForCrosstalkSources_855410_23.stderr parsl.dataflow.dflow DEBUG: Task 2367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2367 has memoization hash 21f0e0a7762f9ff5e66a58fefa82fa13 parsl.dataflow.memoization INFO: Task 2367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2367 try 0 launched on executor multi with executor id 2368 parsl.dataflow.dflow INFO: Standard output for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/718f1584-8339-4cce-b492-9d3b28c3d64c_isrForCrosstalkSources_855456_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/718f1584-8339-4cce-b492-9d3b28c3d64c_isrForCrosstalkSources_855456_28.stderr parsl.dataflow.dflow DEBUG: Task 2368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2368 has memoization hash e9670c598f68b31e7f8db57db854f8a1 parsl.dataflow.memoization INFO: Task 2368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2368 try 0 launched on executor multi with executor id 2369 parsl.dataflow.dflow INFO: Standard output for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4428289a-a4b4-4abd-bdb8-fd064a8766e2_isrForCrosstalkSources_855393_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4428289a-a4b4-4abd-bdb8-fd064a8766e2_isrForCrosstalkSources_855393_4.stderr parsl.dataflow.dflow DEBUG: Task 2369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2369 has memoization hash c16a3a2346e38bb6b51c6c08c2c804de parsl.dataflow.memoization INFO: Task 2369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2369 try 0 launched on executor multi with executor id 2370 parsl.dataflow.dflow INFO: Standard output for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7d3809c0-2c69-44fd-b2d2-c7ec424d94d6_isrForCrosstalkSources_855437_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7d3809c0-2c69-44fd-b2d2-c7ec424d94d6_isrForCrosstalkSources_855437_40.stderr parsl.dataflow.dflow DEBUG: Task 2370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2370 has memoization hash 89763a5a98d8470b5cc1f4316b466c77 parsl.dataflow.memoization INFO: Task 2370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2370 try 0 launched on executor multi with executor id 2371 parsl.dataflow.dflow INFO: Standard output for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5b242329-1331-4bb7-8166-33e823255f12_isrForCrosstalkSources_855394_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5b242329-1331-4bb7-8166-33e823255f12_isrForCrosstalkSources_855394_18.stderr parsl.dataflow.dflow DEBUG: Task 2371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2371 has memoization hash 930ef0f5f5d1745f15de5bbd75f02901 parsl.dataflow.memoization INFO: Task 2371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2371 try 0 launched on executor multi with executor id 2372 parsl.dataflow.dflow INFO: Standard output for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81900804-cb1c-447a-842e-617db47098fd_isrForCrosstalkSources_855443_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81900804-cb1c-447a-842e-617db47098fd_isrForCrosstalkSources_855443_48.stderr parsl.dataflow.dflow DEBUG: Task 2372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2372 has memoization hash 93063b92349539b31bc56c567b0e897e parsl.dataflow.memoization INFO: Task 2372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c398190e64fdcf2c9cb9c74a62a8cfdc with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f04fcea5-c999-4f0b-9beb-c9f0a85f8b2b_isrForCrosstalkSources_855448_49.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f04fcea5-c999-4f0b-9beb-c9f0a85f8b2b_isrForCrosstalkSources_855448_49.stderr parsl.dataflow.dflow INFO: Parsl task 2372 try 0 launched on executor multi with executor id 2373 parsl.dataflow.dflow INFO: Standard output for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/916de712-9e26-42b8-a82e-e91e5dd6a709_isrForCrosstalkSources_855455_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/916de712-9e26-42b8-a82e-e91e5dd6a709_isrForCrosstalkSources_855455_36.stderr parsl.dataflow.dflow DEBUG: Task 2373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2373 has memoization hash 22dfec75f79ff652b8f61af9f89c0af2 parsl.dataflow.memoization INFO: Task 2373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2373 try 0 launched on executor multi with executor id 2374 parsl.dataflow.dflow INFO: Standard output for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9d6db98a-70eb-4819-9e28-28326266fc01_isrForCrosstalkSources_855414_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9d6db98a-70eb-4819-9e28-28326266fc01_isrForCrosstalkSources_855414_46.stderr parsl.dataflow.dflow DEBUG: Task 2374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2374 has memoization hash b6fab2ef847498fb9b000e4c59634ae1 parsl.dataflow.memoization INFO: Task 2374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2374 try 0 launched on executor multi with executor id 2375 parsl.dataflow.dflow INFO: Standard output for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e801a099-9ff1-4044-b978-26dde178baae_isrForCrosstalkSources_855392_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e801a099-9ff1-4044-b978-26dde178baae_isrForCrosstalkSources_855392_15.stderr parsl.dataflow.dflow DEBUG: Task 2375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2375 has memoization hash bcbfcd40304bf69a0d025c3a100ed287 parsl.dataflow.memoization INFO: Task 2375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2375 try 0 launched on executor multi with executor id 2376 parsl.dataflow.dflow INFO: Standard output for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8895118a-0566-425d-8c57-b4440fb32a06_isrForCrosstalkSources_855382_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8895118a-0566-425d-8c57-b4440fb32a06_isrForCrosstalkSources_855382_41.stderr parsl.dataflow.dflow DEBUG: Task 2376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2376 has memoization hash 315a2da9b7aad333a970bb1ab5a61460 parsl.dataflow.memoization INFO: Task 2376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2376 try 0 launched on executor multi with executor id 2377 parsl.dataflow.dflow INFO: Standard output for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/7bc30094-e9ec-48f4-86d7-e90f760df552_isrForCrosstalkSources_855418_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/7bc30094-e9ec-48f4-86d7-e90f760df552_isrForCrosstalkSources_855418_60.stderr parsl.dataflow.dflow DEBUG: Task 2377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2377 has memoization hash 25d0b5e2f1555c129fae3bc1bfe621ab parsl.dataflow.memoization INFO: Task 2377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2377 try 0 launched on executor multi with executor id 2378 parsl.dataflow.dflow INFO: Standard output for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a6b6bcf4-0c50-4767-b5e7-aac165ff81af_isrForCrosstalkSources_855385_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a6b6bcf4-0c50-4767-b5e7-aac165ff81af_isrForCrosstalkSources_855385_13.stderr parsl.dataflow.dflow DEBUG: Task 2378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2378 has memoization hash da4c6470409ea8c9ee8d569530cf1a96 parsl.dataflow.memoization INFO: Task 2378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2378 try 0 launched on executor multi with executor id 2379 parsl.dataflow.dflow INFO: Standard output for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/934d996a-2bca-4d4f-b6c5-209ccaf5d274_isrForCrosstalkSources_855407_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/934d996a-2bca-4d4f-b6c5-209ccaf5d274_isrForCrosstalkSources_855407_54.stderr parsl.dataflow.dflow DEBUG: Task 2379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2379 has memoization hash 39cd914d00f9297946e0e3672aa92f7d parsl.dataflow.memoization INFO: Task 2379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:11 proc_lsst.multi:170 [INFO] [multi] job_id 16964135 proc_lsst.multi INFO: [multi] job_id 16964135 2024-03-09 14:51:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-09 14:51:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964135 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964135 parsl.executors.status_handling DEBUG: Launched block 109 on executor multi with job ID 16964135 parsl.executors.status_handling INFO: Allocated block ID 110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-110 2024-03-09 14:51:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-09 14:51:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2379 try 0 launched on executor multi with executor id 2380 parsl.dataflow.dflow INFO: Standard output for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1987c67e-544f-4533-81ee-d0c2beab1384_isrForCrosstalkSources_855441_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1987c67e-544f-4533-81ee-d0c2beab1384_isrForCrosstalkSources_855441_24.stderr parsl.dataflow.dflow DEBUG: Task 2380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2380 has memoization hash a6436af1fb9ec214ccc642abd93e0d61 parsl.dataflow.memoization INFO: Task 2380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2380 try 0 launched on executor multi with executor id 2381 parsl.dataflow.dflow INFO: Standard output for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81c6ce37-a633-45b2-acac-17dc13168f74_isrForCrosstalkSources_855398_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81c6ce37-a633-45b2-acac-17dc13168f74_isrForCrosstalkSources_855398_34.stderr parsl.dataflow.dflow DEBUG: Task 2381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2381 has memoization hash d111f0a3b6f65be71444e621d1dc9780 parsl.dataflow.memoization INFO: Task 2381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2381 try 0 launched on executor multi with executor id 2382 parsl.dataflow.dflow INFO: Standard output for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/bd3b100f-c840-40b0-9049-ebbf4726b77c_isrForCrosstalkSources_855389_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/bd3b100f-c840-40b0-9049-ebbf4726b77c_isrForCrosstalkSources_855389_39.stderr parsl.dataflow.dflow DEBUG: Task 2382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2382 has memoization hash 632332b5a9ec367e27d67454f171edc8 parsl.dataflow.memoization INFO: Task 2382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2382 try 0 launched on executor multi with executor id 2383 parsl.dataflow.dflow INFO: Standard output for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d51c4726-7730-40d7-9ea6-8c27a6d63640_isrForCrosstalkSources_855423_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d51c4726-7730-40d7-9ea6-8c27a6d63640_isrForCrosstalkSources_855423_2.stderr parsl.dataflow.dflow DEBUG: Task 2383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2383 has memoization hash 3a2fea57af307cc6068e4841b22ca93e parsl.dataflow.memoization INFO: Task 2383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2383 try 0 launched on executor multi with executor id 2384 parsl.dataflow.dflow INFO: Standard output for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/fd2f2765-e4af-4eca-bf6a-884da302469f_isrForCrosstalkSources_855443_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/fd2f2765-e4af-4eca-bf6a-884da302469f_isrForCrosstalkSources_855443_20.stderr parsl.dataflow.dflow DEBUG: Task 2384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2384 has memoization hash 3b55b3b1ab65f3e9627939eac3347c8c parsl.dataflow.memoization INFO: Task 2384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61986706d30b37abee29789df9c7404c with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/81b2baf5-8cd5-4ebf-b2c6-23184f060d33_isrForCrosstalkSources_855386_34.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/81b2baf5-8cd5-4ebf-b2c6-23184f060d33_isrForCrosstalkSources_855386_34.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c959cab582a5a7b4c9bcc4e02019b4e7 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/446e0923-53e0-4442-b7f2-951183a361bd_isrForCrosstalkSources_855443_27.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/446e0923-53e0-4442-b7f2-951183a361bd_isrForCrosstalkSources_855443_27.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10efabc5c87744473df0b205c063347e with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/88f62bb3-3c19-437b-a5eb-7b58455b6786_isrForCrosstalkSources_855396_57.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/88f62bb3-3c19-437b-a5eb-7b58455b6786_isrForCrosstalkSources_855396_57.stderr parsl.dataflow.dflow INFO: Parsl task 2384 try 0 launched on executor multi with executor id 2385 parsl.dataflow.dflow INFO: Standard output for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/68e20802-fa96-47ee-b345-7431d8fbfda8_isrForCrosstalkSources_855445_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/68e20802-fa96-47ee-b345-7431d8fbfda8_isrForCrosstalkSources_855445_52.stderr parsl.dataflow.dflow DEBUG: Task 2385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2385 has memoization hash 0fa9bc436c4d858c02a6b322b22ffc51 parsl.dataflow.memoization INFO: Task 2385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2385 try 0 launched on executor multi with executor id 2386 parsl.dataflow.dflow INFO: Standard output for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4c24027-050e-47b8-afef-ad427dd0ff45_isrForCrosstalkSources_855416_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4c24027-050e-47b8-afef-ad427dd0ff45_isrForCrosstalkSources_855416_29.stderr parsl.dataflow.dflow DEBUG: Task 2386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2386 has memoization hash 82937068c4c569bdae5eba958a1a6d84 parsl.dataflow.memoization INFO: Task 2386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1169d9d1d0e77578b78e319f92adcc0 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d48ec79c-04fd-4f59-93a7-c2c7425f88d9_isrForCrosstalkSources_855445_14.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d48ec79c-04fd-4f59-93a7-c2c7425f88d9_isrForCrosstalkSources_855445_14.stderr 2024-03-09 14:51:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964136 proc_lsst.multi INFO: [multi] job_id 16964136 2024-03-09 14:51:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-09 14:51:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964136 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964136 parsl.executors.status_handling DEBUG: Launched block 110 on executor multi with job ID 16964136 parsl.executors.status_handling INFO: Allocated block ID 111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-111 2024-03-09 14:51:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 2024-03-09 14:51:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-09 14:51:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f645a7350c6485be159677cda9b22b with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2df251d6-0be4-4ed7-bc2d-f6c0dfb88cdf_isrForCrosstalkSources_855422_60.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2df251d6-0be4-4ed7-bc2d-f6c0dfb88cdf_isrForCrosstalkSources_855422_60.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54787dae9b6e95041b0d719e433dbae5 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/947ec90b-957b-4c03-86c1-e577a2bb2393_isrForCrosstalkSources_855394_12.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/947ec90b-957b-4c03-86c1-e577a2bb2393_isrForCrosstalkSources_855394_12.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd145774920193c18c90e37ebd84a04c with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/51a4c46e-21bf-44fa-b19a-4f8f8dd05bec_isrForCrosstalkSources_855423_50.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/51a4c46e-21bf-44fa-b19a-4f8f8dd05bec_isrForCrosstalkSources_855423_50.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fc764fae48905e8457842ef56b58b7e with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0f1a2025-5e9a-4d26-a6ea-2dcc28e31aab_isrForCrosstalkSources_855384_40.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0f1a2025-5e9a-4d26-a6ea-2dcc28e31aab_isrForCrosstalkSources_855384_40.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8056fa7ce42557659312e3f5ffe8d8d4 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ceb88ed0-6670-4b41-b6ea-352c00fe2903_isrForCrosstalkSources_855434_48.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ceb88ed0-6670-4b41-b6ea-352c00fe2903_isrForCrosstalkSources_855434_48.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae8c690e1d71e41d5a0b02838c7e428 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/815de729-47e5-4861-a301-010813a6403f_isrForCrosstalkSources_855389_28.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/815de729-47e5-4861-a301-010813a6403f_isrForCrosstalkSources_855389_28.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f11d81d923e07a1bf8fa462f9e433f with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a1f1e329-2a96-4e56-abaa-32a27f24110b_isrForCrosstalkSources_855429_30.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a1f1e329-2a96-4e56-abaa-32a27f24110b_isrForCrosstalkSources_855429_30.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d64e29e0abe60b2467e434028f9af52 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d56687c1-fae7-473a-8c69-e3e99594c9e2_isrForCrosstalkSources_855383_47.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/d56687c1-fae7-473a-8c69-e3e99594c9e2_isrForCrosstalkSources_855383_47.stderr parsl.dataflow.dflow INFO: Parsl task 2386 try 0 launched on executor multi with executor id 2387 parsl.dataflow.dflow INFO: Standard output for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/120fd0a3-607f-4553-8993-9f2eb2ff929d_isrForCrosstalkSources_855391_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/120fd0a3-607f-4553-8993-9f2eb2ff929d_isrForCrosstalkSources_855391_15.stderr parsl.dataflow.dflow DEBUG: Task 2387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2387 has memoization hash 4bfb9a442ee277a7b674d998dbb93ccc parsl.dataflow.memoization INFO: Task 2387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2387 try 0 launched on executor multi with executor id 2388 parsl.dataflow.dflow INFO: Standard output for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/857d40bf-f758-4460-af23-5daea88c9a1a_isrForCrosstalkSources_855422_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/857d40bf-f758-4460-af23-5daea88c9a1a_isrForCrosstalkSources_855422_22.stderr parsl.dataflow.dflow DEBUG: Task 2388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2388 has memoization hash 8b7ffc8bf562df0f4b876408af1b5ac9 parsl.dataflow.memoization INFO: Task 2388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2388 try 0 launched on executor multi with executor id 2389 parsl.dataflow.dflow INFO: Standard output for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/864881cc-7c09-4086-8b45-d003c0c6b8eb_isrForCrosstalkSources_855402_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/864881cc-7c09-4086-8b45-d003c0c6b8eb_isrForCrosstalkSources_855402_30.stderr parsl.dataflow.dflow DEBUG: Task 2389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2389 has memoization hash a2f51b2bd83c6e27748d8d3f05678f90 parsl.dataflow.memoization INFO: Task 2389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2389 try 0 launched on executor multi with executor id 2390 parsl.dataflow.dflow INFO: Standard output for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c286d705-ad70-4099-8743-5209e1053847_isrForCrosstalkSources_855399_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c286d705-ad70-4099-8743-5209e1053847_isrForCrosstalkSources_855399_29.stderr parsl.dataflow.dflow DEBUG: Task 2390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2390 has memoization hash 844871557f0b0a0b62429512af1ef73d parsl.dataflow.memoization INFO: Task 2390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2390 try 0 launched on executor multi with executor id 2391 parsl.dataflow.dflow INFO: Standard output for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1d8adb58-10de-40d5-a8be-edd59da73695_isrForCrosstalkSources_855455_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1d8adb58-10de-40d5-a8be-edd59da73695_isrForCrosstalkSources_855455_49.stderr parsl.dataflow.dflow DEBUG: Task 2391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2391 has memoization hash d17f9a7e12390560ddb8ea58705d2c82 parsl.dataflow.memoization INFO: Task 2391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2391 try 0 launched on executor multi with executor id 2392 parsl.dataflow.dflow INFO: Standard output for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0945eccd-cf0f-4a3b-bb49-2b2148b2eb6a_isrForCrosstalkSources_855406_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0945eccd-cf0f-4a3b-bb49-2b2148b2eb6a_isrForCrosstalkSources_855406_44.stderr parsl.dataflow.dflow DEBUG: Task 2392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2392 has memoization hash a41d9d5a1af2f27f28c75fddf2a89dbe parsl.dataflow.memoization INFO: Task 2392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2392 try 0 launched on executor multi with executor id 2393 parsl.dataflow.dflow INFO: Standard output for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/b2b7fff4-528c-41f9-afd1-ddf9f73f2d55_isrForCrosstalkSources_855392_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/b2b7fff4-528c-41f9-afd1-ddf9f73f2d55_isrForCrosstalkSources_855392_62.stderr parsl.dataflow.dflow DEBUG: Task 2393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2393 has memoization hash f985c7a378640a9ae9d34310cb75fe42 parsl.dataflow.memoization INFO: Task 2393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2393 try 0 launched on executor multi with executor id 2394 parsl.dataflow.dflow INFO: Standard output for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a6a15bc7-94c9-4147-a92e-324bef78b2f6_isrForCrosstalkSources_855418_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a6a15bc7-94c9-4147-a92e-324bef78b2f6_isrForCrosstalkSources_855418_24.stderr parsl.dataflow.dflow DEBUG: Task 2394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2394 has memoization hash 72c7f1875221e405e3d80ab5b4093cc5 parsl.dataflow.memoization INFO: Task 2394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2394 try 0 launched on executor multi with executor id 2395 parsl.dataflow.dflow INFO: Standard output for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/a3e346e5-a925-4d80-a8d4-a15c1dedd131_isrForCrosstalkSources_855437_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/a3e346e5-a925-4d80-a8d4-a15c1dedd131_isrForCrosstalkSources_855437_10.stderr parsl.dataflow.dflow DEBUG: Task 2395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2395 has memoization hash 74fb933df6b23d7b9029fc351aa31219 parsl.dataflow.memoization INFO: Task 2395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2395 try 0 launched on executor multi with executor id 2396 parsl.dataflow.dflow INFO: Standard output for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/61d95868-715b-4782-8b5e-e4ac7edd0135_isrForCrosstalkSources_855445_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/61d95868-715b-4782-8b5e-e4ac7edd0135_isrForCrosstalkSources_855445_23.stderr parsl.dataflow.dflow DEBUG: Task 2396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2396 has memoization hash 6c9ddb678c506c77dd18e94ccf8b14da parsl.dataflow.memoization INFO: Task 2396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2396 try 0 launched on executor multi with executor id 2397 parsl.dataflow.dflow INFO: Standard output for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b2ad14e2-91eb-4763-a2c1-ef4f48df0220_isrForCrosstalkSources_855396_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b2ad14e2-91eb-4763-a2c1-ef4f48df0220_isrForCrosstalkSources_855396_6.stderr parsl.dataflow.dflow DEBUG: Task 2397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2397 has memoization hash 75ff01a3e519638df87b3ac58eb70d20 parsl.dataflow.memoization INFO: Task 2397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2397 try 0 launched on executor multi with executor id 2398 parsl.dataflow.dflow INFO: Standard output for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c5640b09-d786-4725-ad15-c26340270411_isrForCrosstalkSources_855439_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c5640b09-d786-4725-ad15-c26340270411_isrForCrosstalkSources_855439_48.stderr parsl.dataflow.dflow DEBUG: Task 2398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2398 has memoization hash f90648fb6eae9bf27ec9de97d97e85ee parsl.dataflow.memoization INFO: Task 2398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2398 try 0 launched on executor multi with executor id 2399 parsl.dataflow.dflow INFO: Standard output for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5a934b80-717a-469f-8ada-71284bcb0860_isrForCrosstalkSources_855451_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5a934b80-717a-469f-8ada-71284bcb0860_isrForCrosstalkSources_855451_29.stderr parsl.dataflow.dflow DEBUG: Task 2399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2399 has memoization hash 1f6f3443ba70c3fb7e6edf5e93fa223c parsl.dataflow.memoization INFO: Task 2399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2399 try 0 launched on executor multi with executor id 2400 parsl.dataflow.dflow INFO: Standard output for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ce41ac0-2179-45a5-bc0b-94c02222903a_isrForCrosstalkSources_855379_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ce41ac0-2179-45a5-bc0b-94c02222903a_isrForCrosstalkSources_855379_49.stderr parsl.dataflow.dflow DEBUG: Task 2400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2400 has memoization hash a23050a45217d7623e50d9e5f5fb5225 parsl.dataflow.memoization INFO: Task 2400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2400 try 0 launched on executor multi with executor id 2401 parsl.dataflow.dflow INFO: Standard output for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a585649-3980-4733-9df8-b7236b3b83c1_isrForCrosstalkSources_855400_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a585649-3980-4733-9df8-b7236b3b83c1_isrForCrosstalkSources_855400_28.stderr parsl.dataflow.dflow DEBUG: Task 2401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2401 has memoization hash 00985165a459e06a5c5e477f8c4a1bcc parsl.dataflow.memoization INFO: Task 2401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2401 try 0 launched on executor multi with executor id 2402 parsl.dataflow.dflow INFO: Standard output for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6dad5528-a4df-4bd6-ba27-2ba2a6955eb1_isrForCrosstalkSources_855456_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6dad5528-a4df-4bd6-ba27-2ba2a6955eb1_isrForCrosstalkSources_855456_54.stderr parsl.dataflow.dflow DEBUG: Task 2402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2402 has memoization hash 67804ec3da307fa18cf17001b7b70049 parsl.dataflow.memoization INFO: Task 2402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2402 try 0 launched on executor multi with executor id 2403 parsl.dataflow.dflow INFO: Standard output for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65a30e4e-beab-49a7-91fb-cdba2541702e_isrForCrosstalkSources_855413_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65a30e4e-beab-49a7-91fb-cdba2541702e_isrForCrosstalkSources_855413_34.stderr parsl.dataflow.dflow DEBUG: Task 2403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2403 has memoization hash 698a7fae3c76c4c5619191c961b75695 parsl.dataflow.memoization INFO: Task 2403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2403 try 0 launched on executor multi with executor id 2404 parsl.dataflow.dflow INFO: Standard output for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/e3075180-b866-4d92-a3ce-4a8558bf50c2_isrForCrosstalkSources_855425_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/e3075180-b866-4d92-a3ce-4a8558bf50c2_isrForCrosstalkSources_855425_43.stderr parsl.dataflow.dflow DEBUG: Task 2404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2404 has memoization hash 92d7270d496efbd9b3ee6f8b6b59d222 parsl.dataflow.memoization INFO: Task 2404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2404 try 0 launched on executor multi with executor id 2405 parsl.dataflow.dflow INFO: Standard output for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/edb32e10-4572-4155-a26f-268d2e7c32bb_isrForCrosstalkSources_855456_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/edb32e10-4572-4155-a26f-268d2e7c32bb_isrForCrosstalkSources_855456_15.stderr parsl.dataflow.dflow DEBUG: Task 2405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2405 has memoization hash 90fa5e7de560a7b77edbd058e00dfa59 parsl.dataflow.memoization INFO: Task 2405 had no result in cache 2024-03-09 14:51:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964137 parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] job_id 16964137 2024-03-09 14:51:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-09 14:51:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964137 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964137 parsl.executors.status_handling DEBUG: Launched block 111 on executor multi with job ID 16964137 parsl.executors.status_handling INFO: Allocated block ID 112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-112 2024-03-09 14:51:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2405 try 0 launched on executor multi with executor id 2406 parsl.dataflow.dflow INFO: Standard output for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/171dec6d-b74c-4ff7-aa9e-f0bf86033de8_isrForCrosstalkSources_855427_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/171dec6d-b74c-4ff7-aa9e-f0bf86033de8_isrForCrosstalkSources_855427_37.stderr parsl.dataflow.dflow DEBUG: Task 2406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2406 has memoization hash 02eae4d489d6855e9905ce96e6e12d63 parsl.dataflow.memoization INFO: Task 2406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2406 try 0 launched on executor multi with executor id 2407 parsl.dataflow.dflow INFO: Standard output for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9be537aa-f4e7-4b49-8a31-b4fae92de33d_isrForCrosstalkSources_855438_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9be537aa-f4e7-4b49-8a31-b4fae92de33d_isrForCrosstalkSources_855438_42.stderr parsl.dataflow.dflow DEBUG: Task 2407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2407 has memoization hash 52d84a8b95450b1b812bf6312d5b589e parsl.dataflow.memoization INFO: Task 2407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2407 try 0 launched on executor multi with executor id 2408 parsl.dataflow.dflow INFO: Standard output for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/30f75b6e-5d45-4092-b51e-932f8c573911_isrForCrosstalkSources_855429_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/30f75b6e-5d45-4092-b51e-932f8c573911_isrForCrosstalkSources_855429_22.stderr parsl.dataflow.dflow DEBUG: Task 2408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2408 has memoization hash 210e8e6428c84428f5644017fa9cfd12 parsl.dataflow.memoization INFO: Task 2408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2408 try 0 launched on executor multi with executor id 2409 parsl.dataflow.dflow INFO: Standard output for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/42f26daa-0ede-4330-be4b-96d8842b2420_isrForCrosstalkSources_855400_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/42f26daa-0ede-4330-be4b-96d8842b2420_isrForCrosstalkSources_855400_10.stderr parsl.dataflow.dflow DEBUG: Task 2409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2409 has memoization hash ea2596cf6118f49ada6019d27bb48240 parsl.dataflow.memoization INFO: Task 2409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2409 try 0 launched on executor multi with executor id 2410 parsl.dataflow.dflow INFO: Standard output for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b7cfab68-9d78-40f0-9d9f-6de8893ab605_isrForCrosstalkSources_855379_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b7cfab68-9d78-40f0-9d9f-6de8893ab605_isrForCrosstalkSources_855379_5.stderr parsl.dataflow.dflow DEBUG: Task 2410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2410 has memoization hash 0c3f68099efa696870f827018ea253f5 parsl.dataflow.memoization INFO: Task 2410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2410 try 0 launched on executor multi with executor id 2411 parsl.dataflow.dflow INFO: Standard output for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7b8f17c5-0f47-48e9-851f-1f354b90764e_isrForCrosstalkSources_855403_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7b8f17c5-0f47-48e9-851f-1f354b90764e_isrForCrosstalkSources_855403_33.stderr parsl.dataflow.dflow DEBUG: Task 2411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2411 has memoization hash 6ca91846f0bc0037d4e4fa0f5763f276 parsl.dataflow.memoization INFO: Task 2411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2411 try 0 launched on executor multi with executor id 2412 parsl.dataflow.dflow INFO: Standard output for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bf5eff79-59cb-433f-aab4-3e2ac2baa38d_isrForCrosstalkSources_855381_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bf5eff79-59cb-433f-aab4-3e2ac2baa38d_isrForCrosstalkSources_855381_54.stderr parsl.dataflow.dflow DEBUG: Task 2412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2412 has memoization hash 4203ab142ff6ba25284e0d91a3b197ac parsl.dataflow.memoization INFO: Task 2412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2412 try 0 launched on executor multi with executor id 2413 parsl.dataflow.dflow INFO: Standard output for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1bc0b450-0338-4ba4-b6b1-8debae80609f_isrForCrosstalkSources_855445_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1bc0b450-0338-4ba4-b6b1-8debae80609f_isrForCrosstalkSources_855445_8.stderr parsl.dataflow.dflow DEBUG: Task 2413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2413 has memoization hash 4d8b5ba1ba21c8f90852fa0ed60be6fc parsl.dataflow.memoization INFO: Task 2413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2413 try 0 launched on executor multi with executor id 2414 parsl.dataflow.dflow INFO: Standard output for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9e248002-4b1a-4034-981a-1c92022ebc0c_isrForCrosstalkSources_855427_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9e248002-4b1a-4034-981a-1c92022ebc0c_isrForCrosstalkSources_855427_54.stderr parsl.dataflow.dflow DEBUG: Task 2414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2414 has memoization hash a325289e024287e2d30f3196752b5547 parsl.dataflow.memoization INFO: Task 2414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2414 try 0 launched on executor multi with executor id 2415 parsl.dataflow.dflow INFO: Standard output for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/24ccd384-3ccc-4991-906d-16d55ab97148_isrForCrosstalkSources_855421_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/24ccd384-3ccc-4991-906d-16d55ab97148_isrForCrosstalkSources_855421_62.stderr parsl.dataflow.dflow DEBUG: Task 2415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2415 has memoization hash b129b2b2e7a96060ffbdb7fbfff41f1e parsl.dataflow.memoization INFO: Task 2415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2415 try 0 launched on executor multi with executor id 2416 parsl.dataflow.dflow INFO: Standard output for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8973f42b-49be-4559-85b4-998fdbde92f3_isrForCrosstalkSources_855446_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8973f42b-49be-4559-85b4-998fdbde92f3_isrForCrosstalkSources_855446_51.stderr parsl.dataflow.dflow DEBUG: Task 2416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2416 has memoization hash a783d76e73510f3e605cedf23e5c3b3e parsl.dataflow.memoization INFO: Task 2416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2416 try 0 launched on executor multi with executor id 2417 parsl.dataflow.dflow INFO: Standard output for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1fc4bf3e-8b0e-4ae3-a746-ada9a17ae6bd_isrForCrosstalkSources_855428_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1fc4bf3e-8b0e-4ae3-a746-ada9a17ae6bd_isrForCrosstalkSources_855428_47.stderr parsl.dataflow.dflow DEBUG: Task 2417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2417 has memoization hash c81c21526ee3a60a47e93781203d5dfd parsl.dataflow.memoization INFO: Task 2417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2417 try 0 launched on executor multi with executor id 2418 parsl.dataflow.dflow INFO: Standard output for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/acef806a-cfe5-4f97-b702-8da529f7c0fe_isrForCrosstalkSources_855423_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/acef806a-cfe5-4f97-b702-8da529f7c0fe_isrForCrosstalkSources_855423_54.stderr parsl.dataflow.dflow DEBUG: Task 2418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2418 has memoization hash b6bfebe13ba6439ff0ea7f3f3587c9b8 parsl.dataflow.memoization INFO: Task 2418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2418 try 0 launched on executor multi with executor id 2419 parsl.dataflow.dflow INFO: Standard output for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/21c2338b-46e6-4ecd-8084-23cabfd6c772_isrForCrosstalkSources_855443_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/21c2338b-46e6-4ecd-8084-23cabfd6c772_isrForCrosstalkSources_855443_44.stderr parsl.dataflow.dflow DEBUG: Task 2419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2419 has memoization hash fd6159fe1a449e1b3933c7a4f8ea4613 parsl.dataflow.memoization INFO: Task 2419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2419 try 0 launched on executor multi with executor id 2420 parsl.dataflow.dflow INFO: Standard output for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab10fedd-531f-419d-a1a1-f56d0a92b0d1_isrForCrosstalkSources_855437_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab10fedd-531f-419d-a1a1-f56d0a92b0d1_isrForCrosstalkSources_855437_45.stderr parsl.dataflow.dflow DEBUG: Task 2420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2420 has memoization hash e7ef4ae927bc1adb244d166fc65bcb85 parsl.dataflow.memoization INFO: Task 2420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2420 try 0 launched on executor multi with executor id 2421 parsl.dataflow.dflow INFO: Standard output for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d244876a-b9c8-4fc6-90a5-2319e9043f83_isrForCrosstalkSources_855443_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d244876a-b9c8-4fc6-90a5-2319e9043f83_isrForCrosstalkSources_855443_32.stderr parsl.dataflow.dflow DEBUG: Task 2421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2421 has memoization hash f363baa335183232be486d826e796f3a parsl.dataflow.memoization INFO: Task 2421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2421 try 0 launched on executor multi with executor id 2422 parsl.dataflow.dflow INFO: Standard output for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0140d1bb-43b8-437a-9f6f-7e481a30a21a_isrForCrosstalkSources_855417_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0140d1bb-43b8-437a-9f6f-7e481a30a21a_isrForCrosstalkSources_855417_16.stderr parsl.dataflow.dflow DEBUG: Task 2422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2422 has memoization hash 0b1f7f15107f5eb4e6f0837a561c3381 parsl.dataflow.memoization INFO: Task 2422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2422 try 0 launched on executor multi with executor id 2423 parsl.dataflow.dflow INFO: Standard output for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7edbbe18-fcab-4d38-b891-3dd1c48fd40d_isrForCrosstalkSources_855443_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7edbbe18-fcab-4d38-b891-3dd1c48fd40d_isrForCrosstalkSources_855443_59.stderr parsl.dataflow.dflow DEBUG: Task 2423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2423 has memoization hash 7b890bc0d2dfa1fa6418d359285a1554 parsl.dataflow.memoization INFO: Task 2423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2423 try 0 launched on executor multi with executor id 2424 parsl.dataflow.dflow INFO: Standard output for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a41c0e84-4975-4e1f-b9f7-a3f77fbaccd4_isrForCrosstalkSources_855422_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a41c0e84-4975-4e1f-b9f7-a3f77fbaccd4_isrForCrosstalkSources_855422_62.stderr parsl.dataflow.dflow DEBUG: Task 2424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2424 has memoization hash 077343924ffd50f2a044dc265a06fb21 parsl.dataflow.memoization INFO: Task 2424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2424 try 0 launched on executor multi with executor id 2425 parsl.dataflow.dflow INFO: Standard output for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccd108f1-fdb3-488e-b23d-bb3a81a0f957_isrForCrosstalkSources_855403_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccd108f1-fdb3-488e-b23d-bb3a81a0f957_isrForCrosstalkSources_855403_55.stderr parsl.dataflow.dflow DEBUG: Task 2425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2425 has memoization hash 508a9befde0c240495d4ec01ba635121 parsl.dataflow.memoization INFO: Task 2425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2425 try 0 launched on executor multi with executor id 2426 parsl.dataflow.dflow INFO: Standard output for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fa7380a2-0dbe-4689-91c6-63aa30de0975_isrForCrosstalkSources_855382_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fa7380a2-0dbe-4689-91c6-63aa30de0975_isrForCrosstalkSources_855382_44.stderr parsl.dataflow.dflow DEBUG: Task 2426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2426 has memoization hash a6bd4f6653d325d9eecda02d0ea4c378 parsl.dataflow.memoization INFO: Task 2426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2426 try 0 launched on executor multi with executor id 2427 parsl.dataflow.dflow INFO: Standard output for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/076b7b8e-70f1-413f-95c3-2b17c696b402_isrForCrosstalkSources_855409_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/076b7b8e-70f1-413f-95c3-2b17c696b402_isrForCrosstalkSources_855409_34.stderr 2024-03-09 14:51:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964138 parsl.dataflow.dflow DEBUG: Task 2427 will be sent to executor multi proc_lsst.multi INFO: [multi] job_id 16964138 2024-03-09 14:51:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 57 parsl.dataflow.dflow DEBUG: Adding output dependencies proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-09 14:51:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow INFO: Task 2427 submitted for App isrForCrosstalkSources, not waiting on any dependency proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964138 parsl.dataflow.dflow DEBUG: Task 2427 set to pending state with AppFuture: proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964138 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.executors.status_handling DEBUG: Launched block 112 on executor multi with job ID 16964138 parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.executors.status_handling INFO: Allocated block ID 113 parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-113 2024-03-09 14:51:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 parsl.dataflow.memoization DEBUG: Task 2427 has memoization hash e441d125d41bc6c6fa53faa4a641fc05 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.memoization INFO: Task 2427 had no result in cache proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a66c0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2427 try 0 launched on executor multi with executor id 2428 parsl.dataflow.dflow INFO: Standard output for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/13e5a9e3-5113-4ddd-bf73-306c21814f1d_isrForCrosstalkSources_855454_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/13e5a9e3-5113-4ddd-bf73-306c21814f1d_isrForCrosstalkSources_855454_39.stderr parsl.dataflow.dflow DEBUG: Task 2428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2428 has memoization hash 7bee530f3029529c8d1c288ed3a1e1e2 parsl.dataflow.memoization INFO: Task 2428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2428 try 0 launched on executor multi with executor id 2429 parsl.dataflow.dflow INFO: Standard output for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1cfac012-652b-4731-bf13-8285c9150480_isrForCrosstalkSources_855435_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1cfac012-652b-4731-bf13-8285c9150480_isrForCrosstalkSources_855435_4.stderr parsl.dataflow.dflow DEBUG: Task 2429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2429 has memoization hash a474d5cc63229c03063a743b3ed8a4ae parsl.dataflow.memoization INFO: Task 2429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2429 try 0 launched on executor multi with executor id 2430 parsl.dataflow.dflow INFO: Standard output for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f9db8dfb-5a9d-45a8-b1a0-0d3c62be9095_isrForCrosstalkSources_855421_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f9db8dfb-5a9d-45a8-b1a0-0d3c62be9095_isrForCrosstalkSources_855421_22.stderr parsl.dataflow.dflow DEBUG: Task 2430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2430 has memoization hash 3bcb88aca955a7dc71a29dedaf05b8ad parsl.dataflow.memoization INFO: Task 2430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2430 try 0 launched on executor multi with executor id 2431 parsl.dataflow.dflow INFO: Standard output for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9a66b52e-460d-4af3-80fa-158b260b665d_isrForCrosstalkSources_855387_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9a66b52e-460d-4af3-80fa-158b260b665d_isrForCrosstalkSources_855387_18.stderr parsl.dataflow.dflow DEBUG: Task 2431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2431 has memoization hash b713797041994ba9a5dc213de48fd75c parsl.dataflow.memoization INFO: Task 2431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2431 try 0 launched on executor multi with executor id 2432 parsl.dataflow.dflow INFO: Standard output for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d48ce83d-a6b4-43d4-a81f-9e91bf7c1f55_isrForCrosstalkSources_855397_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d48ce83d-a6b4-43d4-a81f-9e91bf7c1f55_isrForCrosstalkSources_855397_25.stderr parsl.dataflow.dflow DEBUG: Task 2432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2432 has memoization hash 8873b30e7c90530f6031f46bec30ce5b parsl.dataflow.memoization INFO: Task 2432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2432 try 0 launched on executor multi with executor id 2433 parsl.dataflow.dflow INFO: Standard output for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c9056e20-4aa0-47b1-a20a-a23acb680074_isrForCrosstalkSources_855409_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c9056e20-4aa0-47b1-a20a-a23acb680074_isrForCrosstalkSources_855409_45.stderr parsl.dataflow.dflow DEBUG: Task 2433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2433 has memoization hash 60a764c7c40dc8f519be1cc184dd9c41 parsl.dataflow.memoization INFO: Task 2433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2433 try 0 launched on executor multi with executor id 2434 parsl.dataflow.dflow INFO: Standard output for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f0a7803b-bb46-4d05-931f-b461b0e755e1_isrForCrosstalkSources_855384_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f0a7803b-bb46-4d05-931f-b461b0e755e1_isrForCrosstalkSources_855384_58.stderr parsl.dataflow.dflow DEBUG: Task 2434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2434 has memoization hash 2be3b37fdd4f18188f6e57e93c96c5f1 parsl.dataflow.memoization INFO: Task 2434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2434 try 0 launched on executor multi with executor id 2435 parsl.dataflow.dflow INFO: Standard output for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/05d1650b-cb2c-4e07-86be-e6f26e804e1a_isrForCrosstalkSources_855401_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/05d1650b-cb2c-4e07-86be-e6f26e804e1a_isrForCrosstalkSources_855401_57.stderr parsl.dataflow.dflow DEBUG: Task 2435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2435 has memoization hash a90f724e38a0f7359df16f78e9ddfe3d parsl.dataflow.memoization INFO: Task 2435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2435 try 0 launched on executor multi with executor id 2436 parsl.dataflow.dflow INFO: Standard output for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/2e72eaf9-4a7d-4c3b-96f2-c9cd12ecdc72_isrForCrosstalkSources_855448_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/2e72eaf9-4a7d-4c3b-96f2-c9cd12ecdc72_isrForCrosstalkSources_855448_52.stderr parsl.dataflow.dflow DEBUG: Task 2436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2436 has memoization hash afbec66a6e0965013b7f7d4616dedd7a parsl.dataflow.memoization INFO: Task 2436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2436 try 0 launched on executor multi with executor id 2437 parsl.dataflow.dflow INFO: Standard output for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8672af45-df8e-4801-ad3b-6ab9c0affe2c_isrForCrosstalkSources_855443_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8672af45-df8e-4801-ad3b-6ab9c0affe2c_isrForCrosstalkSources_855443_14.stderr parsl.dataflow.dflow DEBUG: Task 2437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2437 has memoization hash d1a153a34090552587bea325ef680a72 parsl.dataflow.memoization INFO: Task 2437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2437 try 0 launched on executor multi with executor id 2438 parsl.dataflow.dflow INFO: Standard output for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8eaa6bf2-45a2-4f26-8920-b42ac3b34507_isrForCrosstalkSources_855402_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8eaa6bf2-45a2-4f26-8920-b42ac3b34507_isrForCrosstalkSources_855402_15.stderr parsl.dataflow.dflow DEBUG: Task 2438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2438 has memoization hash 9b76e59fd84173d216dbb911aa4f08f5 parsl.dataflow.memoization INFO: Task 2438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2438 try 0 launched on executor multi with executor id 2439 parsl.dataflow.dflow INFO: Standard output for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fb764033-0a18-48d5-b10e-d06976ed7777_isrForCrosstalkSources_855387_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fb764033-0a18-48d5-b10e-d06976ed7777_isrForCrosstalkSources_855387_32.stderr parsl.dataflow.dflow DEBUG: Task 2439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2439 has memoization hash a15ad4d652118e2525befe868f7020cc parsl.dataflow.memoization INFO: Task 2439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2439 try 0 launched on executor multi with executor id 2440 parsl.dataflow.dflow INFO: Standard output for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/91530958-2efe-4521-ae82-3f688bfd953b_isrForCrosstalkSources_855410_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/91530958-2efe-4521-ae82-3f688bfd953b_isrForCrosstalkSources_855410_12.stderr parsl.dataflow.dflow DEBUG: Task 2440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2440 has memoization hash 22e1cc2fca304e2bb8c10d87837bdffc parsl.dataflow.memoization INFO: Task 2440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2440 try 0 launched on executor multi with executor id 2441 parsl.dataflow.dflow INFO: Standard output for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1add1895-6d07-44cc-bea7-cdb5d46d1804_isrForCrosstalkSources_855409_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1add1895-6d07-44cc-bea7-cdb5d46d1804_isrForCrosstalkSources_855409_27.stderr parsl.dataflow.dflow DEBUG: Task 2441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2441 has memoization hash d78324891d4bc544d9c08039d2786f3f parsl.dataflow.memoization INFO: Task 2441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2441 try 0 launched on executor multi with executor id 2442 parsl.dataflow.dflow INFO: Standard output for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a28aec16-9dfb-43c5-9f08-108f0b42a020_isrForCrosstalkSources_855380_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a28aec16-9dfb-43c5-9f08-108f0b42a020_isrForCrosstalkSources_855380_53.stderr parsl.dataflow.dflow DEBUG: Task 2442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2442 has memoization hash 88f76d9036e3c8505066713d4de5bb09 parsl.dataflow.memoization INFO: Task 2442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2442 try 0 launched on executor multi with executor id 2443 parsl.dataflow.dflow INFO: Standard output for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/4b66af39-a96e-4bf8-9b52-34f5f2ad5ea3_isrForCrosstalkSources_855416_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/4b66af39-a96e-4bf8-9b52-34f5f2ad5ea3_isrForCrosstalkSources_855416_55.stderr parsl.dataflow.dflow DEBUG: Task 2443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2443 has memoization hash 611cfc2556cb3793e5e80af0fd2fe815 parsl.dataflow.memoization INFO: Task 2443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2443 try 0 launched on executor multi with executor id 2444 parsl.dataflow.dflow INFO: Standard output for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c79315c3-7485-4bd5-aeb4-0e6ca8a5dfa0_isrForCrosstalkSources_855457_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c79315c3-7485-4bd5-aeb4-0e6ca8a5dfa0_isrForCrosstalkSources_855457_34.stderr parsl.dataflow.dflow DEBUG: Task 2444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2444 has memoization hash e3763a49c1bab56269c98cf706b33797 parsl.dataflow.memoization INFO: Task 2444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2444 try 0 launched on executor multi with executor id 2445 parsl.dataflow.dflow INFO: Standard output for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e2e1bac-1503-465c-b58d-c3ab0e660b81_isrForCrosstalkSources_855438_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e2e1bac-1503-465c-b58d-c3ab0e660b81_isrForCrosstalkSources_855438_3.stderr parsl.dataflow.dflow DEBUG: Task 2445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2445 has memoization hash 0e9664a1560d741bfa562ff0e5b01762 parsl.dataflow.memoization INFO: Task 2445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2445 try 0 launched on executor multi with executor id 2446 parsl.dataflow.dflow INFO: Standard output for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/31dc3073-8a09-4b6f-858b-f50ba4946306_isrForCrosstalkSources_855444_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/31dc3073-8a09-4b6f-858b-f50ba4946306_isrForCrosstalkSources_855444_23.stderr parsl.dataflow.dflow DEBUG: Task 2446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2446 has memoization hash 375aab645326e1fb143b1afbfe30340e parsl.dataflow.memoization INFO: Task 2446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2446 try 0 launched on executor multi with executor id 2447 parsl.dataflow.dflow INFO: Standard output for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d075a05-6c23-40b1-b73a-42a243e23902_isrForCrosstalkSources_855459_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d075a05-6c23-40b1-b73a-42a243e23902_isrForCrosstalkSources_855459_49.stderr parsl.dataflow.dflow DEBUG: Task 2447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2447 has memoization hash e46090ad63263d9022f0b432c7cb4f35 parsl.dataflow.memoization INFO: Task 2447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2447 try 0 launched on executor multi with executor id 2448 parsl.dataflow.dflow INFO: Standard output for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d1bc0f8e-08ed-4ef4-acd0-cbac1f2fc6c1_isrForCrosstalkSources_855414_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d1bc0f8e-08ed-4ef4-acd0-cbac1f2fc6c1_isrForCrosstalkSources_855414_15.stderr parsl.dataflow.dflow DEBUG: Task 2448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2448 has memoization hash 8ae227f9b6745ced4776ababed6c73cf parsl.dataflow.memoization INFO: Task 2448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2448 try 0 launched on executor multi with executor id 2449 parsl.dataflow.dflow INFO: Standard output for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5a3c351a-7181-4799-a856-720fe28481b8_isrForCrosstalkSources_855404_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5a3c351a-7181-4799-a856-720fe28481b8_isrForCrosstalkSources_855404_4.stderr parsl.dataflow.dflow DEBUG: Task 2449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2449 has memoization hash 703a82e3505d47e42bc093c6af17c028 parsl.dataflow.memoization INFO: Task 2449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964139 proc_lsst.multi INFO: [multi] job_id 16964139 2024-03-09 14:51:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-09 14:51:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964139 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964139 parsl.executors.status_handling DEBUG: Launched block 113 on executor multi with job ID 16964139 parsl.executors.status_handling INFO: Allocated block ID 114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-114 2024-03-09 14:51:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-09 14:51:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2449 try 0 launched on executor multi with executor id 2450 parsl.dataflow.dflow INFO: Standard output for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/55da655a-a87c-487a-9675-979f97a07ab4_isrForCrosstalkSources_855438_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/55da655a-a87c-487a-9675-979f97a07ab4_isrForCrosstalkSources_855438_15.stderr parsl.dataflow.dflow DEBUG: Task 2450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2450 has memoization hash 4538a8ee3ddf7947a041f2a57adf0f3a parsl.dataflow.memoization INFO: Task 2450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6675bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2450 try 0 launched on executor multi with executor id 2451 parsl.dataflow.dflow INFO: Standard output for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e9562881-01aa-4786-995d-67ffd1280231_isrForCrosstalkSources_855414_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e9562881-01aa-4786-995d-67ffd1280231_isrForCrosstalkSources_855414_26.stderr parsl.dataflow.dflow DEBUG: Task 2451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2451 has memoization hash 9ba89ab6afdc2dcea4d48ab8a36ebe49 parsl.dataflow.memoization INFO: Task 2451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2451 try 0 launched on executor multi with executor id 2452 parsl.dataflow.dflow INFO: Standard output for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/099c71a8-7b63-4f6c-a910-41dd2896c344_isrForCrosstalkSources_855423_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/099c71a8-7b63-4f6c-a910-41dd2896c344_isrForCrosstalkSources_855423_5.stderr parsl.dataflow.dflow DEBUG: Task 2452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2452 has memoization hash b10e261ea53cf1b2470e66978a607a3f parsl.dataflow.memoization INFO: Task 2452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2452 try 0 launched on executor multi with executor id 2453 parsl.dataflow.dflow INFO: Standard output for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1a865d0e-8843-4078-aeeb-4a7b999e3ae2_isrForCrosstalkSources_855425_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1a865d0e-8843-4078-aeeb-4a7b999e3ae2_isrForCrosstalkSources_855425_10.stderr parsl.dataflow.dflow DEBUG: Task 2453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2453 has memoization hash e1df770d16f8e1b5b557e9c959350ba2 parsl.dataflow.memoization INFO: Task 2453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2453 try 0 launched on executor multi with executor id 2454 parsl.dataflow.dflow INFO: Standard output for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b5828a9-0d70-4314-ad78-9fbbe78ea600_isrForCrosstalkSources_855416_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b5828a9-0d70-4314-ad78-9fbbe78ea600_isrForCrosstalkSources_855416_13.stderr parsl.dataflow.dflow DEBUG: Task 2454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2454 has memoization hash eceaa265d5e5c07ad9c8523eac85b9ef parsl.dataflow.memoization INFO: Task 2454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2454 try 0 launched on executor multi with executor id 2455 parsl.dataflow.dflow INFO: Standard output for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/d6ca3a75-3d6d-4b9c-8c3f-22db40d261ab_isrForCrosstalkSources_855427_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/d6ca3a75-3d6d-4b9c-8c3f-22db40d261ab_isrForCrosstalkSources_855427_57.stderr parsl.dataflow.dflow DEBUG: Task 2455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2455 has memoization hash 3bd1866de0d46af54577aef2b44c519e parsl.dataflow.memoization INFO: Task 2455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2455 try 0 launched on executor multi with executor id 2456 parsl.dataflow.dflow INFO: Standard output for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4ae4d2-b3d8-4426-b4df-f6d73f551d7a_isrForCrosstalkSources_855393_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4ae4d2-b3d8-4426-b4df-f6d73f551d7a_isrForCrosstalkSources_855393_38.stderr parsl.dataflow.dflow DEBUG: Task 2456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2456 has memoization hash ee816bb85f41156d74732741f80d21b4 parsl.dataflow.memoization INFO: Task 2456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2456 try 0 launched on executor multi with executor id 2457 parsl.dataflow.dflow INFO: Standard output for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8fcc1070-325a-4df1-98f1-6ede50274f23_isrForCrosstalkSources_855419_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8fcc1070-325a-4df1-98f1-6ede50274f23_isrForCrosstalkSources_855419_44.stderr parsl.dataflow.dflow DEBUG: Task 2457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2457 has memoization hash 8903409b077290ea44ce841ac0620e6b parsl.dataflow.memoization INFO: Task 2457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2457 try 0 launched on executor multi with executor id 2458 parsl.dataflow.dflow INFO: Standard output for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/6248a671-7ce6-45bd-a3cb-a00cfbeb90d5_isrForCrosstalkSources_855405_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/6248a671-7ce6-45bd-a3cb-a00cfbeb90d5_isrForCrosstalkSources_855405_53.stderr parsl.dataflow.dflow DEBUG: Task 2458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2458 has memoization hash af6ff7320dd2c3836c3feeaae3322fd9 parsl.dataflow.memoization INFO: Task 2458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2458 try 0 launched on executor multi with executor id 2459 parsl.dataflow.dflow INFO: Standard output for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d17c8be4-bc84-419d-a641-c0efa38131ba_isrForCrosstalkSources_855397_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d17c8be4-bc84-419d-a641-c0efa38131ba_isrForCrosstalkSources_855397_9.stderr parsl.dataflow.dflow DEBUG: Task 2459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2459 has memoization hash a3b703c3eaaa41797179b87232f487b2 parsl.dataflow.memoization INFO: Task 2459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2459 try 0 launched on executor multi with executor id 2460 parsl.dataflow.dflow INFO: Standard output for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/42380226-790d-42cb-93a7-3c10c32ba870_isrForCrosstalkSources_855379_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/42380226-790d-42cb-93a7-3c10c32ba870_isrForCrosstalkSources_855379_4.stderr parsl.dataflow.dflow DEBUG: Task 2460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2460 has memoization hash 3d1417b018e2258395ec84cc2dab7b9e parsl.dataflow.memoization INFO: Task 2460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2460 try 0 launched on executor multi with executor id 2461 parsl.dataflow.dflow INFO: Standard output for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f527f74e-d0bd-4e3a-96ff-a2634bf84ce2_isrForCrosstalkSources_855420_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f527f74e-d0bd-4e3a-96ff-a2634bf84ce2_isrForCrosstalkSources_855420_31.stderr parsl.dataflow.dflow DEBUG: Task 2461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2461 has memoization hash 3ed54e5fc3bb933b93429f95423884c4 parsl.dataflow.memoization INFO: Task 2461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2461 try 0 launched on executor multi with executor id 2462 parsl.dataflow.dflow INFO: Standard output for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6ab4bb0c-0cd9-4e67-8079-7500138d4ba2_isrForCrosstalkSources_855413_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6ab4bb0c-0cd9-4e67-8079-7500138d4ba2_isrForCrosstalkSources_855413_44.stderr parsl.dataflow.dflow DEBUG: Task 2462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2462 has memoization hash c1dcd95cc77bf88876beb939fad31150 parsl.dataflow.memoization INFO: Task 2462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2462 try 0 launched on executor multi with executor id 2463 parsl.dataflow.dflow INFO: Standard output for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/70778215-8cae-4b16-9159-ad176360662a_isrForCrosstalkSources_855438_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/70778215-8cae-4b16-9159-ad176360662a_isrForCrosstalkSources_855438_6.stderr parsl.dataflow.dflow DEBUG: Task 2463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2463 has memoization hash 721303b0fa3e5986ac1a7d175c2d6717 parsl.dataflow.memoization INFO: Task 2463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2463 try 0 launched on executor multi with executor id 2464 parsl.dataflow.dflow INFO: Standard output for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9a60ddd8-13bb-42b6-a605-f4cb0e3da924_isrForCrosstalkSources_855452_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9a60ddd8-13bb-42b6-a605-f4cb0e3da924_isrForCrosstalkSources_855452_15.stderr parsl.dataflow.dflow DEBUG: Task 2464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2464 has memoization hash feaf81ccc31f7d4859d40c18ac82073d parsl.dataflow.memoization INFO: Task 2464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2464 try 0 launched on executor multi with executor id 2465 parsl.dataflow.dflow INFO: Standard output for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dc164cad-064c-4de9-b168-a3cab107cf7c_isrForCrosstalkSources_855387_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dc164cad-064c-4de9-b168-a3cab107cf7c_isrForCrosstalkSources_855387_10.stderr parsl.dataflow.dflow DEBUG: Task 2465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2465 has memoization hash 71409be9f5fb20a0ecfdb622a62d3216 parsl.dataflow.memoization INFO: Task 2465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2465 try 0 launched on executor multi with executor id 2466 parsl.dataflow.dflow INFO: Standard output for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a76724ee-57e5-48b2-9577-613021527946_isrForCrosstalkSources_855405_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a76724ee-57e5-48b2-9577-613021527946_isrForCrosstalkSources_855405_24.stderr parsl.dataflow.dflow DEBUG: Task 2466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2466 has memoization hash 0ceb96c3cf219e92c32f08446e61a8ca parsl.dataflow.memoization INFO: Task 2466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2466 try 0 launched on executor multi with executor id 2467 parsl.dataflow.dflow INFO: Standard output for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6a460e78-92db-4269-b7e3-d884088a2bb3_isrForCrosstalkSources_855387_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6a460e78-92db-4269-b7e3-d884088a2bb3_isrForCrosstalkSources_855387_30.stderr parsl.dataflow.dflow DEBUG: Task 2467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2467 has memoization hash 24471df705bab1e5bbfe4599382d0605 parsl.dataflow.memoization INFO: Task 2467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2467 try 0 launched on executor multi with executor id 2468 parsl.dataflow.dflow INFO: Standard output for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6c92e573-fc6e-4440-a1e4-2c1e16101cfc_isrForCrosstalkSources_855415_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6c92e573-fc6e-4440-a1e4-2c1e16101cfc_isrForCrosstalkSources_855415_9.stderr parsl.dataflow.dflow DEBUG: Task 2468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2468 has memoization hash 73624d2385559090fa77acb775b69cfb parsl.dataflow.memoization INFO: Task 2468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2468 try 0 launched on executor multi with executor id 2469 parsl.dataflow.dflow INFO: Standard output for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a07909f5-2d3a-41a3-add5-e40304d2b1e1_isrForCrosstalkSources_855444_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a07909f5-2d3a-41a3-add5-e40304d2b1e1_isrForCrosstalkSources_855444_30.stderr parsl.dataflow.dflow DEBUG: Task 2469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2469 has memoization hash 5d004663c76018c595f7a4cc8832fb9a parsl.dataflow.memoization INFO: Task 2469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2469 try 0 launched on executor multi with executor id 2470 parsl.dataflow.dflow INFO: Standard output for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/9c5be387-9e0c-4997-88ef-7103470f7243_isrForCrosstalkSources_855407_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/9c5be387-9e0c-4997-88ef-7103470f7243_isrForCrosstalkSources_855407_21.stderr parsl.dataflow.dflow DEBUG: Task 2470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2470 has memoization hash 492e4932e33871d9ec16cc0d313fdba0 parsl.dataflow.memoization INFO: Task 2470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2470 try 0 launched on executor multi with executor id 2471 parsl.dataflow.dflow INFO: Standard output for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/00d102e9-0340-4dc9-9c0a-f341531d50db_isrForCrosstalkSources_855410_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/00d102e9-0340-4dc9-9c0a-f341531d50db_isrForCrosstalkSources_855410_57.stderr parsl.dataflow.dflow DEBUG: Task 2471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2471 has memoization hash d3888d5be948d657550b3139963c6fae parsl.dataflow.memoization INFO: Task 2471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2471 try 0 launched on executor multi with executor id 2472 parsl.dataflow.dflow INFO: Standard output for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/befe6a48-45cc-4955-a883-c614c72ff450_isrForCrosstalkSources_855401_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/befe6a48-45cc-4955-a883-c614c72ff450_isrForCrosstalkSources_855401_26.stderr parsl.dataflow.dflow DEBUG: Task 2472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2472 has memoization hash b120ecb8f0c14067e63f64b4a07e5fee parsl.dataflow.memoization INFO: Task 2472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2472 try 0 launched on executor multi with executor id 2473 parsl.dataflow.dflow INFO: Standard output for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6c04b816-b521-422b-882b-6802a365adcb_isrForCrosstalkSources_855393_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6c04b816-b521-422b-882b-6802a365adcb_isrForCrosstalkSources_855393_3.stderr parsl.dataflow.dflow DEBUG: Task 2473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2473 has memoization hash 11a85039561ef0106e28d30ba575fe8f parsl.dataflow.memoization INFO: Task 2473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2473 try 0 launched on executor multi with executor id 2474 parsl.dataflow.dflow INFO: Standard output for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/daf91724-fe72-44ba-896e-acb5709b9939_isrForCrosstalkSources_855406_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/daf91724-fe72-44ba-896e-acb5709b9939_isrForCrosstalkSources_855406_56.stderr parsl.dataflow.dflow DEBUG: Task 2474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2474 has memoization hash aa9ed632fd6b712051e76e4778920aab parsl.dataflow.memoization INFO: Task 2474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2474 try 0 launched on executor multi with executor id 2475 parsl.dataflow.dflow INFO: Standard output for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/91c156dd-9595-4234-b2d0-5ed66dff2d83_isrForCrosstalkSources_855439_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/91c156dd-9595-4234-b2d0-5ed66dff2d83_isrForCrosstalkSources_855439_16.stderr parsl.dataflow.dflow DEBUG: Task 2475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2475 has memoization hash 9c79cb39dfbeab682a4dedc9d2677e1f parsl.dataflow.memoization INFO: Task 2475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2475 try 0 launched on executor multi with executor id 2476 parsl.dataflow.dflow INFO: Standard output for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1b66172-5ef1-43d6-8ce6-3d032ac92069_isrForCrosstalkSources_855458_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1b66172-5ef1-43d6-8ce6-3d032ac92069_isrForCrosstalkSources_855458_58.stderr parsl.dataflow.dflow DEBUG: Task 2476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2476 has memoization hash 92fe20366010f7a3fd16ecec93617d8a parsl.dataflow.memoization INFO: Task 2476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2476 try 0 launched on executor multi with executor id 2477 parsl.dataflow.dflow INFO: Standard output for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/153de814-c0ae-43e3-b93a-0fe8d033202f_isrForCrosstalkSources_855447_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/153de814-c0ae-43e3-b93a-0fe8d033202f_isrForCrosstalkSources_855447_46.stderr parsl.dataflow.dflow DEBUG: Task 2477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2477 has memoization hash 5fb6de448581f6a8880387256a5cc30c parsl.dataflow.memoization INFO: Task 2477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:16 proc_lsst.multi:170 [INFO] [multi] job_id 16964140 proc_lsst.multi INFO: [multi] job_id 16964140 2024-03-09 14:51:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-09 14:51:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964140 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964140 parsl.executors.status_handling DEBUG: Launched block 114 on executor multi with job ID 16964140 parsl.executors.status_handling INFO: Allocated block ID 115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-115 2024-03-09 14:51:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2477 try 0 launched on executor multi with executor id 2478 parsl.dataflow.dflow INFO: Standard output for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d53d0048-e981-4f91-a15d-ff1f705a4292_isrForCrosstalkSources_855410_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d53d0048-e981-4f91-a15d-ff1f705a4292_isrForCrosstalkSources_855410_33.stderr parsl.dataflow.dflow DEBUG: Task 2478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2478 has memoization hash b08e9632dca68836caf6e84e6d79d5a6 parsl.dataflow.memoization INFO: Task 2478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2478 try 0 launched on executor multi with executor id 2479 parsl.dataflow.dflow INFO: Standard output for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3640b6-802e-48f5-a463-d4166e51f79e_isrForCrosstalkSources_855445_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3640b6-802e-48f5-a463-d4166e51f79e_isrForCrosstalkSources_855445_38.stderr parsl.dataflow.dflow DEBUG: Task 2479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2479 has memoization hash 7a08a8d9c2a60c196df7cf7c7b22f9dd parsl.dataflow.memoization INFO: Task 2479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2479 try 0 launched on executor multi with executor id 2480 parsl.dataflow.dflow INFO: Standard output for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f59a73ed-dd66-4bd0-ba00-a64755fd60aa_isrForCrosstalkSources_855416_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f59a73ed-dd66-4bd0-ba00-a64755fd60aa_isrForCrosstalkSources_855416_62.stderr parsl.dataflow.dflow DEBUG: Task 2480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2480 has memoization hash 5e3990d6bded9b9cd625cfa656c2068e parsl.dataflow.memoization INFO: Task 2480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2480 try 0 launched on executor multi with executor id 2481 parsl.dataflow.dflow INFO: Standard output for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/18c587e3-9859-4c41-b060-9dc0042cdd5e_isrForCrosstalkSources_855430_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/18c587e3-9859-4c41-b060-9dc0042cdd5e_isrForCrosstalkSources_855430_25.stderr parsl.dataflow.dflow DEBUG: Task 2481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2481 has memoization hash ab79d843b69bf6566039a0adebf370ee parsl.dataflow.memoization INFO: Task 2481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2481 try 0 launched on executor multi with executor id 2482 parsl.dataflow.dflow INFO: Standard output for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9719a20c-4cdf-47a5-84e0-cf3f842d0226_isrForCrosstalkSources_855446_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9719a20c-4cdf-47a5-84e0-cf3f842d0226_isrForCrosstalkSources_855446_53.stderr parsl.dataflow.dflow DEBUG: Task 2482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2482 has memoization hash 6e667942f988c9248ef57f0cb663ca5a parsl.dataflow.memoization INFO: Task 2482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2482 try 0 launched on executor multi with executor id 2483 parsl.dataflow.dflow INFO: Standard output for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c63893d9-f769-4264-a6e6-6780d6e0ac28_isrForCrosstalkSources_855434_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c63893d9-f769-4264-a6e6-6780d6e0ac28_isrForCrosstalkSources_855434_13.stderr parsl.dataflow.dflow DEBUG: Task 2483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2483 has memoization hash 71a2e766b72850e1da2e1fc2a2a59eb9 parsl.dataflow.memoization INFO: Task 2483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2483 try 0 launched on executor multi with executor id 2484 parsl.dataflow.dflow INFO: Standard output for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/73270931-a8ce-4493-9da6-6306a4f01ac6_isrForCrosstalkSources_855457_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/73270931-a8ce-4493-9da6-6306a4f01ac6_isrForCrosstalkSources_855457_36.stderr parsl.dataflow.dflow DEBUG: Task 2484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2484 has memoization hash 9f721278baed680998913815c0be0028 parsl.dataflow.memoization INFO: Task 2484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2484 try 0 launched on executor multi with executor id 2485 parsl.dataflow.dflow INFO: Standard output for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/048edd6e-75e7-463f-ba26-4fe51d2a0d82_isrForCrosstalkSources_855422_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/048edd6e-75e7-463f-ba26-4fe51d2a0d82_isrForCrosstalkSources_855422_53.stderr parsl.dataflow.dflow DEBUG: Task 2485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2485 has memoization hash d23b7147748d4fa83a6a02dea8550e78 parsl.dataflow.memoization INFO: Task 2485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2485 try 0 launched on executor multi with executor id 2486 parsl.dataflow.dflow INFO: Standard output for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6068ca33-d2fc-44a9-9370-ef0405c01a37_isrForCrosstalkSources_855383_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6068ca33-d2fc-44a9-9370-ef0405c01a37_isrForCrosstalkSources_855383_7.stderr parsl.dataflow.dflow DEBUG: Task 2486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2486 has memoization hash ad5db8d5ed38f2c201c9a66fccfe9948 parsl.dataflow.memoization INFO: Task 2486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2486 try 0 launched on executor multi with executor id 2487 parsl.dataflow.dflow INFO: Standard output for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a2d6a55f-9069-4b67-801e-3bff01e8e43f_isrForCrosstalkSources_855427_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a2d6a55f-9069-4b67-801e-3bff01e8e43f_isrForCrosstalkSources_855427_32.stderr parsl.dataflow.dflow DEBUG: Task 2487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2487 has memoization hash 0378385c8a14977fa3134b94dea1ad7b parsl.dataflow.memoization INFO: Task 2487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2487 try 0 launched on executor multi with executor id 2488 parsl.dataflow.dflow INFO: Standard output for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/be97d3db-1c07-47c5-bc68-686b8768814f_isrForCrosstalkSources_855445_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/be97d3db-1c07-47c5-bc68-686b8768814f_isrForCrosstalkSources_855445_46.stderr parsl.dataflow.dflow DEBUG: Task 2488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2488 has memoization hash 558d9b0df03e309909872e936b40db25 parsl.dataflow.memoization INFO: Task 2488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2488 try 0 launched on executor multi with executor id 2489 parsl.dataflow.dflow INFO: Standard output for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5517cb26-d4c2-479a-a648-9aca657b82c3_isrForCrosstalkSources_855453_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5517cb26-d4c2-479a-a648-9aca657b82c3_isrForCrosstalkSources_855453_26.stderr parsl.dataflow.dflow DEBUG: Task 2489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2489 has memoization hash d19c0db96330c6a31788e0c971aa48dc parsl.dataflow.memoization INFO: Task 2489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2489 try 0 launched on executor multi with executor id 2490 parsl.dataflow.dflow INFO: Standard output for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/26dfdd45-d715-4bef-88f7-4130d9bae02f_isrForCrosstalkSources_855383_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/26dfdd45-d715-4bef-88f7-4130d9bae02f_isrForCrosstalkSources_855383_16.stderr parsl.dataflow.dflow DEBUG: Task 2490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2490 has memoization hash c695f9b1bac22d9f9c138afb924ba18c parsl.dataflow.memoization INFO: Task 2490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2490 try 0 launched on executor multi with executor id 2491 parsl.dataflow.dflow INFO: Standard output for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/7f6eef0f-2c95-4cac-a911-0f14a1e35a35_isrForCrosstalkSources_855417_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/7f6eef0f-2c95-4cac-a911-0f14a1e35a35_isrForCrosstalkSources_855417_36.stderr parsl.dataflow.dflow DEBUG: Task 2491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2491 has memoization hash 62d32c16d454ed1dd4b8902ee5668ff3 parsl.dataflow.memoization INFO: Task 2491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2491 try 0 launched on executor multi with executor id 2492 parsl.dataflow.dflow INFO: Standard output for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/bc5eb85b-bd76-46c3-93c1-087ba7510987_isrForCrosstalkSources_855451_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/bc5eb85b-bd76-46c3-93c1-087ba7510987_isrForCrosstalkSources_855451_2.stderr parsl.dataflow.dflow DEBUG: Task 2492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2492 has memoization hash 18d09e7f395e54f8c0da165e2f08a2e9 parsl.dataflow.memoization INFO: Task 2492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2492 try 0 launched on executor multi with executor id 2493 parsl.dataflow.dflow INFO: Standard output for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a22df2fd-1c69-4db8-bca2-d596a0d151e5_isrForCrosstalkSources_855445_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a22df2fd-1c69-4db8-bca2-d596a0d151e5_isrForCrosstalkSources_855445_34.stderr parsl.dataflow.dflow DEBUG: Task 2493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2493 has memoization hash 5b92533bbfd32abc862103167ca9f4cf parsl.dataflow.memoization INFO: Task 2493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2493 try 0 launched on executor multi with executor id 2494 parsl.dataflow.dflow INFO: Standard output for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c378fb7b-18c3-4994-8692-56a91122f305_isrForCrosstalkSources_855448_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c378fb7b-18c3-4994-8692-56a91122f305_isrForCrosstalkSources_855448_16.stderr parsl.dataflow.dflow DEBUG: Task 2494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2494 has memoization hash 6fe620ceb8ae78f1f6a1af2832aead76 parsl.dataflow.memoization INFO: Task 2494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2494 try 0 launched on executor multi with executor id 2495 parsl.dataflow.dflow INFO: Standard output for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9e48b560-a799-4391-9e24-592141702347_isrForCrosstalkSources_855389_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9e48b560-a799-4391-9e24-592141702347_isrForCrosstalkSources_855389_36.stderr parsl.dataflow.dflow DEBUG: Task 2495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2495 has memoization hash 0ef128c02e501e513fc1b20fe7c8ddbd parsl.dataflow.memoization INFO: Task 2495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2495 try 0 launched on executor multi with executor id 2496 parsl.dataflow.dflow INFO: Standard output for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/308799e6-c0ac-4305-8a69-2bd354aea7d4_isrForCrosstalkSources_855443_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/308799e6-c0ac-4305-8a69-2bd354aea7d4_isrForCrosstalkSources_855443_62.stderr parsl.dataflow.dflow DEBUG: Task 2496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2496 has memoization hash dfb5eff276bfc9efb70f07c1eaf8a4fc parsl.dataflow.memoization INFO: Task 2496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2496 try 0 launched on executor multi with executor id 2497 parsl.dataflow.dflow INFO: Standard output for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b1e4fac6-5457-4fe4-ab8d-456bdf4cc0bd_isrForCrosstalkSources_855411_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b1e4fac6-5457-4fe4-ab8d-456bdf4cc0bd_isrForCrosstalkSources_855411_8.stderr parsl.dataflow.dflow DEBUG: Task 2497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2497 has memoization hash cbcd8f96cc18e192607f3f880ade14a8 parsl.dataflow.memoization INFO: Task 2497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2497 try 0 launched on executor multi with executor id 2498 parsl.dataflow.dflow INFO: Standard output for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/95d46ed0-9af5-46d4-bbd1-043f23f4094a_isrForCrosstalkSources_855419_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/95d46ed0-9af5-46d4-bbd1-043f23f4094a_isrForCrosstalkSources_855419_39.stderr parsl.dataflow.dflow DEBUG: Task 2498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2498 has memoization hash c7fb2c73d6cc36c904d2fe8d69f4c13f parsl.dataflow.memoization INFO: Task 2498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2498 try 0 launched on executor multi with executor id 2499 parsl.dataflow.dflow INFO: Standard output for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e2d78afc-4e74-4dc2-95c3-17152a03d1d0_isrForCrosstalkSources_855399_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e2d78afc-4e74-4dc2-95c3-17152a03d1d0_isrForCrosstalkSources_855399_44.stderr parsl.dataflow.dflow DEBUG: Task 2499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2499 has memoization hash e956e803cc58bdfb81f3bc5a2ed1d410 parsl.dataflow.memoization INFO: Task 2499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2499 try 0 launched on executor multi with executor id 2500 parsl.dataflow.dflow INFO: Standard output for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a809978a-640f-4d43-8da0-dd62843f55ec_isrForCrosstalkSources_855422_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a809978a-640f-4d43-8da0-dd62843f55ec_isrForCrosstalkSources_855422_40.stderr parsl.dataflow.dflow DEBUG: Task 2500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2500 has memoization hash 45638ac04310cfbb6fd519c841081325 parsl.dataflow.memoization INFO: Task 2500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2500 try 0 launched on executor multi with executor id 2501 parsl.dataflow.dflow INFO: Standard output for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/db417d29-03ca-46e3-a757-57ca676c08e3_isrForCrosstalkSources_855403_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/db417d29-03ca-46e3-a757-57ca676c08e3_isrForCrosstalkSources_855403_4.stderr parsl.dataflow.dflow DEBUG: Task 2501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2501 has memoization hash 5561ff9c9e618afddd4599efc26b6ce7 parsl.dataflow.memoization INFO: Task 2501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2501 try 0 launched on executor multi with executor id 2502 parsl.dataflow.dflow INFO: Standard output for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/cf3286dd-5d6d-45bf-bdcc-e3cb0d7c1213_isrForCrosstalkSources_855389_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/cf3286dd-5d6d-45bf-bdcc-e3cb0d7c1213_isrForCrosstalkSources_855389_29.stderr parsl.dataflow.dflow DEBUG: Task 2502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2502 has memoization hash ebce370c9afc8792f056b025d238882c parsl.dataflow.memoization INFO: Task 2502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2502 try 0 launched on executor multi with executor id 2503 parsl.dataflow.dflow INFO: Standard output for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f079f0b8-e0b4-4482-94f4-e85371bb9bf5_isrForCrosstalkSources_855379_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f079f0b8-e0b4-4482-94f4-e85371bb9bf5_isrForCrosstalkSources_855379_29.stderr parsl.dataflow.dflow DEBUG: Task 2503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2503 has memoization hash 6b0465a07dff81c034cbb1034c79ef19 parsl.dataflow.memoization INFO: Task 2503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2503 try 0 launched on executor multi with executor id 2504 parsl.dataflow.dflow INFO: Standard output for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/acd859d9-16d1-4e43-ac46-6116d78fce00_isrForCrosstalkSources_855396_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/acd859d9-16d1-4e43-ac46-6116d78fce00_isrForCrosstalkSources_855396_59.stderr parsl.dataflow.dflow DEBUG: Task 2504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2504 has memoization hash 75d98e4fdbdce11f3da67e95659c87ad parsl.dataflow.memoization INFO: Task 2504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2504 try 0 launched on executor multi with executor id 2505 parsl.dataflow.dflow INFO: Standard output for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d0207ac-7596-4fa4-a801-5568395b4324_isrForCrosstalkSources_855388_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d0207ac-7596-4fa4-a801-5568395b4324_isrForCrosstalkSources_855388_57.stderr parsl.dataflow.dflow DEBUG: Task 2505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2505 has memoization hash 640e1986b169b5711287932a455e5da8 parsl.dataflow.memoization INFO: Task 2505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2505 try 0 launched on executor multi with executor id 2506 parsl.dataflow.dflow INFO: Standard output for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7b1a3c5a-b52e-40e9-ac05-15a7ee1a38f4_isrForCrosstalkSources_855421_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7b1a3c5a-b52e-40e9-ac05-15a7ee1a38f4_isrForCrosstalkSources_855421_2.stderr parsl.dataflow.dflow DEBUG: Task 2506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2506 has memoization hash 13dcf499580181b65bd46a70ec9b2ef5 parsl.dataflow.memoization INFO: Task 2506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2506 try 0 launched on executor multi with executor id 2507 parsl.dataflow.dflow INFO: Standard output for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/f0258b47-8a9e-473d-bf7d-d3ce125c8638_isrForCrosstalkSources_855386_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/f0258b47-8a9e-473d-bf7d-d3ce125c8638_isrForCrosstalkSources_855386_52.stderr parsl.dataflow.dflow DEBUG: Task 2507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2507 has memoization hash e6e921665ed0b7daa8b7f7310652a681 parsl.dataflow.memoization INFO: Task 2507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2507 try 0 launched on executor multi with executor id 2508 parsl.dataflow.dflow INFO: Standard output for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ca13b301-1b74-4412-ae1e-197453e79fa8_isrForCrosstalkSources_855447_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ca13b301-1b74-4412-ae1e-197453e79fa8_isrForCrosstalkSources_855447_49.stderr parsl.dataflow.dflow DEBUG: Task 2508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2508 has memoization hash da74dc92fb8d4de97bd314a1c118d6f8 parsl.dataflow.memoization INFO: Task 2508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2508 try 0 launched on executor multi with executor id 2509 parsl.dataflow.dflow INFO: Standard output for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b8d2b781-4cc1-4721-b5a3-e5354e92873b_isrForCrosstalkSources_855425_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b8d2b781-4cc1-4721-b5a3-e5354e92873b_isrForCrosstalkSources_855425_26.stderr parsl.dataflow.dflow DEBUG: Task 2509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2509 has memoization hash 7c8b0638154a5691c6f25fe71c6ddcdf parsl.dataflow.memoization INFO: Task 2509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2509 try 0 launched on executor multi with executor id 2510 parsl.dataflow.dflow INFO: Standard output for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ebe987a-4fa4-42e4-86a6-e3358006a93b_isrForCrosstalkSources_855401_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ebe987a-4fa4-42e4-86a6-e3358006a93b_isrForCrosstalkSources_855401_11.stderr parsl.dataflow.dflow DEBUG: Task 2510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2510 has memoization hash 312a5b1eb0b3ea3b0452491e869ff339 parsl.dataflow.memoization INFO: Task 2510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2510 try 0 launched on executor multi with executor id 2511 parsl.dataflow.dflow INFO: Standard output for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c85b9d48-bc1d-4b3b-b650-94ec3bb66669_isrForCrosstalkSources_855446_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c85b9d48-bc1d-4b3b-b650-94ec3bb66669_isrForCrosstalkSources_855446_11.stderr parsl.dataflow.dflow DEBUG: Task 2511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2511 has memoization hash fd0f00ccf9746fe600f29b10719dadfa parsl.dataflow.memoization INFO: Task 2511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2511 try 0 launched on executor multi with executor id 2512 parsl.dataflow.dflow INFO: Standard output for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6cfa263b-643b-4c5a-a072-0d2851665d8e_isrForCrosstalkSources_855392_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6cfa263b-643b-4c5a-a072-0d2851665d8e_isrForCrosstalkSources_855392_42.stderr parsl.dataflow.dflow DEBUG: Task 2512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2512 has memoization hash 104be2bfaa340ac97eee5bf66847bac0 parsl.dataflow.memoization INFO: Task 2512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:16 proc_lsst.multi:170 [INFO] [multi] job_id 16964141 proc_lsst.multi INFO: [multi] job_id 16964141 2024-03-09 14:51:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-09 14:51:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964141 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964141 parsl.executors.status_handling DEBUG: Launched block 115 on executor multi with job ID 16964141 parsl.executors.status_handling INFO: Allocated block ID 116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-116 2024-03-09 14:51:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-09 14:51:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2512 try 0 launched on executor multi with executor id 2513 parsl.dataflow.dflow INFO: Standard output for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1363d343-3add-405d-b5b3-810662c1bd93_isrForCrosstalkSources_855445_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1363d343-3add-405d-b5b3-810662c1bd93_isrForCrosstalkSources_855445_15.stderr parsl.dataflow.dflow DEBUG: Task 2513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2513 has memoization hash add7e7c64112ef59b4f4d2329c05965d parsl.dataflow.memoization INFO: Task 2513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2513 try 0 launched on executor multi with executor id 2514 parsl.dataflow.dflow INFO: Standard output for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c654e4b3-6c7f-4fb5-a63b-84045b2ac2b2_isrForCrosstalkSources_855413_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c654e4b3-6c7f-4fb5-a63b-84045b2ac2b2_isrForCrosstalkSources_855413_61.stderr parsl.dataflow.dflow DEBUG: Task 2514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2514 has memoization hash abeb14d00572a01992e71be119ef9907 parsl.dataflow.memoization INFO: Task 2514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2514 try 0 launched on executor multi with executor id 2515 parsl.dataflow.dflow INFO: Standard output for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/71c3a46e-10f1-4768-8454-4cfba64a8531_isrForCrosstalkSources_855415_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/71c3a46e-10f1-4768-8454-4cfba64a8531_isrForCrosstalkSources_855415_6.stderr parsl.dataflow.dflow DEBUG: Task 2515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2515 has memoization hash 80a2c789d6cbd1906ed4857e0c420dbe parsl.dataflow.memoization INFO: Task 2515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2515 try 0 launched on executor multi with executor id 2516 parsl.dataflow.dflow INFO: Standard output for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b83c57f3-9e98-4e91-b1ec-d5c5eddccbd9_isrForCrosstalkSources_855452_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b83c57f3-9e98-4e91-b1ec-d5c5eddccbd9_isrForCrosstalkSources_855452_45.stderr parsl.dataflow.dflow DEBUG: Task 2516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2516 has memoization hash 56c6558e85ade6c0fc0c05896f4b63cc parsl.dataflow.memoization INFO: Task 2516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:17 proc_lsst.multi:170 [INFO] [multi] job_id 16964142 proc_lsst.multi INFO: [multi] job_id 16964142 2024-03-09 14:51:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-09 14:51:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964142 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964142 parsl.executors.status_handling DEBUG: Launched block 116 on executor multi with job ID 16964142 parsl.executors.status_handling INFO: Allocated block ID 117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-117 2024-03-09 14:51:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 2024-03-09 14:51:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-09 14:51:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2516 try 0 launched on executor multi with executor id 2517 parsl.dataflow.dflow INFO: Standard output for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/83d00e3e-1408-4cb9-a8c4-743b90675aaf_isrForCrosstalkSources_855398_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/83d00e3e-1408-4cb9-a8c4-743b90675aaf_isrForCrosstalkSources_855398_45.stderr parsl.dataflow.dflow DEBUG: Task 2517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2517 has memoization hash 085cff9b5d5053df1b6786ea0b59355c parsl.dataflow.memoization INFO: Task 2517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2517 try 0 launched on executor multi with executor id 2518 parsl.dataflow.dflow INFO: Standard output for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/35e1ae4c-1d11-4de9-a669-f55b5775d14d_isrForCrosstalkSources_855449_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/35e1ae4c-1d11-4de9-a669-f55b5775d14d_isrForCrosstalkSources_855449_39.stderr parsl.dataflow.dflow DEBUG: Task 2518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2518 has memoization hash 156720f7b75381271b6abe12a8f319d3 parsl.dataflow.memoization INFO: Task 2518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2518 try 0 launched on executor multi with executor id 2519 parsl.dataflow.dflow INFO: Standard output for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/61f39e8d-132d-44a8-b8c7-2bfa62959b54_isrForCrosstalkSources_855424_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/61f39e8d-132d-44a8-b8c7-2bfa62959b54_isrForCrosstalkSources_855424_15.stderr parsl.dataflow.dflow DEBUG: Task 2519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2519 has memoization hash 1fb26bb0afb4c043d6ace63eb4c8c40e parsl.dataflow.memoization INFO: Task 2519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2519 try 0 launched on executor multi with executor id 2520 parsl.dataflow.dflow INFO: Standard output for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/75f20608-51e0-4b1b-bc8d-26f6916767c8_isrForCrosstalkSources_855381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/75f20608-51e0-4b1b-bc8d-26f6916767c8_isrForCrosstalkSources_855381_2.stderr parsl.dataflow.dflow DEBUG: Task 2520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2520 has memoization hash 98a76e78f043975ae14b1a484deac047 parsl.dataflow.memoization INFO: Task 2520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2520 try 0 launched on executor multi with executor id 2521 parsl.dataflow.dflow INFO: Standard output for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/afdf8c8a-6960-42f8-8b3f-304e072b28d3_isrForCrosstalkSources_855447_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/afdf8c8a-6960-42f8-8b3f-304e072b28d3_isrForCrosstalkSources_855447_9.stderr parsl.dataflow.dflow DEBUG: Task 2521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2521 has memoization hash 8e05343ed6e2b5dfb9b2dd1c78ac1b10 parsl.dataflow.memoization INFO: Task 2521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964143 proc_lsst.multi INFO: [multi] job_id 16964143 2024-03-09 14:51:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-09 14:51:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964143 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964143 parsl.executors.status_handling DEBUG: Launched block 117 on executor multi with job ID 16964143 parsl.executors.status_handling INFO: Allocated block ID 118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-118 2024-03-09 14:51:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 2024-03-09 14:51:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-09 14:51:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2521 try 0 launched on executor multi with executor id 2522 parsl.dataflow.dflow INFO: Standard output for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8174c320-144d-47e2-bfe6-7e49c3690198_isrForCrosstalkSources_855428_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8174c320-144d-47e2-bfe6-7e49c3690198_isrForCrosstalkSources_855428_9.stderr parsl.dataflow.dflow DEBUG: Task 2522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2522 has memoization hash 3c29b077337def96f734088d2c8f8d3c parsl.dataflow.memoization INFO: Task 2522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2522 try 0 launched on executor multi with executor id 2523 parsl.dataflow.dflow INFO: Standard output for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/6094fce6-59f7-4deb-a28d-cc0d8a1cd15c_isrForCrosstalkSources_855416_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/6094fce6-59f7-4deb-a28d-cc0d8a1cd15c_isrForCrosstalkSources_855416_10.stderr parsl.dataflow.dflow DEBUG: Task 2523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2523 has memoization hash d8e41e63bfeff6b9cc5169d1a501cd94 parsl.dataflow.memoization INFO: Task 2523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2523 try 0 launched on executor multi with executor id 2524 parsl.dataflow.dflow INFO: Standard output for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/af4ef68a-607c-45ae-b1c5-85d3a8e11342_isrForCrosstalkSources_855386_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/af4ef68a-607c-45ae-b1c5-85d3a8e11342_isrForCrosstalkSources_855386_48.stderr parsl.dataflow.dflow DEBUG: Task 2524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2524 has memoization hash 03edc826b3c68841c1dedb1fe7b63137 parsl.dataflow.memoization INFO: Task 2524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2524 try 0 launched on executor multi with executor id 2525 parsl.dataflow.dflow INFO: Standard output for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/918bca7c-8929-459c-b78a-ad31137193da_isrForCrosstalkSources_855447_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/918bca7c-8929-459c-b78a-ad31137193da_isrForCrosstalkSources_855447_43.stderr parsl.dataflow.dflow DEBUG: Task 2525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2525 has memoization hash 3e5726636c68e8b7b932f11efcb8fcc2 parsl.dataflow.memoization INFO: Task 2525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2525 try 0 launched on executor multi with executor id 2526 parsl.dataflow.dflow INFO: Standard output for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/53f9b073-76c6-4198-a0a0-d881bfe2340c_isrForCrosstalkSources_855425_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/53f9b073-76c6-4198-a0a0-d881bfe2340c_isrForCrosstalkSources_855425_31.stderr parsl.dataflow.dflow DEBUG: Task 2526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2526 has memoization hash d97cf74f3372d6ce3d24b3792ff222be parsl.dataflow.memoization INFO: Task 2526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2526 try 0 launched on executor multi with executor id 2527 parsl.dataflow.dflow INFO: Standard output for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/d78ff4f8-50fb-4f30-887f-b6d9c811c738_isrForCrosstalkSources_855396_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/d78ff4f8-50fb-4f30-887f-b6d9c811c738_isrForCrosstalkSources_855396_50.stderr parsl.dataflow.dflow DEBUG: Task 2527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2527 has memoization hash b29bfa713899a935a4b0ce8f724ab58f parsl.dataflow.memoization INFO: Task 2527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964144 proc_lsst.multi INFO: [multi] job_id 16964144 2024-03-09 14:51:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-09 14:51:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964144 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964144 parsl.executors.status_handling DEBUG: Launched block 118 on executor multi with job ID 16964144 parsl.executors.status_handling INFO: Allocated block ID 119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-119 2024-03-09 14:51:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2527 try 0 launched on executor multi with executor id 2528 parsl.dataflow.dflow INFO: Standard output for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c6804db7-8254-4fdc-ac96-b56bdff6c1db_isrForCrosstalkSources_855448_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c6804db7-8254-4fdc-ac96-b56bdff6c1db_isrForCrosstalkSources_855448_27.stderr parsl.dataflow.dflow DEBUG: Task 2528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2528 has memoization hash d2c03eea5fea590986a3356952075faa parsl.dataflow.memoization INFO: Task 2528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2528 try 0 launched on executor multi with executor id 2529 parsl.dataflow.dflow INFO: Standard output for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8ca9fdae-5048-479b-9f5b-bec7f01783f6_isrForCrosstalkSources_855393_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8ca9fdae-5048-479b-9f5b-bec7f01783f6_isrForCrosstalkSources_855393_36.stderr parsl.dataflow.dflow DEBUG: Task 2529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2529 has memoization hash ca00ac30889d6f8a3fa2fa66370755bf parsl.dataflow.memoization INFO: Task 2529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2529 try 0 launched on executor multi with executor id 2530 parsl.dataflow.dflow INFO: Standard output for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8de8c96d-1d9e-49c9-aa46-d85c1605728b_isrForCrosstalkSources_855408_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8de8c96d-1d9e-49c9-aa46-d85c1605728b_isrForCrosstalkSources_855408_3.stderr parsl.dataflow.dflow DEBUG: Task 2530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2530 has memoization hash 6adf35d36c951cdc8182df3e2cfc36d3 parsl.dataflow.memoization INFO: Task 2530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2530 try 0 launched on executor multi with executor id 2531 parsl.dataflow.dflow INFO: Standard output for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/26863c32-ddd8-4a9d-8738-c36f35a3ec0a_isrForCrosstalkSources_855393_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/26863c32-ddd8-4a9d-8738-c36f35a3ec0a_isrForCrosstalkSources_855393_27.stderr parsl.dataflow.dflow DEBUG: Task 2531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2531 has memoization hash 227d484982bfde7210309aec14190c61 parsl.dataflow.memoization INFO: Task 2531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2531 try 0 launched on executor multi with executor id 2532 parsl.dataflow.dflow INFO: Standard output for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/705caf75-d18a-4575-ba7b-e09ae3b71545_isrForCrosstalkSources_855398_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/705caf75-d18a-4575-ba7b-e09ae3b71545_isrForCrosstalkSources_855398_10.stderr parsl.dataflow.dflow DEBUG: Task 2532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2532 has memoization hash f68934e8cba640d2454cd048409e005d parsl.dataflow.memoization INFO: Task 2532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2532 try 0 launched on executor multi with executor id 2533 parsl.dataflow.dflow INFO: Standard output for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/5c841d25-0bbc-457f-976a-a5f8619794fb_isrForCrosstalkSources_855414_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/5c841d25-0bbc-457f-976a-a5f8619794fb_isrForCrosstalkSources_855414_5.stderr parsl.dataflow.dflow DEBUG: Task 2533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2533 has memoization hash 11baad6d41eb53005d5bd6492fc6da25 parsl.dataflow.memoization INFO: Task 2533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964145 proc_lsst.multi INFO: [multi] job_id 16964145 2024-03-09 14:51:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-09 14:51:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964145 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964145 parsl.executors.status_handling DEBUG: Launched block 119 on executor multi with job ID 16964145 parsl.executors.status_handling INFO: Allocated block ID 120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-120 2024-03-09 14:51:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-09 14:51:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2533 try 0 launched on executor multi with executor id 2534 parsl.dataflow.dflow INFO: Standard output for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/cb669df7-0e85-475e-923a-3b9d4ecf05eb_isrForCrosstalkSources_855431_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/cb669df7-0e85-475e-923a-3b9d4ecf05eb_isrForCrosstalkSources_855431_2.stderr parsl.dataflow.dflow DEBUG: Task 2534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2534 has memoization hash dc0f932164cc14bf6ea5d7d73e4dd1fd parsl.dataflow.memoization INFO: Task 2534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2534 try 0 launched on executor multi with executor id 2535 parsl.dataflow.dflow INFO: Standard output for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/cf407e3a-49d1-4b21-b061-3787a143e7da_isrForCrosstalkSources_855404_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/cf407e3a-49d1-4b21-b061-3787a143e7da_isrForCrosstalkSources_855404_55.stderr parsl.dataflow.dflow DEBUG: Task 2535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2535 has memoization hash 49f93c69444e27aa6018e55699eba18c parsl.dataflow.memoization INFO: Task 2535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2535 try 0 launched on executor multi with executor id 2536 parsl.dataflow.dflow INFO: Standard output for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fbb284ae-b300-4eea-8877-7ad0e9904994_isrForCrosstalkSources_855450_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fbb284ae-b300-4eea-8877-7ad0e9904994_isrForCrosstalkSources_855450_10.stderr parsl.dataflow.dflow DEBUG: Task 2536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2536 has memoization hash 170b193c6dffca71431edcb3a4780f83 parsl.dataflow.memoization INFO: Task 2536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2536 try 0 launched on executor multi with executor id 2537 parsl.dataflow.dflow INFO: Standard output for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9714cd7f-d9d2-44c1-a2b7-8b9e3ca49afa_isrForCrosstalkSources_855387_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9714cd7f-d9d2-44c1-a2b7-8b9e3ca49afa_isrForCrosstalkSources_855387_13.stderr parsl.dataflow.dflow DEBUG: Task 2537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2537 has memoization hash 173630ac502011ee1859fa7975452a6d parsl.dataflow.memoization INFO: Task 2537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2537 try 0 launched on executor multi with executor id 2538 parsl.dataflow.dflow INFO: Standard output for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/76e377bf-a14b-497a-b7d2-f1661d4f7546_isrForCrosstalkSources_855428_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/76e377bf-a14b-497a-b7d2-f1661d4f7546_isrForCrosstalkSources_855428_19.stderr parsl.dataflow.dflow DEBUG: Task 2538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2538 has memoization hash f5db64b8b5da2f79591b2ee95a7a43d7 parsl.dataflow.memoization INFO: Task 2538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2538 try 0 launched on executor multi with executor id 2539 parsl.dataflow.dflow INFO: Standard output for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/3908a21e-cd0a-489c-821b-bf4558df7c56_isrForCrosstalkSources_855444_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/3908a21e-cd0a-489c-821b-bf4558df7c56_isrForCrosstalkSources_855444_7.stderr parsl.dataflow.dflow DEBUG: Task 2539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2539 has memoization hash 816770326bee61a9ffc871b16ea81370 parsl.dataflow.memoization INFO: Task 2539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2539 try 0 launched on executor multi with executor id 2540 parsl.dataflow.dflow INFO: Standard output for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/72dd28ea-5ff5-41c1-893e-2dff6343b322_isrForCrosstalkSources_855442_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/72dd28ea-5ff5-41c1-893e-2dff6343b322_isrForCrosstalkSources_855442_43.stderr parsl.dataflow.dflow DEBUG: Task 2540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2540 has memoization hash 3b3fc0043debe5bddcc4fa172ee66fda parsl.dataflow.memoization INFO: Task 2540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2540 try 0 launched on executor multi with executor id 2541 parsl.dataflow.dflow INFO: Standard output for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/edccc128-d39d-4ee9-8b6c-df8703bbc9c6_isrForCrosstalkSources_855450_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/edccc128-d39d-4ee9-8b6c-df8703bbc9c6_isrForCrosstalkSources_855450_20.stderr parsl.dataflow.dflow DEBUG: Task 2541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2541 has memoization hash 6e2e23cc849690e8843347a020017131 parsl.dataflow.memoization INFO: Task 2541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2541 try 0 launched on executor multi with executor id 2542 parsl.dataflow.dflow INFO: Standard output for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/39d206e5-1847-418e-a8a0-3cbb5f67b4cf_isrForCrosstalkSources_855413_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/39d206e5-1847-418e-a8a0-3cbb5f67b4cf_isrForCrosstalkSources_855413_55.stderr parsl.dataflow.dflow DEBUG: Task 2542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2542 has memoization hash 06d7b0857b7cde9df03381673757ebb5 parsl.dataflow.memoization INFO: Task 2542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2542 try 0 launched on executor multi with executor id 2543 parsl.dataflow.dflow INFO: Standard output for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7443119e-463b-42d6-8d79-642463d6aa8e_isrForCrosstalkSources_855440_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7443119e-463b-42d6-8d79-642463d6aa8e_isrForCrosstalkSources_855440_14.stderr parsl.dataflow.dflow DEBUG: Task 2543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2543 has memoization hash 8f2bb4d2abb47fee92b47c9847d5f367 parsl.dataflow.memoization INFO: Task 2543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2543 try 0 launched on executor multi with executor id 2544 parsl.dataflow.dflow INFO: Standard output for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ff47cab0-625f-40be-b9fa-53b3f7dfbb2c_isrForCrosstalkSources_855438_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ff47cab0-625f-40be-b9fa-53b3f7dfbb2c_isrForCrosstalkSources_855438_33.stderr parsl.dataflow.dflow DEBUG: Task 2544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2544 has memoization hash fac20b3a1da084dd9e4841051315d793 parsl.dataflow.memoization INFO: Task 2544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2544 try 0 launched on executor multi with executor id 2545 parsl.dataflow.dflow INFO: Standard output for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/bbfd66eb-9d3d-4d46-9c8b-7b685dc3d704_isrForCrosstalkSources_855395_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/bbfd66eb-9d3d-4d46-9c8b-7b685dc3d704_isrForCrosstalkSources_855395_32.stderr parsl.dataflow.dflow DEBUG: Task 2545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2545 has memoization hash 9c1f61ea66da5596a7cae88a35afab39 parsl.dataflow.memoization INFO: Task 2545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2545 try 0 launched on executor multi with executor id 2546 parsl.dataflow.dflow INFO: Standard output for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/755b2108-1702-4bb0-81b7-d28a92d6e3cb_isrForCrosstalkSources_855399_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/755b2108-1702-4bb0-81b7-d28a92d6e3cb_isrForCrosstalkSources_855399_41.stderr parsl.dataflow.dflow DEBUG: Task 2546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2546 has memoization hash 8ec1903c69c55e0f42c6843240934d73 parsl.dataflow.memoization INFO: Task 2546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2546 try 0 launched on executor multi with executor id 2547 parsl.dataflow.dflow INFO: Standard output for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0040b903-8e4d-401e-bded-d12c0042c078_isrForCrosstalkSources_855412_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0040b903-8e4d-401e-bded-d12c0042c078_isrForCrosstalkSources_855412_34.stderr parsl.dataflow.dflow DEBUG: Task 2547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2547 has memoization hash 8edf9d1d099bdec5f5e9362938ce30bd parsl.dataflow.memoization INFO: Task 2547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2547 try 0 launched on executor multi with executor id 2548 parsl.dataflow.dflow INFO: Standard output for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a39353c0-a578-48bf-8399-e55d3c81fb4a_isrForCrosstalkSources_855398_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a39353c0-a578-48bf-8399-e55d3c81fb4a_isrForCrosstalkSources_855398_1.stderr parsl.dataflow.dflow DEBUG: Task 2548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2548 has memoization hash 95d6a5b16f68ae82e72208ba3ee4ddf4 parsl.dataflow.memoization INFO: Task 2548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2548 try 0 launched on executor multi with executor id 2549 parsl.dataflow.dflow INFO: Standard output for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1a2fa7cf-067b-4f56-b7ad-846acae846b5_isrForCrosstalkSources_855383_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1a2fa7cf-067b-4f56-b7ad-846acae846b5_isrForCrosstalkSources_855383_12.stderr parsl.dataflow.dflow DEBUG: Task 2549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2549 has memoization hash 20bfec3a0b8f075733c32a47a60d9e42 parsl.dataflow.memoization INFO: Task 2549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2549 try 0 launched on executor multi with executor id 2550 parsl.dataflow.dflow INFO: Standard output for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/129d906e-92df-4ac6-af57-d00d9935eb3e_isrForCrosstalkSources_855425_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/129d906e-92df-4ac6-af57-d00d9935eb3e_isrForCrosstalkSources_855425_59.stderr parsl.dataflow.dflow DEBUG: Task 2550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2550 has memoization hash 07d1cca0fe1596a5205d6738539d749f parsl.dataflow.memoization INFO: Task 2550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2550 try 0 launched on executor multi with executor id 2551 parsl.dataflow.dflow INFO: Standard output for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed2ebf13-8605-4a6a-8010-f9b1b9396dee_isrForCrosstalkSources_855453_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed2ebf13-8605-4a6a-8010-f9b1b9396dee_isrForCrosstalkSources_855453_49.stderr parsl.dataflow.dflow DEBUG: Task 2551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2551 has memoization hash e949251d03e7ad0a7de7c6c5a24a969c parsl.dataflow.memoization INFO: Task 2551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2551 try 0 launched on executor multi with executor id 2552 parsl.dataflow.dflow INFO: Standard output for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f35f9170-0262-4df5-b360-e41e6209d19b_isrForCrosstalkSources_855439_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f35f9170-0262-4df5-b360-e41e6209d19b_isrForCrosstalkSources_855439_13.stderr parsl.dataflow.dflow DEBUG: Task 2552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2552 has memoization hash b0f700cae579e5d506da29569673fa8e parsl.dataflow.memoization INFO: Task 2552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:20 proc_lsst.multi:170 [INFO] [multi] job_id 16964146 proc_lsst.multi INFO: [multi] job_id 16964146 2024-03-09 14:51:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-09 14:51:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964146 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964146 parsl.executors.status_handling DEBUG: Launched block 120 on executor multi with job ID 16964146 parsl.executors.status_handling INFO: Allocated block ID 121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-121 2024-03-09 14:51:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 2024-03-09 14:51:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-09 14:51:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2552 try 0 launched on executor multi with executor id 2553 parsl.dataflow.dflow INFO: Standard output for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4acf74c3-0a6e-471b-afef-a978b8d3089a_isrForCrosstalkSources_855435_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4acf74c3-0a6e-471b-afef-a978b8d3089a_isrForCrosstalkSources_855435_45.stderr parsl.dataflow.dflow DEBUG: Task 2553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2553 has memoization hash 2ec8df5b44b32478d05e06c668f37065 parsl.dataflow.memoization INFO: Task 2553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a644c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2553 try 0 launched on executor multi with executor id 2554 parsl.dataflow.dflow INFO: Standard output for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d1de614c-4398-44bf-8cb2-c70d8c699b5a_isrForCrosstalkSources_855409_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d1de614c-4398-44bf-8cb2-c70d8c699b5a_isrForCrosstalkSources_855409_25.stderr parsl.dataflow.dflow DEBUG: Task 2554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2554 has memoization hash da27ea7db72d414cac76e9d24a2d06ad parsl.dataflow.memoization INFO: Task 2554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2554 try 0 launched on executor multi with executor id 2555 parsl.dataflow.dflow INFO: Standard output for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/484b8a4d-03ff-43f6-957e-89e7f8ae21c7_isrForCrosstalkSources_855447_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/484b8a4d-03ff-43f6-957e-89e7f8ae21c7_isrForCrosstalkSources_855447_62.stderr parsl.dataflow.dflow DEBUG: Task 2555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2555 has memoization hash 6d9f3a8a34598eb2a30697ab26a7ca35 parsl.dataflow.memoization INFO: Task 2555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2555 try 0 launched on executor multi with executor id 2556 parsl.dataflow.dflow INFO: Standard output for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/196d0f63-7d92-42ee-9718-efda3e9f2a17_isrForCrosstalkSources_855457_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/196d0f63-7d92-42ee-9718-efda3e9f2a17_isrForCrosstalkSources_855457_20.stderr parsl.dataflow.dflow DEBUG: Task 2556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2556 has memoization hash 2106c126267a38d33129d890bf212b5e parsl.dataflow.memoization INFO: Task 2556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2556 try 0 launched on executor multi with executor id 2557 parsl.dataflow.dflow INFO: Standard output for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/56b1f32c-3c2e-408c-a08a-aeb9cbe13d21_isrForCrosstalkSources_855389_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/56b1f32c-3c2e-408c-a08a-aeb9cbe13d21_isrForCrosstalkSources_855389_31.stderr parsl.dataflow.dflow DEBUG: Task 2557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2557 has memoization hash 0344cd396b27d7fcba34c147c9761ec9 parsl.dataflow.memoization INFO: Task 2557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2557 try 0 launched on executor multi with executor id 2558 parsl.dataflow.dflow INFO: Standard output for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f413a3cd-eb45-424e-b7df-ff458de72eec_isrForCrosstalkSources_855454_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f413a3cd-eb45-424e-b7df-ff458de72eec_isrForCrosstalkSources_855454_60.stderr parsl.dataflow.dflow DEBUG: Task 2558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2558 has memoization hash 4b4db8e88c9ed00dc09ec3c555696750 parsl.dataflow.memoization INFO: Task 2558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2558 try 0 launched on executor multi with executor id 2559 parsl.dataflow.dflow INFO: Standard output for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a20a60f7-aaf7-417d-8859-ede176d8b590_isrForCrosstalkSources_855383_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a20a60f7-aaf7-417d-8859-ede176d8b590_isrForCrosstalkSources_855383_21.stderr parsl.dataflow.dflow DEBUG: Task 2559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2559 has memoization hash 0c60acd5eb9b49ee56a80f09033929e9 parsl.dataflow.memoization INFO: Task 2559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2559 try 0 launched on executor multi with executor id 2560 parsl.dataflow.dflow INFO: Standard output for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6ef4b051-0992-4368-8365-5db6630ba646_isrForCrosstalkSources_855445_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6ef4b051-0992-4368-8365-5db6630ba646_isrForCrosstalkSources_855445_6.stderr parsl.dataflow.dflow DEBUG: Task 2560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2560 has memoization hash 9d327faf248231b4ad91e8f8bd8d138a parsl.dataflow.memoization INFO: Task 2560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2560 try 0 launched on executor multi with executor id 2561 parsl.dataflow.dflow INFO: Standard output for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20d9b9ec-30b4-4750-91ba-23da5b3e071d_isrForCrosstalkSources_855418_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20d9b9ec-30b4-4750-91ba-23da5b3e071d_isrForCrosstalkSources_855418_3.stderr parsl.dataflow.dflow DEBUG: Task 2561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2561 has memoization hash 966b6802c6d231f25fa8a01253447c73 parsl.dataflow.memoization INFO: Task 2561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2561 try 0 launched on executor multi with executor id 2562 parsl.dataflow.dflow INFO: Standard output for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a622c46e-2f4b-4240-b3b4-2f2e9e052f82_isrForCrosstalkSources_855381_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a622c46e-2f4b-4240-b3b4-2f2e9e052f82_isrForCrosstalkSources_855381_61.stderr parsl.dataflow.dflow DEBUG: Task 2562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2562 has memoization hash 524cff39a29323f1aeda440054705c4d parsl.dataflow.memoization INFO: Task 2562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2562 try 0 launched on executor multi with executor id 2563 parsl.dataflow.dflow INFO: Standard output for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/30b5f41a-19db-4c12-96f9-babbea6f6112_isrForCrosstalkSources_855451_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/30b5f41a-19db-4c12-96f9-babbea6f6112_isrForCrosstalkSources_855451_23.stderr parsl.dataflow.dflow DEBUG: Task 2563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2563 has memoization hash 401977b09db2c1b7d34ab483ab59d049 parsl.dataflow.memoization INFO: Task 2563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2563 try 0 launched on executor multi with executor id 2564 parsl.dataflow.dflow INFO: Standard output for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a29424c1-76ff-4dc9-aec1-9eccfb0a5347_isrForCrosstalkSources_855404_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a29424c1-76ff-4dc9-aec1-9eccfb0a5347_isrForCrosstalkSources_855404_59.stderr parsl.dataflow.dflow DEBUG: Task 2564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2564 has memoization hash f8e7ffc7ec8563172f34aca739c05a5c parsl.dataflow.memoization INFO: Task 2564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2564 try 0 launched on executor multi with executor id 2565 parsl.dataflow.dflow INFO: Standard output for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/43c2c253-7a4b-42ce-b338-46fc31ccab39_isrForCrosstalkSources_855386_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/43c2c253-7a4b-42ce-b338-46fc31ccab39_isrForCrosstalkSources_855386_26.stderr parsl.dataflow.dflow DEBUG: Task 2565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2565 has memoization hash cddcd824833efba9f784c79d40b15069 parsl.dataflow.memoization INFO: Task 2565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2565 try 0 launched on executor multi with executor id 2566 parsl.dataflow.dflow INFO: Standard output for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dbf1a0a9-40f6-45b4-bd22-96d5d4c213ec_isrForCrosstalkSources_855441_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dbf1a0a9-40f6-45b4-bd22-96d5d4c213ec_isrForCrosstalkSources_855441_34.stderr parsl.dataflow.dflow DEBUG: Task 2566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2566 has memoization hash 1c22ef2f822e5749b6783d49606400f9 parsl.dataflow.memoization INFO: Task 2566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2566 try 0 launched on executor multi with executor id 2567 parsl.dataflow.dflow INFO: Standard output for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ad5ef9c-8b2b-435a-9eac-ff8ea0e9a6e2_isrForCrosstalkSources_855418_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ad5ef9c-8b2b-435a-9eac-ff8ea0e9a6e2_isrForCrosstalkSources_855418_62.stderr parsl.dataflow.dflow DEBUG: Task 2567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2567 has memoization hash 2e0a031e8dfd6c62864eea2a24fe72d9 parsl.dataflow.memoization INFO: Task 2567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2567 try 0 launched on executor multi with executor id 2568 parsl.dataflow.dflow INFO: Standard output for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c4943726-336b-4af5-ba7e-b145950c69fb_isrForCrosstalkSources_855380_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c4943726-336b-4af5-ba7e-b145950c69fb_isrForCrosstalkSources_855380_10.stderr parsl.dataflow.dflow DEBUG: Task 2568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2568 has memoization hash a82242c92d58165413b0a7b1fdba0202 parsl.dataflow.memoization INFO: Task 2568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2568 try 0 launched on executor multi with executor id 2569 parsl.dataflow.dflow INFO: Standard output for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/b3ee987b-21ac-42ba-884c-af2ac8cc2bd8_isrForCrosstalkSources_855449_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/b3ee987b-21ac-42ba-884c-af2ac8cc2bd8_isrForCrosstalkSources_855449_27.stderr parsl.dataflow.dflow DEBUG: Task 2569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2569 has memoization hash 46307bd573b66c1044687d3435127dce parsl.dataflow.memoization INFO: Task 2569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2569 try 0 launched on executor multi with executor id 2570 parsl.dataflow.dflow INFO: Standard output for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5b795e20-e9ee-4461-affa-7f17c2329eb8_isrForCrosstalkSources_855422_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5b795e20-e9ee-4461-affa-7f17c2329eb8_isrForCrosstalkSources_855422_52.stderr parsl.dataflow.dflow DEBUG: Task 2570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2570 has memoization hash 7200b6e44940b215a9c4c55d833c9668 parsl.dataflow.memoization INFO: Task 2570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2570 try 0 launched on executor multi with executor id 2571 parsl.dataflow.dflow INFO: Standard output for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c5981058-52aa-4211-841b-1272e1033133_isrForCrosstalkSources_855433_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c5981058-52aa-4211-841b-1272e1033133_isrForCrosstalkSources_855433_35.stderr parsl.dataflow.dflow DEBUG: Task 2571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2571 has memoization hash f5fe742aecaa1ea6e0b6ab34309ee1ab parsl.dataflow.memoization INFO: Task 2571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2571 try 0 launched on executor multi with executor id 2572 parsl.dataflow.dflow INFO: Standard output for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82776559-131a-4fdf-bed4-55ee12ce42e2_isrForCrosstalkSources_855440_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82776559-131a-4fdf-bed4-55ee12ce42e2_isrForCrosstalkSources_855440_3.stderr parsl.dataflow.dflow DEBUG: Task 2572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2572 has memoization hash 33e2b7f956cb807ade675c7226ae7ad7 parsl.dataflow.memoization INFO: Task 2572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2572 try 0 launched on executor multi with executor id 2573 parsl.dataflow.dflow INFO: Standard output for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bbe6ed9d-2502-43f5-b169-9a78c0a6a225_isrForCrosstalkSources_855380_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bbe6ed9d-2502-43f5-b169-9a78c0a6a225_isrForCrosstalkSources_855380_55.stderr parsl.dataflow.dflow DEBUG: Task 2573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2573 has memoization hash 4b9696b311ebef69d132dc17197d3104 parsl.dataflow.memoization INFO: Task 2573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2573 try 0 launched on executor multi with executor id 2574 parsl.dataflow.dflow INFO: Standard output for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/adf10067-85b5-499a-8c0f-f684688d7502_isrForCrosstalkSources_855404_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/adf10067-85b5-499a-8c0f-f684688d7502_isrForCrosstalkSources_855404_53.stderr parsl.dataflow.dflow DEBUG: Task 2574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2574 has memoization hash 511aeb58ef20d34efef7c3512fb80e2c parsl.dataflow.memoization INFO: Task 2574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2574 try 0 launched on executor multi with executor id 2575 parsl.dataflow.dflow INFO: Standard output for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a1fafc8-852d-4729-9911-1f35a729888f_isrForCrosstalkSources_855447_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a1fafc8-852d-4729-9911-1f35a729888f_isrForCrosstalkSources_855447_29.stderr parsl.dataflow.dflow DEBUG: Task 2575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2575 has memoization hash 3d63f4c61c51b0490271a5fa817e6885 parsl.dataflow.memoization INFO: Task 2575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2575 try 0 launched on executor multi with executor id 2576 parsl.dataflow.dflow INFO: Standard output for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d7fb63a0-d8d5-4d86-a1a3-c69fa60bd599_isrForCrosstalkSources_855409_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d7fb63a0-d8d5-4d86-a1a3-c69fa60bd599_isrForCrosstalkSources_855409_41.stderr parsl.dataflow.dflow DEBUG: Task 2576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2576 has memoization hash 3722847c06d7007089baf3683fa27905 parsl.dataflow.memoization INFO: Task 2576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2576 try 0 launched on executor multi with executor id 2577 parsl.dataflow.dflow INFO: Standard output for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9f804cac-f320-4a04-8e5a-a606c05271c1_isrForCrosstalkSources_855394_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9f804cac-f320-4a04-8e5a-a606c05271c1_isrForCrosstalkSources_855394_2.stderr parsl.dataflow.dflow DEBUG: Task 2577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2577 has memoization hash 0bbe790242a8cf0b1a1d61dab7c9ee74 parsl.dataflow.memoization INFO: Task 2577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2577 try 0 launched on executor multi with executor id 2578 parsl.dataflow.dflow INFO: Standard output for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e1a10005-8b30-4bde-8422-dcc0cf2962f2_isrForCrosstalkSources_855380_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e1a10005-8b30-4bde-8422-dcc0cf2962f2_isrForCrosstalkSources_855380_48.stderr parsl.dataflow.dflow DEBUG: Task 2578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2578 has memoization hash a25234f808ee399f8b1b00fbe777941a parsl.dataflow.memoization INFO: Task 2578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2578 try 0 launched on executor multi with executor id 2579 parsl.dataflow.dflow INFO: Standard output for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d409610a-8d69-4a28-9df2-ac10c521f0a8_isrForCrosstalkSources_855428_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d409610a-8d69-4a28-9df2-ac10c521f0a8_isrForCrosstalkSources_855428_4.stderr parsl.dataflow.dflow DEBUG: Task 2579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2579 has memoization hash 2186822b35bcc7366664ec08db484287 parsl.dataflow.memoization INFO: Task 2579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2579 try 0 launched on executor multi with executor id 2580 parsl.dataflow.dflow INFO: Standard output for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5bdc3a2a-1304-4e01-bccb-2a45dc5e5209_isrForCrosstalkSources_855396_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5bdc3a2a-1304-4e01-bccb-2a45dc5e5209_isrForCrosstalkSources_855396_25.stderr parsl.dataflow.dflow DEBUG: Task 2580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2580 has memoization hash 1626a84d3226c72a7ec4af3c00cd13dc parsl.dataflow.memoization INFO: Task 2580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2580 try 0 launched on executor multi with executor id 2581 parsl.dataflow.dflow INFO: Standard output for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/65553bf8-cdb1-4b5f-884b-a30c40a8e71d_isrForCrosstalkSources_855400_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/65553bf8-cdb1-4b5f-884b-a30c40a8e71d_isrForCrosstalkSources_855400_17.stderr parsl.dataflow.dflow DEBUG: Task 2581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2581 has memoization hash 12f37de86a9565a543785db2a5cea5bd parsl.dataflow.memoization INFO: Task 2581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2581 try 0 launched on executor multi with executor id 2582 parsl.dataflow.dflow INFO: Standard output for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/469803fc-f36d-4278-a715-8beb4fdfae5c_isrForCrosstalkSources_855406_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/469803fc-f36d-4278-a715-8beb4fdfae5c_isrForCrosstalkSources_855406_23.stderr parsl.dataflow.dflow DEBUG: Task 2582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2582 has memoization hash cec39cd79b987c6b226a765c03bf326d parsl.dataflow.memoization INFO: Task 2582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2582 try 0 launched on executor multi with executor id 2583 parsl.dataflow.dflow INFO: Standard output for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/530abfac-689c-44cd-8738-0205cc179bbb_isrForCrosstalkSources_855434_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/530abfac-689c-44cd-8738-0205cc179bbb_isrForCrosstalkSources_855434_4.stderr parsl.dataflow.dflow DEBUG: Task 2583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2583 has memoization hash 6967effb988d2ea2dfca367918a1b860 parsl.dataflow.memoization INFO: Task 2583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2583 try 0 launched on executor multi with executor id 2584 parsl.dataflow.dflow INFO: Standard output for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f9c5d21a-e1fa-4212-9f3b-af41b5edb171_isrForCrosstalkSources_855452_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f9c5d21a-e1fa-4212-9f3b-af41b5edb171_isrForCrosstalkSources_855452_61.stderr parsl.dataflow.dflow DEBUG: Task 2584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2584 has memoization hash 2353a4cc298ae5831aa164b346722b68 parsl.dataflow.memoization INFO: Task 2584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2584 try 0 launched on executor multi with executor id 2585 parsl.dataflow.dflow INFO: Standard output for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/35444b75-964f-4c87-b470-ef28fe88ddfd_isrForCrosstalkSources_855440_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/35444b75-964f-4c87-b470-ef28fe88ddfd_isrForCrosstalkSources_855440_55.stderr parsl.dataflow.dflow DEBUG: Task 2585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2585 has memoization hash d7dfd9a3bbc46375d800b4903f584db5 parsl.dataflow.memoization INFO: Task 2585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2585 try 0 launched on executor multi with executor id 2586 parsl.dataflow.dflow INFO: Standard output for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a4be636e-785d-4eab-935f-108332bf4ccb_isrForCrosstalkSources_855386_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a4be636e-785d-4eab-935f-108332bf4ccb_isrForCrosstalkSources_855386_30.stderr parsl.dataflow.dflow DEBUG: Task 2586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2586 has memoization hash 2ff57f8d4c388a8d9e761cd43adfae6d parsl.dataflow.memoization INFO: Task 2586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:21 proc_lsst.multi:170 [INFO] [multi] job_id 16964147 proc_lsst.multi INFO: [multi] job_id 16964147 2024-03-09 14:51:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-09 14:51:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964147 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964147 parsl.executors.status_handling DEBUG: Launched block 121 on executor multi with job ID 16964147 parsl.executors.status_handling INFO: Allocated block ID 122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-122 2024-03-09 14:51:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2586 try 0 launched on executor multi with executor id 2587 parsl.dataflow.dflow INFO: Standard output for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ed90738b-040a-4fcc-a467-7d068e82c3bd_isrForCrosstalkSources_855381_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ed90738b-040a-4fcc-a467-7d068e82c3bd_isrForCrosstalkSources_855381_17.stderr parsl.dataflow.dflow DEBUG: Task 2587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2587 has memoization hash 5c1de937a0c574d6e85820268d8e7a6e parsl.dataflow.memoization INFO: Task 2587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2587 try 0 launched on executor multi with executor id 2588 parsl.dataflow.dflow INFO: Standard output for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/934d3f97-da6e-4e89-904a-9cc5b363cdee_isrForCrosstalkSources_855429_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/934d3f97-da6e-4e89-904a-9cc5b363cdee_isrForCrosstalkSources_855429_54.stderr parsl.dataflow.dflow DEBUG: Task 2588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2588 has memoization hash 360b0bf900dabeb1d14e296b39cd529d parsl.dataflow.memoization INFO: Task 2588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2588 try 0 launched on executor multi with executor id 2589 parsl.dataflow.dflow INFO: Standard output for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8bbfdb9e-1539-4be5-9441-9982851ce31d_isrForCrosstalkSources_855408_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8bbfdb9e-1539-4be5-9441-9982851ce31d_isrForCrosstalkSources_855408_36.stderr parsl.dataflow.dflow DEBUG: Task 2589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2589 has memoization hash 02e9883ae0e42bb644a85ac4f277ffd5 parsl.dataflow.memoization INFO: Task 2589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2589 try 0 launched on executor multi with executor id 2590 parsl.dataflow.dflow INFO: Standard output for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f564721e-0ff8-4d28-8ad7-fc1a546cd8f1_isrForCrosstalkSources_855409_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f564721e-0ff8-4d28-8ad7-fc1a546cd8f1_isrForCrosstalkSources_855409_60.stderr parsl.dataflow.dflow DEBUG: Task 2590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2590 has memoization hash 0969fd98fc700204e24d26f9c556fde1 parsl.dataflow.memoization INFO: Task 2590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2590 try 0 launched on executor multi with executor id 2591 parsl.dataflow.dflow INFO: Standard output for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1229a30a-ca6c-4f3a-8338-18a370c7c8a1_isrForCrosstalkSources_855412_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1229a30a-ca6c-4f3a-8338-18a370c7c8a1_isrForCrosstalkSources_855412_41.stderr parsl.dataflow.dflow DEBUG: Task 2591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2591 has memoization hash d3c7d92277afa8d5af507c6a1a243851 parsl.dataflow.memoization INFO: Task 2591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2591 try 0 launched on executor multi with executor id 2592 parsl.dataflow.dflow INFO: Standard output for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/568d35f2-a798-42c2-bb96-ff235504ac5d_isrForCrosstalkSources_855386_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/568d35f2-a798-42c2-bb96-ff235504ac5d_isrForCrosstalkSources_855386_36.stderr parsl.dataflow.dflow DEBUG: Task 2592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2592 has memoization hash f22c7d18413ca134d12d33ecb855db85 parsl.dataflow.memoization INFO: Task 2592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2592 try 0 launched on executor multi with executor id 2593 parsl.dataflow.dflow INFO: Standard output for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7cc4e343-3e87-4f90-aed5-110cfd68c5e8_isrForCrosstalkSources_855412_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7cc4e343-3e87-4f90-aed5-110cfd68c5e8_isrForCrosstalkSources_855412_59.stderr parsl.dataflow.dflow DEBUG: Task 2593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2593 has memoization hash 7a9dfea077589611a06fca81e4d51906 parsl.dataflow.memoization INFO: Task 2593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2593 try 0 launched on executor multi with executor id 2594 parsl.dataflow.dflow INFO: Standard output for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82c64291-e258-4afa-b5fc-3f444a21ea9b_isrForCrosstalkSources_855440_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82c64291-e258-4afa-b5fc-3f444a21ea9b_isrForCrosstalkSources_855440_45.stderr parsl.dataflow.dflow DEBUG: Task 2594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2594 has memoization hash 096c195f3f38ec8081d900978dbd06b1 parsl.dataflow.memoization INFO: Task 2594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2594 try 0 launched on executor multi with executor id 2595 parsl.dataflow.dflow INFO: Standard output for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/856f2ec4-b563-49af-b452-92efa8e774f5_isrForCrosstalkSources_855430_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/856f2ec4-b563-49af-b452-92efa8e774f5_isrForCrosstalkSources_855430_8.stderr parsl.dataflow.dflow DEBUG: Task 2595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2595 has memoization hash 35a9a08c98555ebb114d88115b6a118c parsl.dataflow.memoization INFO: Task 2595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2595 try 0 launched on executor multi with executor id 2596 parsl.dataflow.dflow INFO: Standard output for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2d1cc952-b318-4a5e-8c05-425b648fef96_isrForCrosstalkSources_855447_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2d1cc952-b318-4a5e-8c05-425b648fef96_isrForCrosstalkSources_855447_11.stderr parsl.dataflow.dflow DEBUG: Task 2596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2596 has memoization hash b571ad62187740a34eef84137ec40bd2 parsl.dataflow.memoization INFO: Task 2596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2596 try 0 launched on executor multi with executor id 2597 parsl.dataflow.dflow INFO: Standard output for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/79c0087e-a9f6-4191-974e-f153e60535f6_isrForCrosstalkSources_855400_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/79c0087e-a9f6-4191-974e-f153e60535f6_isrForCrosstalkSources_855400_4.stderr parsl.dataflow.dflow DEBUG: Task 2597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2597 has memoization hash e1c36ad87194cbc9f4d2d3a4e0f40b1d parsl.dataflow.memoization INFO: Task 2597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2597 try 0 launched on executor multi with executor id 2598 parsl.dataflow.dflow INFO: Standard output for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0a520dd5-ed2e-4f49-ac7c-53f30c14485a_isrForCrosstalkSources_855419_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0a520dd5-ed2e-4f49-ac7c-53f30c14485a_isrForCrosstalkSources_855419_57.stderr parsl.dataflow.dflow DEBUG: Task 2598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2598 has memoization hash 8ed65a1f7ccf9a0a4464792e44aaf2a7 parsl.dataflow.memoization INFO: Task 2598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2598 try 0 launched on executor multi with executor id 2599 parsl.dataflow.dflow INFO: Standard output for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/63c36c85-7fdc-4b45-9b3a-b1822b62e865_isrForCrosstalkSources_855449_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/63c36c85-7fdc-4b45-9b3a-b1822b62e865_isrForCrosstalkSources_855449_10.stderr parsl.dataflow.dflow DEBUG: Task 2599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2599 has memoization hash 379b00d1b599380672e292e059aa968d parsl.dataflow.memoization INFO: Task 2599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2599 try 0 launched on executor multi with executor id 2600 parsl.dataflow.dflow INFO: Standard output for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f7ad424a-23b4-44c2-ab0e-4b4e60381c5b_isrForCrosstalkSources_855420_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f7ad424a-23b4-44c2-ab0e-4b4e60381c5b_isrForCrosstalkSources_855420_26.stderr parsl.dataflow.dflow DEBUG: Task 2600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2600 has memoization hash 8208c4534cb88fa67441d6f5910f9b09 parsl.dataflow.memoization INFO: Task 2600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2600 try 0 launched on executor multi with executor id 2601 parsl.dataflow.dflow INFO: Standard output for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/4346d26c-f654-492b-9569-5c48bfdbf942_isrForCrosstalkSources_855428_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/4346d26c-f654-492b-9569-5c48bfdbf942_isrForCrosstalkSources_855428_56.stderr parsl.dataflow.dflow DEBUG: Task 2601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2601 has memoization hash 2033c3048626bdf9c1f5d5d5f2e7e438 parsl.dataflow.memoization INFO: Task 2601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2601 try 0 launched on executor multi with executor id 2602 parsl.dataflow.dflow INFO: Standard output for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/89048f1e-2bc8-4dfb-8871-70fba59dd2b2_isrForCrosstalkSources_855380_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/89048f1e-2bc8-4dfb-8871-70fba59dd2b2_isrForCrosstalkSources_855380_14.stderr parsl.dataflow.dflow DEBUG: Task 2602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2602 has memoization hash 22bfec2f5bdea4bec30705224055d3ff parsl.dataflow.memoization INFO: Task 2602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2602 try 0 launched on executor multi with executor id 2603 parsl.dataflow.dflow INFO: Standard output for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/4eca778f-afe9-4ac2-ab2d-d6a33f7a6c8c_isrForCrosstalkSources_855402_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/4eca778f-afe9-4ac2-ab2d-d6a33f7a6c8c_isrForCrosstalkSources_855402_48.stderr parsl.dataflow.dflow DEBUG: Task 2603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2603 has memoization hash 5611b40844da93460b51178edd8ca2e8 parsl.dataflow.memoization INFO: Task 2603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2603 try 0 launched on executor multi with executor id 2604 parsl.dataflow.dflow INFO: Standard output for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/1d325502-6326-4eeb-8e88-20e02d4b4b60_isrForCrosstalkSources_855379_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/1d325502-6326-4eeb-8e88-20e02d4b4b60_isrForCrosstalkSources_855379_24.stderr parsl.dataflow.dflow DEBUG: Task 2604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2604 has memoization hash 836570b2d4364235193bd5c05af89c37 parsl.dataflow.memoization INFO: Task 2604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2604 try 0 launched on executor multi with executor id 2605 parsl.dataflow.dflow INFO: Standard output for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/14711fb2-21db-4c9a-af4d-2c39961a5def_isrForCrosstalkSources_855390_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/14711fb2-21db-4c9a-af4d-2c39961a5def_isrForCrosstalkSources_855390_11.stderr parsl.dataflow.dflow DEBUG: Task 2605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2605 has memoization hash 871bfb8c381882942c317fea6ed119bc parsl.dataflow.memoization INFO: Task 2605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2605 try 0 launched on executor multi with executor id 2606 parsl.dataflow.dflow INFO: Standard output for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/37ad1e4e-b503-42ec-ac0d-3bdbefa52b59_isrForCrosstalkSources_855379_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/37ad1e4e-b503-42ec-ac0d-3bdbefa52b59_isrForCrosstalkSources_855379_62.stderr parsl.dataflow.dflow DEBUG: Task 2606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2606 has memoization hash ad2c3441bec9ddc612fb9c78df441806 parsl.dataflow.memoization INFO: Task 2606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2606 try 0 launched on executor multi with executor id 2607 parsl.dataflow.dflow INFO: Standard output for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/885242df-7aed-4a53-9d3e-f8a0ce72701d_isrForCrosstalkSources_855445_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/885242df-7aed-4a53-9d3e-f8a0ce72701d_isrForCrosstalkSources_855445_9.stderr parsl.dataflow.dflow DEBUG: Task 2607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2607 has memoization hash 2ef56c3019ebc807a826b1784c6363ec parsl.dataflow.memoization INFO: Task 2607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2607 try 0 launched on executor multi with executor id 2608 parsl.dataflow.dflow INFO: Standard output for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab0d87a1-8d99-471e-aa05-aba3293509bc_isrForCrosstalkSources_855420_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab0d87a1-8d99-471e-aa05-aba3293509bc_isrForCrosstalkSources_855420_8.stderr parsl.dataflow.dflow DEBUG: Task 2608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2608 has memoization hash 10a8d1fe04610d736a203b791969585c parsl.dataflow.memoization INFO: Task 2608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2608 try 0 launched on executor multi with executor id 2609 parsl.dataflow.dflow INFO: Standard output for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/61e031f6-1351-4242-9e7b-40c2584a30b2_isrForCrosstalkSources_855403_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/61e031f6-1351-4242-9e7b-40c2584a30b2_isrForCrosstalkSources_855403_50.stderr parsl.dataflow.dflow DEBUG: Task 2609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2609 has memoization hash 14545e78a7613b179a48cb78c2d63a95 parsl.dataflow.memoization INFO: Task 2609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2609 try 0 launched on executor multi with executor id 2610 parsl.dataflow.dflow INFO: Standard output for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f4d0fa0b-fd9c-42dc-9d81-9aa074612bc3_isrForCrosstalkSources_855451_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f4d0fa0b-fd9c-42dc-9d81-9aa074612bc3_isrForCrosstalkSources_855451_17.stderr parsl.dataflow.dflow DEBUG: Task 2610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2610 has memoization hash e8cc85c0414959be84c4acd1646d5f47 parsl.dataflow.memoization INFO: Task 2610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2610 try 0 launched on executor multi with executor id 2611 parsl.dataflow.dflow INFO: Standard output for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be4cecd2-ff8b-4964-b25a-43cd7ff9f797_isrForCrosstalkSources_855425_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be4cecd2-ff8b-4964-b25a-43cd7ff9f797_isrForCrosstalkSources_855425_2.stderr parsl.dataflow.dflow DEBUG: Task 2611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2611 has memoization hash 387835c4b44159233e794f575675e7ac parsl.dataflow.memoization INFO: Task 2611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2611 try 0 launched on executor multi with executor id 2612 parsl.dataflow.dflow INFO: Standard output for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/873e0479-f344-403c-8375-3be2c1fdc0e9_isrForCrosstalkSources_855439_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/873e0479-f344-403c-8375-3be2c1fdc0e9_isrForCrosstalkSources_855439_37.stderr parsl.dataflow.dflow DEBUG: Task 2612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2612 has memoization hash 2348ab559200e1e6d7dde8d156d40ff4 parsl.dataflow.memoization INFO: Task 2612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2612 try 0 launched on executor multi with executor id 2613 parsl.dataflow.dflow INFO: Standard output for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ddcb12f2-a888-4810-a8e5-fff36e9d8239_isrForCrosstalkSources_855403_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ddcb12f2-a888-4810-a8e5-fff36e9d8239_isrForCrosstalkSources_855403_31.stderr parsl.dataflow.dflow DEBUG: Task 2613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2613 has memoization hash 4cc2cd0cb1bd75a324d5a4c1c18d02d1 parsl.dataflow.memoization INFO: Task 2613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 520bf8664b6bd9f558202020f63ba801 with result from task 284 parsl.dataflow.dflow INFO: Parsl task 2613 try 0 launched on executor multi with executor id 2614 parsl.dataflow.dflow INFO: Standard output for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7c7c610f-9533-476f-8008-43c4440e875b_isrForCrosstalkSources_855459_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7c7c610f-9533-476f-8008-43c4440e875b_isrForCrosstalkSources_855459_25.stderr parsl.dataflow.dflow DEBUG: Task 2614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2614 has memoization hash de083c144b22f783acd0d33e6c5a32dd parsl.dataflow.memoization INFO: Task 2614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/63a095ff-6d9a-4baf-8f97-ae8cc0e3a9a2_isrForCrosstalkSources_855379_14.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/63a095ff-6d9a-4baf-8f97-ae8cc0e3a9a2_isrForCrosstalkSources_855379_14.stderr parsl.dataflow.dflow INFO: Parsl task 2614 try 0 launched on executor multi with executor id 2615 parsl.dataflow.dflow INFO: Standard output for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/626e8b04-dab2-43c3-8337-61a1bac424cb_isrForCrosstalkSources_855390_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/626e8b04-dab2-43c3-8337-61a1bac424cb_isrForCrosstalkSources_855390_5.stderr parsl.dataflow.dflow DEBUG: Task 2615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2615 has memoization hash 9f5ed09f81323e3fba890f67bb7badb1 parsl.dataflow.memoization INFO: Task 2615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:21 proc_lsst.multi:170 [INFO] [multi] job_id 16964148 proc_lsst.multi INFO: [multi] job_id 16964148 2024-03-09 14:51:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-09 14:51:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964148 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964148 parsl.executors.status_handling DEBUG: Launched block 122 on executor multi with job ID 16964148 parsl.executors.status_handling INFO: Allocated block ID 123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-123 2024-03-09 14:51:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-09 14:51:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2615 try 0 launched on executor multi with executor id 2616 parsl.dataflow.dflow INFO: Standard output for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/785093c9-3116-4fad-9629-cc4f7e93c57a_isrForCrosstalkSources_855458_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/785093c9-3116-4fad-9629-cc4f7e93c57a_isrForCrosstalkSources_855458_48.stderr parsl.dataflow.dflow DEBUG: Task 2616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2616 has memoization hash 478f945e29f0fdf3e7ee5ea26b7cc34e parsl.dataflow.memoization INFO: Task 2616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2616 try 0 launched on executor multi with executor id 2617 parsl.dataflow.dflow INFO: Standard output for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/62bd80a3-79c0-4258-a11a-8c2a5593b02e_isrForCrosstalkSources_855386_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/62bd80a3-79c0-4258-a11a-8c2a5593b02e_isrForCrosstalkSources_855386_59.stderr parsl.dataflow.dflow DEBUG: Task 2617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2617 has memoization hash 498e86be4099d6cf7257cff312a8f190 parsl.dataflow.memoization INFO: Task 2617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2617 try 0 launched on executor multi with executor id 2618 parsl.dataflow.dflow INFO: Standard output for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/144221a3-6af4-4e44-8ca2-66d863ab50e6_isrForCrosstalkSources_855432_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/144221a3-6af4-4e44-8ca2-66d863ab50e6_isrForCrosstalkSources_855432_2.stderr parsl.dataflow.dflow DEBUG: Task 2618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2618 has memoization hash 9dfb86a89ebb12780f9b51dba297f353 parsl.dataflow.memoization INFO: Task 2618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2618 try 0 launched on executor multi with executor id 2619 parsl.dataflow.dflow INFO: Standard output for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/beddd4e5-2c32-4ad1-b0b0-3895368eec5d_isrForCrosstalkSources_855399_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/beddd4e5-2c32-4ad1-b0b0-3895368eec5d_isrForCrosstalkSources_855399_47.stderr parsl.dataflow.dflow DEBUG: Task 2619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2619 has memoization hash 2ae10c294a62fac813af30e10e4d2a62 parsl.dataflow.memoization INFO: Task 2619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f2ac01d984c8f9d06c5e0accc8b5965 with result from task 371 parsl.dataflow.dflow INFO: Parsl task 2619 try 0 launched on executor multi with executor id 2620 parsl.dataflow.dflow INFO: Standard output for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5f194826-ae73-43a8-8a06-caa1c38325be_isrForCrosstalkSources_855450_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5f194826-ae73-43a8-8a06-caa1c38325be_isrForCrosstalkSources_855450_12.stderr parsl.dataflow.dflow DEBUG: Task 2620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2620 has memoization hash 5193272aaaa0205c79b4a5cc34919e61 parsl.dataflow.memoization INFO: Task 2620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2620 try 0 launched on executor multi with executor id 2621 parsl.dataflow.dflow INFO: Standard output for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ed8f32c-558d-42db-8873-4fb0dded2c19_isrForCrosstalkSources_855396_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ed8f32c-558d-42db-8873-4fb0dded2c19_isrForCrosstalkSources_855396_47.stderr parsl.dataflow.dflow DEBUG: Task 2621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2621 has memoization hash ba3d642598cf47598cc947d49dad61e3 parsl.dataflow.memoization INFO: Task 2621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/088f26e9-0d45-45d2-aa89-cf07dccc5aa8_isrForCrosstalkSources_855389_20.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/088f26e9-0d45-45d2-aa89-cf07dccc5aa8_isrForCrosstalkSources_855389_20.stderr parsl.dataflow.dflow INFO: Parsl task 2621 try 0 launched on executor multi with executor id 2622 parsl.dataflow.dflow INFO: Standard output for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eb9b7a0b-fb71-45b6-8300-5467ed3d9538_isrForCrosstalkSources_855423_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eb9b7a0b-fb71-45b6-8300-5467ed3d9538_isrForCrosstalkSources_855423_27.stderr parsl.dataflow.dflow DEBUG: Task 2622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2622 has memoization hash 99f9c1682c37396050e996e91e2906ee parsl.dataflow.memoization INFO: Task 2622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2622 try 0 launched on executor multi with executor id 2623 parsl.dataflow.dflow INFO: Standard output for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a162ada0-d948-4189-a7cf-289d98dd36b3_isrForCrosstalkSources_855380_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a162ada0-d948-4189-a7cf-289d98dd36b3_isrForCrosstalkSources_855380_33.stderr parsl.dataflow.dflow DEBUG: Task 2623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2623 has memoization hash 724d861ea71e2c4b3b5161d8f332df73 parsl.dataflow.memoization INFO: Task 2623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2623 try 0 launched on executor multi with executor id 2624 parsl.dataflow.dflow INFO: Standard output for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4c5223a6-60fa-4711-988a-570863e10a89_isrForCrosstalkSources_855415_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4c5223a6-60fa-4711-988a-570863e10a89_isrForCrosstalkSources_855415_21.stderr parsl.dataflow.dflow DEBUG: Task 2624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2624 has memoization hash d1caefce6e06044992acc845d344e6a6 parsl.dataflow.memoization INFO: Task 2624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2624 try 0 launched on executor multi with executor id 2625 parsl.dataflow.dflow INFO: Standard output for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/7c144a72-932e-4025-8ce9-f1e43d10c1df_isrForCrosstalkSources_855432_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/7c144a72-932e-4025-8ce9-f1e43d10c1df_isrForCrosstalkSources_855432_12.stderr parsl.dataflow.dflow DEBUG: Task 2625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2625 has memoization hash 7b72fd69f019dcd96571477daf8c162a parsl.dataflow.memoization INFO: Task 2625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2625 try 0 launched on executor multi with executor id 2626 parsl.dataflow.dflow INFO: Standard output for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a7990f8a-1fb0-44e7-9059-b5a2db7c9bd0_isrForCrosstalkSources_855393_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a7990f8a-1fb0-44e7-9059-b5a2db7c9bd0_isrForCrosstalkSources_855393_43.stderr parsl.dataflow.dflow DEBUG: Task 2626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2626 has memoization hash 76d12faf5641494c47560515c26f2dee parsl.dataflow.memoization INFO: Task 2626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2626 try 0 launched on executor multi with executor id 2627 parsl.dataflow.dflow INFO: Standard output for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6c92fba4-cb25-4f53-8b18-459a3b9d488c_isrForCrosstalkSources_855401_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6c92fba4-cb25-4f53-8b18-459a3b9d488c_isrForCrosstalkSources_855401_1.stderr parsl.dataflow.dflow DEBUG: Task 2627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2627 has memoization hash b87db32c7e4dade102013e43e8a61458 parsl.dataflow.memoization INFO: Task 2627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2627 try 0 launched on executor multi with executor id 2628 parsl.dataflow.dflow INFO: Standard output for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/60f93199-c3ea-44ac-92de-63a6f50c1dca_isrForCrosstalkSources_855449_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/60f93199-c3ea-44ac-92de-63a6f50c1dca_isrForCrosstalkSources_855449_17.stderr parsl.dataflow.dflow DEBUG: Task 2628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2628 has memoization hash b7a1b290c3466c21eb850e273cbe5b76 parsl.dataflow.memoization INFO: Task 2628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0006f0806362b92d546304b21db030c with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2fd31897-d9c9-4b76-b0de-3981b76fadf8_isrForCrosstalkSources_855441_19.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2fd31897-d9c9-4b76-b0de-3981b76fadf8_isrForCrosstalkSources_855441_19.stderr parsl.dataflow.dflow INFO: Parsl task 2628 try 0 launched on executor multi with executor id 2629 parsl.dataflow.dflow INFO: Standard output for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6b0dfa55-96ad-46ef-a513-38c882a20bb2_isrForCrosstalkSources_855431_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6b0dfa55-96ad-46ef-a513-38c882a20bb2_isrForCrosstalkSources_855431_56.stderr parsl.dataflow.dflow DEBUG: Task 2629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2629 has memoization hash a6e7fe6a3baafc06bb664f6499398887 parsl.dataflow.memoization INFO: Task 2629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2629 try 0 launched on executor multi with executor id 2630 parsl.dataflow.dflow INFO: Standard output for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/217442bb-0da0-48a7-8503-95da57427e05_isrForCrosstalkSources_855412_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/217442bb-0da0-48a7-8503-95da57427e05_isrForCrosstalkSources_855412_5.stderr parsl.dataflow.dflow DEBUG: Task 2630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2630 has memoization hash 53d838d99bf3a279e5145ec7ab464f08 parsl.dataflow.memoization INFO: Task 2630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2630 try 0 launched on executor multi with executor id 2631 parsl.dataflow.dflow INFO: Standard output for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/71d7052f-7a4c-47a3-80c0-4a69ebeb8615_isrForCrosstalkSources_855407_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/71d7052f-7a4c-47a3-80c0-4a69ebeb8615_isrForCrosstalkSources_855407_14.stderr parsl.dataflow.dflow DEBUG: Task 2631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2631 has memoization hash 021d603c6b5f4dad2e5df2390e0c181e parsl.dataflow.memoization INFO: Task 2631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2631 try 0 launched on executor multi with executor id 2632 parsl.dataflow.dflow INFO: Standard output for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7c90a5a5-e500-4824-853a-e9862e67820b_isrForCrosstalkSources_855380_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7c90a5a5-e500-4824-853a-e9862e67820b_isrForCrosstalkSources_855380_39.stderr parsl.dataflow.dflow DEBUG: Task 2632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2632 has memoization hash 233cddacce7200c32f8e79f56d6edec9 parsl.dataflow.memoization INFO: Task 2632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2632 try 0 launched on executor multi with executor id 2633 parsl.dataflow.dflow INFO: Standard output for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/007786ed-30fd-4415-901e-1c60e28c22a4_isrForCrosstalkSources_855404_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/007786ed-30fd-4415-901e-1c60e28c22a4_isrForCrosstalkSources_855404_24.stderr parsl.dataflow.dflow DEBUG: Task 2633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2633 has memoization hash a25877a7cbb4ce5fa6c396ee83ca2665 parsl.dataflow.memoization INFO: Task 2633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2633 try 0 launched on executor multi with executor id 2634 parsl.dataflow.dflow INFO: Standard output for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b8b642cd-03e8-4097-b588-ede7d72b3e69_isrForCrosstalkSources_855436_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b8b642cd-03e8-4097-b588-ede7d72b3e69_isrForCrosstalkSources_855436_3.stderr parsl.dataflow.dflow DEBUG: Task 2634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2634 has memoization hash e01c4de5eff3d4027fda93612a3b9c9b parsl.dataflow.memoization INFO: Task 2634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2634 try 0 launched on executor multi with executor id 2635 parsl.dataflow.dflow INFO: Standard output for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5536c7e0-bb6a-4fba-95c3-8d05ee534572_isrForCrosstalkSources_855454_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5536c7e0-bb6a-4fba-95c3-8d05ee534572_isrForCrosstalkSources_855454_8.stderr parsl.dataflow.dflow DEBUG: Task 2635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2635 has memoization hash 0d2b0100f3bcd585f5bf200f2388f115 parsl.dataflow.memoization INFO: Task 2635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2635 try 0 launched on executor multi with executor id 2636 parsl.dataflow.dflow INFO: Standard output for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/71ecedce-bbbe-48d5-acb1-44257e877c9c_isrForCrosstalkSources_855435_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/71ecedce-bbbe-48d5-acb1-44257e877c9c_isrForCrosstalkSources_855435_44.stderr parsl.dataflow.dflow DEBUG: Task 2636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2636 has memoization hash f9273fc14a9c9fb7b81c8b8788180557 parsl.dataflow.memoization INFO: Task 2636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2636 try 0 launched on executor multi with executor id 2637 parsl.dataflow.dflow INFO: Standard output for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/997ea58c-a6bc-43df-bac2-bb9752cb2448_isrForCrosstalkSources_855452_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/997ea58c-a6bc-43df-bac2-bb9752cb2448_isrForCrosstalkSources_855452_19.stderr parsl.dataflow.dflow DEBUG: Task 2637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2637 has memoization hash 5b7f6b71ce06237192a41a799d0ee9b7 parsl.dataflow.memoization INFO: Task 2637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:22 proc_lsst.multi:170 [INFO] [multi] job_id 16964149 proc_lsst.multi INFO: [multi] job_id 16964149 2024-03-09 14:51:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-09 14:51:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964149 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964149 parsl.executors.status_handling DEBUG: Launched block 123 on executor multi with job ID 16964149 parsl.executors.status_handling INFO: Allocated block ID 124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-124 2024-03-09 14:51:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2637 try 0 launched on executor multi with executor id 2638 parsl.dataflow.dflow INFO: Standard output for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9e228086-0263-4f38-8589-724737c64982_isrForCrosstalkSources_855431_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9e228086-0263-4f38-8589-724737c64982_isrForCrosstalkSources_855431_55.stderr parsl.dataflow.dflow DEBUG: Task 2638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2638 has memoization hash fe6c1ddd6a086ca26c3cee708fd49b9d parsl.dataflow.memoization INFO: Task 2638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2638 try 0 launched on executor multi with executor id 2639 parsl.dataflow.dflow INFO: Standard output for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/781f06e1-f676-451c-965b-7782a3b69a77_isrForCrosstalkSources_855458_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/781f06e1-f676-451c-965b-7782a3b69a77_isrForCrosstalkSources_855458_8.stderr parsl.dataflow.dflow DEBUG: Task 2639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2639 has memoization hash 86dbb1701c34a989bb31e3005427a3a0 parsl.dataflow.memoization INFO: Task 2639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2639 try 0 launched on executor multi with executor id 2640 parsl.dataflow.dflow INFO: Standard output for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f8aaa21-1a7a-4ebd-b915-eb9d4589cd97_isrForCrosstalkSources_855441_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f8aaa21-1a7a-4ebd-b915-eb9d4589cd97_isrForCrosstalkSources_855441_43.stderr parsl.dataflow.dflow DEBUG: Task 2640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2640 has memoization hash b90bbe9b554377120fbfc461d5f8dcd5 parsl.dataflow.memoization INFO: Task 2640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2640 try 0 launched on executor multi with executor id 2641 parsl.dataflow.dflow INFO: Standard output for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3d852e45-d425-4058-b522-808e288040b9_isrForCrosstalkSources_855405_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3d852e45-d425-4058-b522-808e288040b9_isrForCrosstalkSources_855405_31.stderr parsl.dataflow.dflow DEBUG: Task 2641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2641 has memoization hash f832c4f386bf04643f300eef813f2f7b parsl.dataflow.memoization INFO: Task 2641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2641 try 0 launched on executor multi with executor id 2642 parsl.dataflow.dflow INFO: Standard output for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/07db19af-e567-44f4-afff-712475211bf2_isrForCrosstalkSources_855454_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/07db19af-e567-44f4-afff-712475211bf2_isrForCrosstalkSources_855454_28.stderr parsl.dataflow.dflow DEBUG: Task 2642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2642 has memoization hash 83121c5dd6e5b06cc9ce2e35762f6f61 parsl.dataflow.memoization INFO: Task 2642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2642 try 0 launched on executor multi with executor id 2643 parsl.dataflow.dflow INFO: Standard output for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4dc85625-7a34-49b9-a6fd-107c196775bd_isrForCrosstalkSources_855430_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4dc85625-7a34-49b9-a6fd-107c196775bd_isrForCrosstalkSources_855430_4.stderr parsl.dataflow.dflow DEBUG: Task 2643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2643 has memoization hash b416f7e35c38f946176fd864cd332bb1 parsl.dataflow.memoization INFO: Task 2643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84bd926e6e6a7c478160e69382c26f27 with result from task 390 parsl.dataflow.dflow INFO: Parsl task 2643 try 0 launched on executor multi with executor id 2644 parsl.dataflow.dflow INFO: Standard output for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5b48c56d-bd0d-41cb-b03b-0cdaaa7cb52c_isrForCrosstalkSources_855435_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5b48c56d-bd0d-41cb-b03b-0cdaaa7cb52c_isrForCrosstalkSources_855435_28.stderr parsl.dataflow.dflow DEBUG: Task 2644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Task 2644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 2644 set to pending state with AppFuture: parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/196600e6-7d29-4001-86d0-ed770af2899f_isrForCrosstalkSources_855449_7.stdout parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/196600e6-7d29-4001-86d0-ed770af2899f_isrForCrosstalkSources_855449_7.stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2644 has memoization hash 888b4672728e094ec0190dacfce20644 parsl.dataflow.memoization INFO: Task 2644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3be5f2213aa876052a15b76ee8ffd99 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/f4fc59f8-92f8-40db-ba4a-5e1355b58312_isrForCrosstalkSources_855435_26.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/f4fc59f8-92f8-40db-ba4a-5e1355b58312_isrForCrosstalkSources_855435_26.stderr parsl.dataflow.dflow INFO: Parsl task 2644 try 0 launched on executor multi with executor id 2645 parsl.dataflow.dflow INFO: Standard output for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3959229d-27e3-4da7-9468-037664625658_isrForCrosstalkSources_855442_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3959229d-27e3-4da7-9468-037664625658_isrForCrosstalkSources_855442_2.stderr parsl.dataflow.dflow DEBUG: Task 2645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2645 has memoization hash 8a545224f7a8fa9c7b948ce83c704b23 parsl.dataflow.memoization INFO: Task 2645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2645 try 0 launched on executor multi with executor id 2646 parsl.dataflow.dflow INFO: Standard output for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fb1dfe06-9662-4a82-89d2-f77614de6f5d_isrForCrosstalkSources_855401_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fb1dfe06-9662-4a82-89d2-f77614de6f5d_isrForCrosstalkSources_855401_33.stderr parsl.dataflow.dflow DEBUG: Task 2646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2646 has memoization hash 04d46b7fe9786d5b5cb82095d65fdfea parsl.dataflow.memoization INFO: Task 2646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2646 try 0 launched on executor multi with executor id 2647 parsl.dataflow.dflow INFO: Standard output for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f33c1c05-9c25-40fd-83fd-67f404e13b94_isrForCrosstalkSources_855428_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f33c1c05-9c25-40fd-83fd-67f404e13b94_isrForCrosstalkSources_855428_6.stderr parsl.dataflow.dflow DEBUG: Task 2647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2647 has memoization hash e7c06c0b55d8c169b29ee44dac64a814 parsl.dataflow.memoization INFO: Task 2647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2647 try 0 launched on executor multi with executor id 2648 parsl.dataflow.dflow INFO: Standard output for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b05b4eae-518b-4370-bc8b-06bd666fa819_isrForCrosstalkSources_855414_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b05b4eae-518b-4370-bc8b-06bd666fa819_isrForCrosstalkSources_855414_3.stderr parsl.dataflow.dflow DEBUG: Task 2648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2648 has memoization hash 3be507614cc4a7dffa6678825e179dec parsl.dataflow.memoization INFO: Task 2648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2648 try 0 launched on executor multi with executor id 2649 parsl.dataflow.dflow INFO: Standard output for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a5ce2269-71ce-4cb7-91e4-e1b53d805de8_isrForCrosstalkSources_855434_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a5ce2269-71ce-4cb7-91e4-e1b53d805de8_isrForCrosstalkSources_855434_35.stderr parsl.dataflow.dflow DEBUG: Task 2649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2649 has memoization hash 987be4866672968908498d1ccc63ea97 parsl.dataflow.memoization INFO: Task 2649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2649 try 0 launched on executor multi with executor id 2650 parsl.dataflow.dflow INFO: Standard output for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d7dad6bf-e353-4717-bedb-7057a30d8add_isrForCrosstalkSources_855428_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d7dad6bf-e353-4717-bedb-7057a30d8add_isrForCrosstalkSources_855428_2.stderr parsl.dataflow.dflow DEBUG: Task 2650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2650 has memoization hash 77c0d763db62a64689c5777050e416ee parsl.dataflow.memoization INFO: Task 2650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2650 try 0 launched on executor multi with executor id 2651 parsl.dataflow.dflow INFO: Standard output for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e5563863-e8f4-46c5-b38e-4fcc737b8d75_isrForCrosstalkSources_855451_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e5563863-e8f4-46c5-b38e-4fcc737b8d75_isrForCrosstalkSources_855451_42.stderr parsl.dataflow.dflow DEBUG: Task 2651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2651 has memoization hash 7d3f54fbf1ec7720ed73a8c07ed4558d parsl.dataflow.memoization INFO: Task 2651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2651 try 0 launched on executor multi with executor id 2652 parsl.dataflow.dflow INFO: Standard output for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/df2f36ac-2e31-4bd6-937c-849cb61de4ef_isrForCrosstalkSources_855457_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/df2f36ac-2e31-4bd6-937c-849cb61de4ef_isrForCrosstalkSources_855457_40.stderr parsl.dataflow.dflow DEBUG: Task 2652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2652 has memoization hash 30a9088d1f30a41f11ea12e5fc6d5a94 parsl.dataflow.memoization INFO: Task 2652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2652 try 0 launched on executor multi with executor id 2653 parsl.dataflow.dflow INFO: Standard output for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/63625687-3f84-4379-8eb1-fd2af4f327fc_isrForCrosstalkSources_855389_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/63625687-3f84-4379-8eb1-fd2af4f327fc_isrForCrosstalkSources_855389_23.stderr parsl.dataflow.dflow DEBUG: Task 2653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2653 has memoization hash d0a29c778ca9e8f10b29a9373fcf5252 parsl.dataflow.memoization INFO: Task 2653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2653 try 0 launched on executor multi with executor id 2654 parsl.dataflow.dflow INFO: Standard output for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d96ed436-c274-429a-b949-3c48da0d2ed9_isrForCrosstalkSources_855421_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d96ed436-c274-429a-b949-3c48da0d2ed9_isrForCrosstalkSources_855421_50.stderr parsl.dataflow.dflow DEBUG: Task 2654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2654 has memoization hash afc4719ec6eeb7fe99cec21744c91c4f parsl.dataflow.memoization INFO: Task 2654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2654 try 0 launched on executor multi with executor id 2655 parsl.dataflow.dflow INFO: Standard output for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d8451d6f-4596-45ee-9b38-abcc3dff143f_isrForCrosstalkSources_855428_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d8451d6f-4596-45ee-9b38-abcc3dff143f_isrForCrosstalkSources_855428_41.stderr parsl.dataflow.dflow DEBUG: Task 2655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2655 has memoization hash 7493dc983cc1d42a239316979c5295e5 parsl.dataflow.memoization INFO: Task 2655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2655 try 0 launched on executor multi with executor id 2656 parsl.dataflow.dflow INFO: Standard output for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c672642f-b9d7-457e-8f1a-c981a13417c5_isrForCrosstalkSources_855451_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c672642f-b9d7-457e-8f1a-c981a13417c5_isrForCrosstalkSources_855451_3.stderr parsl.dataflow.dflow DEBUG: Task 2656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2656 has memoization hash e1cfb2b71ecf7b7b7e16985aeba5366b parsl.dataflow.memoization INFO: Task 2656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2656 try 0 launched on executor multi with executor id 2657 parsl.dataflow.dflow INFO: Standard output for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d35e5f44-169e-427b-a90e-991f7020afde_isrForCrosstalkSources_855428_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d35e5f44-169e-427b-a90e-991f7020afde_isrForCrosstalkSources_855428_7.stderr parsl.dataflow.dflow DEBUG: Task 2657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2657 has memoization hash 67b21f0b6cf281ef318cebd0a9e03fb0 parsl.dataflow.memoization INFO: Task 2657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2657 try 0 launched on executor multi with executor id 2658 parsl.dataflow.dflow INFO: Standard output for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2b9c421d-9453-4d1d-837e-114476e6dafe_isrForCrosstalkSources_855421_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2b9c421d-9453-4d1d-837e-114476e6dafe_isrForCrosstalkSources_855421_14.stderr parsl.dataflow.dflow DEBUG: Task 2658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2658 has memoization hash e48ea1240305c28bd396f3c0bdebb2a1 parsl.dataflow.memoization INFO: Task 2658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2658 try 0 launched on executor multi with executor id 2659 parsl.dataflow.dflow INFO: Standard output for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b876dae-f02f-4d3f-a7c4-99d9858b59e2_isrForCrosstalkSources_855416_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b876dae-f02f-4d3f-a7c4-99d9858b59e2_isrForCrosstalkSources_855416_9.stderr parsl.dataflow.dflow DEBUG: Task 2659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2659 has memoization hash 239d5088575a2941375b8a2acbdd9abc parsl.dataflow.memoization INFO: Task 2659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47cbe44febcd804f54035466391da83c with result from task 403 parsl.dataflow.dflow INFO: Parsl task 2659 try 0 launched on executor multi with executor id 2660 parsl.dataflow.dflow INFO: Standard output for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/d14bb54f-2837-4c76-8b88-1cc00999e011_isrForCrosstalkSources_855412_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/d14bb54f-2837-4c76-8b88-1cc00999e011_isrForCrosstalkSources_855412_40.stderr parsl.dataflow.dflow DEBUG: Task 2660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2660 has memoization hash 305dc3b88c7fd65246af4aef1ad5fc3f parsl.dataflow.memoization INFO: Task 2660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c1be2b2f-900e-428e-940e-8f70f174f550_isrForCrosstalkSources_855387_45.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c1be2b2f-900e-428e-940e-8f70f174f550_isrForCrosstalkSources_855387_45.stderr parsl.dataflow.dflow INFO: Parsl task 2660 try 0 launched on executor multi with executor id 2661 parsl.dataflow.dflow INFO: Standard output for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/34a0477b-482f-4b82-abf6-66322e01e7da_isrForCrosstalkSources_855390_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/34a0477b-482f-4b82-abf6-66322e01e7da_isrForCrosstalkSources_855390_20.stderr parsl.dataflow.dflow DEBUG: Task 2661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2661 has memoization hash 92c0a6b3cf5bb773c384746ccfcf93ca parsl.dataflow.memoization INFO: Task 2661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2661 try 0 launched on executor multi with executor id 2662 parsl.dataflow.dflow INFO: Standard output for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e7d1f125-8966-4c52-ae66-f3a6b419e131_isrForCrosstalkSources_855384_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e7d1f125-8966-4c52-ae66-f3a6b419e131_isrForCrosstalkSources_855384_32.stderr parsl.dataflow.dflow DEBUG: Task 2662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2662 has memoization hash b7512a21f81694e721f5b5eebd4a6318 parsl.dataflow.memoization INFO: Task 2662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2662 try 0 launched on executor multi with executor id 2663 parsl.dataflow.dflow INFO: Standard output for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/47517000-238b-4994-aaa6-b7716f69f751_isrForCrosstalkSources_855446_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/47517000-238b-4994-aaa6-b7716f69f751_isrForCrosstalkSources_855446_18.stderr parsl.dataflow.dflow DEBUG: Task 2663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2663 has memoization hash 6e951975dde71d4dcdb3e1d2466c7e9a parsl.dataflow.memoization INFO: Task 2663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2663 try 0 launched on executor multi with executor id 2664 parsl.dataflow.dflow INFO: Standard output for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea7baaf7-5ac0-4483-a398-352622121fb2_isrForCrosstalkSources_855390_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea7baaf7-5ac0-4483-a398-352622121fb2_isrForCrosstalkSources_855390_1.stderr parsl.dataflow.dflow DEBUG: Task 2664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2664 has memoization hash cbb8ca2d50525378b870fc8a32e16690 parsl.dataflow.memoization INFO: Task 2664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2664 try 0 launched on executor multi with executor id 2665 parsl.dataflow.dflow INFO: Standard output for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b039d81a-c362-49c2-9afe-839061cf75ca_isrForCrosstalkSources_855435_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b039d81a-c362-49c2-9afe-839061cf75ca_isrForCrosstalkSources_855435_11.stderr parsl.dataflow.dflow DEBUG: Task 2665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2665 has memoization hash 0feff71e3b00a077e343116599c15b5e parsl.dataflow.memoization INFO: Task 2665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2665 try 0 launched on executor multi with executor id 2666 parsl.dataflow.dflow INFO: Standard output for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dcc0f1a4-1b0b-48b9-838d-48218d617e08_isrForCrosstalkSources_855456_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dcc0f1a4-1b0b-48b9-838d-48218d617e08_isrForCrosstalkSources_855456_27.stderr parsl.dataflow.dflow DEBUG: Task 2666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2666 has memoization hash 862fce69e013523992312d78f4ebe59b parsl.dataflow.memoization INFO: Task 2666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2666 try 0 launched on executor multi with executor id 2667 parsl.dataflow.dflow INFO: Standard output for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/3e10f6fd-e4f8-4039-9ce5-d3d4f84ec232_isrForCrosstalkSources_855428_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/3e10f6fd-e4f8-4039-9ce5-d3d4f84ec232_isrForCrosstalkSources_855428_16.stderr parsl.dataflow.dflow DEBUG: Task 2667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2667 has memoization hash 848032236dba575c13d31f0fc5f84eda parsl.dataflow.memoization INFO: Task 2667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2667 try 0 launched on executor multi with executor id 2668 parsl.dataflow.dflow INFO: Standard output for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da1e43a8-985e-42bc-a0f7-fb18c72dadf6_isrForCrosstalkSources_855404_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da1e43a8-985e-42bc-a0f7-fb18c72dadf6_isrForCrosstalkSources_855404_9.stderr parsl.dataflow.dflow DEBUG: Task 2668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2668 has memoization hash b15ac171608f940d4d8f43249ef49977 parsl.dataflow.memoization INFO: Task 2668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2668 try 0 launched on executor multi with executor id 2669 parsl.dataflow.dflow INFO: Standard output for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0cda55cd-c7fe-4710-a60b-6d0ad36a8b90_isrForCrosstalkSources_855457_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0cda55cd-c7fe-4710-a60b-6d0ad36a8b90_isrForCrosstalkSources_855457_55.stderr parsl.dataflow.dflow DEBUG: Task 2669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2669 has memoization hash 61d3e0d07e044331c3ce813784bdd451 parsl.dataflow.memoization INFO: Task 2669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2669 try 0 launched on executor multi with executor id 2670 parsl.dataflow.dflow INFO: Standard output for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d149318b-9ce7-4ea7-8c1a-3ac417694eab_isrForCrosstalkSources_855416_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d149318b-9ce7-4ea7-8c1a-3ac417694eab_isrForCrosstalkSources_855416_40.stderr parsl.dataflow.dflow DEBUG: Task 2670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2670 has memoization hash 00ac30a88a7a9399ceac7937c1c9be75 parsl.dataflow.memoization INFO: Task 2670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2670 try 0 launched on executor multi with executor id 2671 parsl.dataflow.dflow INFO: Standard output for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/15af6f28-0b1e-4728-8687-2fd2d0b50ad3_isrForCrosstalkSources_855433_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/15af6f28-0b1e-4728-8687-2fd2d0b50ad3_isrForCrosstalkSources_855433_10.stderr parsl.dataflow.dflow DEBUG: Task 2671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2671 has memoization hash 7077324106a329ec20ee555bee782975 parsl.dataflow.memoization INFO: Task 2671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2671 try 0 launched on executor multi with executor id 2672 parsl.dataflow.dflow INFO: Standard output for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/79273be5-bbed-47a9-8f96-002e62ee06bb_isrForCrosstalkSources_855437_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/79273be5-bbed-47a9-8f96-002e62ee06bb_isrForCrosstalkSources_855437_19.stderr parsl.dataflow.dflow DEBUG: Task 2672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2672 has memoization hash 70184acfcef2158aa59a9ece8440805d parsl.dataflow.memoization INFO: Task 2672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2672 try 0 launched on executor multi with executor id 2673 parsl.dataflow.dflow INFO: Standard output for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/58ea4a9f-d77c-4b48-bbf2-6c1b01f569cc_isrForCrosstalkSources_855429_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/58ea4a9f-d77c-4b48-bbf2-6c1b01f569cc_isrForCrosstalkSources_855429_35.stderr parsl.dataflow.dflow DEBUG: Task 2673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2673 has memoization hash 5f1cfb96bcf71e3ca4614ee4b0957044 parsl.dataflow.memoization INFO: Task 2673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:22 proc_lsst.multi:170 [INFO] [multi] job_id 16964150 proc_lsst.multi INFO: [multi] job_id 16964150 2024-03-09 14:51:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-09 14:51:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964150 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964150 parsl.executors.status_handling DEBUG: Launched block 124 on executor multi with job ID 16964150 parsl.executors.status_handling INFO: Allocated block ID 125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-125 2024-03-09 14:51:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-09 14:51:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2673 try 0 launched on executor multi with executor id 2674 parsl.dataflow.dflow INFO: Standard output for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c641d690-288c-4639-bb9c-e1ffec6b1231_isrForCrosstalkSources_855452_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c641d690-288c-4639-bb9c-e1ffec6b1231_isrForCrosstalkSources_855452_28.stderr parsl.dataflow.dflow DEBUG: Task 2674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2674 has memoization hash 83d81ee32c222af40b3f9144f5ac4ae2 parsl.dataflow.memoization INFO: Task 2674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a64d3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2674 try 0 launched on executor multi with executor id 2675 parsl.dataflow.dflow INFO: Standard output for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2efc5bc8-43eb-4c68-8424-b8a39325e0e3_isrForCrosstalkSources_855382_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2efc5bc8-43eb-4c68-8424-b8a39325e0e3_isrForCrosstalkSources_855382_21.stderr parsl.dataflow.dflow DEBUG: Task 2675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2675 has memoization hash 445cafc79c9811491d6ce19e2be7791f parsl.dataflow.memoization INFO: Task 2675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2675 try 0 launched on executor multi with executor id 2676 parsl.dataflow.dflow INFO: Standard output for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3cda799c-5382-4425-97c2-68ae7ee6a2f8_isrForCrosstalkSources_855389_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3cda799c-5382-4425-97c2-68ae7ee6a2f8_isrForCrosstalkSources_855389_17.stderr parsl.dataflow.dflow DEBUG: Task 2676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2676 has memoization hash b8f0248e28e043ec878361c32b2f68cb parsl.dataflow.memoization INFO: Task 2676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2676 try 0 launched on executor multi with executor id 2677 parsl.dataflow.dflow INFO: Standard output for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8f64edcd-8389-4fbe-94ad-75dc2911c61c_isrForCrosstalkSources_855392_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8f64edcd-8389-4fbe-94ad-75dc2911c61c_isrForCrosstalkSources_855392_21.stderr parsl.dataflow.dflow DEBUG: Task 2677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2677 has memoization hash 996d66aab58fbd5b8cd445ace6558990 parsl.dataflow.memoization INFO: Task 2677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2677 try 0 launched on executor multi with executor id 2678 parsl.dataflow.dflow INFO: Standard output for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2d818d0f-890a-447f-a2ff-fa732ceba02f_isrForCrosstalkSources_855452_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2d818d0f-890a-447f-a2ff-fa732ceba02f_isrForCrosstalkSources_855452_52.stderr parsl.dataflow.dflow DEBUG: Task 2678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2678 has memoization hash eeb2730c159343b52a020ea9b1122066 parsl.dataflow.memoization INFO: Task 2678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2678 try 0 launched on executor multi with executor id 2679 parsl.dataflow.dflow INFO: Standard output for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c74a43a6-8c6e-4a26-9129-cb5ebc6af2d5_isrForCrosstalkSources_855427_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c74a43a6-8c6e-4a26-9129-cb5ebc6af2d5_isrForCrosstalkSources_855427_44.stderr parsl.dataflow.dflow DEBUG: Task 2679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2679 has memoization hash ba7b7508214b98e7c597ebdb05fe65e7 parsl.dataflow.memoization INFO: Task 2679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2679 try 0 launched on executor multi with executor id 2680 parsl.dataflow.dflow INFO: Standard output for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b1cb9fa3-b826-4dfb-814d-1abf50875437_isrForCrosstalkSources_855436_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b1cb9fa3-b826-4dfb-814d-1abf50875437_isrForCrosstalkSources_855436_59.stderr parsl.dataflow.dflow DEBUG: Task 2680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2680 has memoization hash 41f20fc0dbe85f6c30db303c426544cc parsl.dataflow.memoization INFO: Task 2680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7d934e6026b3c09c421975d86435571 with result from task 414 parsl.dataflow.dflow INFO: Parsl task 2680 try 0 launched on executor multi with executor id 2681 parsl.dataflow.dflow INFO: Standard output for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e5f518a5-e826-4136-ad01-c42021713a75_isrForCrosstalkSources_855384_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e5f518a5-e826-4136-ad01-c42021713a75_isrForCrosstalkSources_855384_30.stderr parsl.dataflow.dflow DEBUG: Task 2681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2681 has memoization hash a7c9b7e748a387c1f626e2c590ffb65c parsl.dataflow.memoization INFO: Task 2681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2681 try 0 launched on executor multi with executor id 2682 parsl.dataflow.dflow INFO: Standard output for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b356bd46-a268-4247-ae2f-f1cff0c531fd_isrForCrosstalkSources_855426_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b356bd46-a268-4247-ae2f-f1cff0c531fd_isrForCrosstalkSources_855426_20.stderr parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b791784c-6f9e-498f-8b1b-57b0c9473d81_isrForCrosstalkSources_855447_37.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b791784c-6f9e-498f-8b1b-57b0c9473d81_isrForCrosstalkSources_855447_37.stderr parsl.dataflow.dflow DEBUG: Task 2682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2682 has memoization hash 9f520003b95a48ea25cb065c4fa28360 parsl.dataflow.memoization INFO: Task 2682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2682 try 0 launched on executor multi with executor id 2683 parsl.dataflow.dflow INFO: Standard output for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fa6edf02-39c7-4ada-902c-3ed4d97fed56_isrForCrosstalkSources_855417_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fa6edf02-39c7-4ada-902c-3ed4d97fed56_isrForCrosstalkSources_855417_10.stderr parsl.dataflow.dflow DEBUG: Task 2683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2683 has memoization hash feca53257fb49468538b252e487f2d52 parsl.dataflow.memoization INFO: Task 2683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2683 try 0 launched on executor multi with executor id 2684 parsl.dataflow.dflow INFO: Standard output for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/df1739c4-bdaa-4a23-901d-68131c907316_isrForCrosstalkSources_855379_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/df1739c4-bdaa-4a23-901d-68131c907316_isrForCrosstalkSources_855379_9.stderr parsl.dataflow.dflow DEBUG: Task 2684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2684 has memoization hash e15db26a8b53f3c7404152ddb21ecf6e parsl.dataflow.memoization INFO: Task 2684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2684 try 0 launched on executor multi with executor id 2685 parsl.dataflow.dflow INFO: Standard output for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/728b185e-f293-487c-b945-e1fbcdd6006f_isrForCrosstalkSources_855412_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/728b185e-f293-487c-b945-e1fbcdd6006f_isrForCrosstalkSources_855412_58.stderr parsl.dataflow.dflow DEBUG: Task 2685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2685 has memoization hash 08fbc75ecb1c92765a6eff9057428e26 parsl.dataflow.memoization INFO: Task 2685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2685 try 0 launched on executor multi with executor id 2686 parsl.dataflow.dflow INFO: Standard output for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4cb625d7-df84-40b1-a313-8ab40ec33ed6_isrForCrosstalkSources_855379_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4cb625d7-df84-40b1-a313-8ab40ec33ed6_isrForCrosstalkSources_855379_50.stderr parsl.dataflow.dflow DEBUG: Task 2686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2686 has memoization hash 7ecac04293d33b044799f72306a3a701 parsl.dataflow.memoization INFO: Task 2686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2686 try 0 launched on executor multi with executor id 2687 parsl.dataflow.dflow INFO: Standard output for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1fc3cca6-46e0-4310-82a6-f2ddc955abf2_isrForCrosstalkSources_855383_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1fc3cca6-46e0-4310-82a6-f2ddc955abf2_isrForCrosstalkSources_855383_49.stderr parsl.dataflow.dflow DEBUG: Task 2687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2687 has memoization hash 395411c07b4d0b720116b10301eed47b parsl.dataflow.memoization INFO: Task 2687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2687 try 0 launched on executor multi with executor id 2688 parsl.dataflow.dflow INFO: Standard output for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a8992d08-a00b-4ef9-9c67-899168504c8c_isrForCrosstalkSources_855438_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a8992d08-a00b-4ef9-9c67-899168504c8c_isrForCrosstalkSources_855438_55.stderr parsl.dataflow.dflow DEBUG: Task 2688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2688 has memoization hash 73bdc4aa3800f5b222f161bf39a03a82 parsl.dataflow.memoization INFO: Task 2688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2688 try 0 launched on executor multi with executor id 2689 parsl.dataflow.dflow INFO: Standard output for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1d8d4eb0-2d21-4b62-ac5b-cb528611696b_isrForCrosstalkSources_855423_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1d8d4eb0-2d21-4b62-ac5b-cb528611696b_isrForCrosstalkSources_855423_11.stderr parsl.dataflow.dflow DEBUG: Task 2689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2689 has memoization hash b39f68706e6c7331654a94daacaa005a parsl.dataflow.memoization INFO: Task 2689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2689 try 0 launched on executor multi with executor id 2690 parsl.dataflow.dflow INFO: Standard output for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/c5bd018c-0d80-452d-b2e0-8385c7cb9af2_isrForCrosstalkSources_855437_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/c5bd018c-0d80-452d-b2e0-8385c7cb9af2_isrForCrosstalkSources_855437_3.stderr parsl.dataflow.dflow DEBUG: Task 2690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2690 has memoization hash 49e0400149b78718ca76ffc6600e2057 parsl.dataflow.memoization INFO: Task 2690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e7600a25273f04e3b3407bc1d5782be with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/307da723-04e8-48bb-9893-c148a3dd434c_isrForCrosstalkSources_855448_14.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/307da723-04e8-48bb-9893-c148a3dd434c_isrForCrosstalkSources_855448_14.stderr parsl.dataflow.dflow INFO: Parsl task 2690 try 0 launched on executor multi with executor id 2691 parsl.dataflow.dflow INFO: Standard output for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7ecc3dc4-2653-4c38-894c-c49a277ce2b4_isrForCrosstalkSources_855401_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7ecc3dc4-2653-4c38-894c-c49a277ce2b4_isrForCrosstalkSources_855401_25.stderr parsl.dataflow.dflow DEBUG: Task 2691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2691 has memoization hash b048e5a29f0183725335c1b36e184fe2 parsl.dataflow.memoization INFO: Task 2691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8322c920c7f158ad4ee56a3a2d4c9191 with result from task 442 parsl.dataflow.dflow INFO: Parsl task 2691 try 0 launched on executor multi with executor id 2692 parsl.dataflow.dflow INFO: Standard output for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3d812fe9-28ea-45ff-a502-898cbbf59496_isrForCrosstalkSources_855397_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3d812fe9-28ea-45ff-a502-898cbbf59496_isrForCrosstalkSources_855397_41.stderr parsl.dataflow.dflow DEBUG: Task 2692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2692 has memoization hash efe168efbf7c1205c32e0bdb9996cd1d parsl.dataflow.memoization INFO: Task 2692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2692 try 0 launched on executor multi with executor id 2693 parsl.dataflow.dflow INFO: Standard output for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/96b92f8c-2bec-4eba-bc89-72b7d998ceaa_isrForCrosstalkSources_855447_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/96b92f8c-2bec-4eba-bc89-72b7d998ceaa_isrForCrosstalkSources_855447_19.stderr parsl.dataflow.dflow DEBUG: Task 2693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2693 has memoization hash 097ea7774f20dc7a25698d9dc5969ac1 parsl.dataflow.memoization INFO: Task 2693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2693 try 0 launched on executor multi with executor id 2694 parsl.dataflow.dflow INFO: Standard output for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7c57a126-009c-4f8d-9ebb-e1196b20c81c_isrForCrosstalkSources_855457_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7c57a126-009c-4f8d-9ebb-e1196b20c81c_isrForCrosstalkSources_855457_21.stderr parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/82657c2e-d87d-43c5-aba1-fb36e696ecac_isrForCrosstalkSources_855437_33.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/82657c2e-d87d-43c5-aba1-fb36e696ecac_isrForCrosstalkSources_855437_33.stderr parsl.dataflow.dflow DEBUG: Task 2694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2694 has memoization hash e84f66b552794b81642d8114e9162a81 parsl.dataflow.memoization INFO: Task 2694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2694 try 0 launched on executor multi with executor id 2695 parsl.dataflow.dflow INFO: Standard output for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0453c26-4031-4683-b84d-901c512e1077_isrForCrosstalkSources_855408_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0453c26-4031-4683-b84d-901c512e1077_isrForCrosstalkSources_855408_10.stderr parsl.dataflow.dflow DEBUG: Task 2695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2695 has memoization hash c6102a828006d478c2ea5c36c1d9363a parsl.dataflow.memoization INFO: Task 2695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2695 try 0 launched on executor multi with executor id 2696 parsl.dataflow.dflow INFO: Standard output for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54547a89-ea3b-49a4-bd9a-d227fdc3778d_isrForCrosstalkSources_855445_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54547a89-ea3b-49a4-bd9a-d227fdc3778d_isrForCrosstalkSources_855445_18.stderr parsl.dataflow.dflow DEBUG: Task 2696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2696 has memoization hash 9c65ce79fa2f787a2a0f776a295040ab parsl.dataflow.memoization INFO: Task 2696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2696 try 0 launched on executor multi with executor id 2697 parsl.dataflow.dflow INFO: Standard output for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf57d470-06e1-4617-b263-4f35542493e8_isrForCrosstalkSources_855432_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf57d470-06e1-4617-b263-4f35542493e8_isrForCrosstalkSources_855432_19.stderr parsl.dataflow.dflow DEBUG: Task 2697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2697 has memoization hash a7b6301cb21fd1d5540229a509acd979 parsl.dataflow.memoization INFO: Task 2697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2697 try 0 launched on executor multi with executor id 2698 parsl.dataflow.dflow INFO: Standard output for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/3a347184-1302-4745-9f16-c3e31b350a9e_isrForCrosstalkSources_855398_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/3a347184-1302-4745-9f16-c3e31b350a9e_isrForCrosstalkSources_855398_13.stderr parsl.dataflow.dflow DEBUG: Task 2698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2698 has memoization hash 822e245cf3ab435b29aea18b7a1425b8 parsl.dataflow.memoization INFO: Task 2698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2698 try 0 launched on executor multi with executor id 2699 parsl.dataflow.dflow INFO: Standard output for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8b46c0d7-6c64-4478-a658-18a7b602a0f1_isrForCrosstalkSources_855443_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8b46c0d7-6c64-4478-a658-18a7b602a0f1_isrForCrosstalkSources_855443_51.stderr parsl.dataflow.dflow DEBUG: Task 2699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2699 has memoization hash 30603e0f9b9e1bcdb3d4c6b2d3e997d8 parsl.dataflow.memoization INFO: Task 2699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:23 proc_lsst.multi:170 [INFO] [multi] job_id 16964151 proc_lsst.multi INFO: [multi] job_id 16964151 2024-03-09 14:51:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-09 14:51:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964151 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964151 parsl.executors.status_handling DEBUG: Launched block 125 on executor multi with job ID 16964151 parsl.executors.status_handling INFO: Allocated block ID 126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-126 2024-03-09 14:51:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2699 try 0 launched on executor multi with executor id 2700 parsl.dataflow.dflow INFO: Standard output for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3109761e-6e0f-47f2-816c-04aa16e3853b_isrForCrosstalkSources_855397_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3109761e-6e0f-47f2-816c-04aa16e3853b_isrForCrosstalkSources_855397_61.stderr parsl.dataflow.dflow DEBUG: Task 2700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2700 has memoization hash b914f615082f36175f1b4968fe842eb9 parsl.dataflow.memoization INFO: Task 2700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2700 try 0 launched on executor multi with executor id 2701 parsl.dataflow.dflow INFO: Standard output for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67d94c7a-ae78-45cc-96e6-c8574b419661_isrForCrosstalkSources_855393_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67d94c7a-ae78-45cc-96e6-c8574b419661_isrForCrosstalkSources_855393_47.stderr parsl.dataflow.dflow DEBUG: Task 2701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2701 has memoization hash db670316144c9b5b263f205a6c9e706d parsl.dataflow.memoization INFO: Task 2701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2701 try 0 launched on executor multi with executor id 2702 parsl.dataflow.dflow INFO: Standard output for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/064e6457-75bf-4b1f-b109-06fd55366be8_isrForCrosstalkSources_855405_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/064e6457-75bf-4b1f-b109-06fd55366be8_isrForCrosstalkSources_855405_43.stderr parsl.dataflow.dflow DEBUG: Task 2702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2702 has memoization hash cce15a342631d1f9446319d7e7e1c95e parsl.dataflow.memoization INFO: Task 2702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63237e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2702 try 0 launched on executor multi with executor id 2703 parsl.dataflow.dflow INFO: Standard output for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/15fb0303-717a-4434-97d8-01b7704c233f_isrForCrosstalkSources_855436_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/15fb0303-717a-4434-97d8-01b7704c233f_isrForCrosstalkSources_855436_19.stderr parsl.dataflow.dflow DEBUG: Task 2703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2703 has memoization hash a40354c37330ef7a37ca85ec70f2e5ff parsl.dataflow.memoization INFO: Task 2703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63232e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2703 try 0 launched on executor multi with executor id 2704 parsl.dataflow.dflow INFO: Standard output for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/913b6b8d-af6b-45af-acbd-47481a269765_isrForCrosstalkSources_855428_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/913b6b8d-af6b-45af-acbd-47481a269765_isrForCrosstalkSources_855428_51.stderr parsl.dataflow.dflow DEBUG: Task 2704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2704 has memoization hash 8eae43dd389791e680676154d6f295f9 parsl.dataflow.memoization INFO: Task 2704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2704 try 0 launched on executor multi with executor id 2705 parsl.dataflow.dflow INFO: Standard output for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a61819b-7fc2-44a7-8017-75682d1e1fd0_isrForCrosstalkSources_855396_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a61819b-7fc2-44a7-8017-75682d1e1fd0_isrForCrosstalkSources_855396_3.stderr parsl.dataflow.dflow DEBUG: Task 2705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2705 has memoization hash 0cb2cf64159638e5cf5bea1db70bd560 parsl.dataflow.memoization INFO: Task 2705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2705 try 0 launched on executor multi with executor id 2706 parsl.dataflow.dflow INFO: Standard output for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/78ec3619-04b1-41c9-8373-9ba64550b6a5_isrForCrosstalkSources_855406_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/78ec3619-04b1-41c9-8373-9ba64550b6a5_isrForCrosstalkSources_855406_29.stderr parsl.dataflow.dflow DEBUG: Task 2706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2706 has memoization hash 583acab1dbb628f663b5bb7cdb6df788 parsl.dataflow.memoization INFO: Task 2706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2706 try 0 launched on executor multi with executor id 2707 parsl.dataflow.dflow INFO: Standard output for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cba82d1-d5dd-4b92-8841-78e3e6afe878_isrForCrosstalkSources_855403_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cba82d1-d5dd-4b92-8841-78e3e6afe878_isrForCrosstalkSources_855403_27.stderr parsl.dataflow.dflow DEBUG: Task 2707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.dflow DEBUG: Task 2707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Storing app cache entry d0097fb15f6c8e6fc5ef39736ad9cc42 with result from task 405 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2707 has memoization hash 966215fd33b61e3b6995c5b951c6a9f2 parsl.dataflow.memoization INFO: Task 2707 had no result in cache parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5a533f14-4490-47a4-a3f0-36330b79ccc0_isrForCrosstalkSources_855389_16.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5a533f14-4490-47a4-a3f0-36330b79ccc0_isrForCrosstalkSources_855389_16.stderr parsl.dataflow.dflow INFO: Parsl task 2707 try 0 launched on executor multi with executor id 2708 parsl.dataflow.dflow INFO: Standard output for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/54bd31fa-093b-4e49-afb0-1001d4ba52b9_isrForCrosstalkSources_855401_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/54bd31fa-093b-4e49-afb0-1001d4ba52b9_isrForCrosstalkSources_855401_15.stderr parsl.dataflow.dflow DEBUG: Task 2708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2708 has memoization hash b46c578aec35b8b61791714b594af737 parsl.dataflow.memoization INFO: Task 2708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2708 try 0 launched on executor multi with executor id 2709 parsl.dataflow.dflow INFO: Standard output for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9316aed5-edf1-4755-a4ee-b6dcb24b8a4d_isrForCrosstalkSources_855450_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9316aed5-edf1-4755-a4ee-b6dcb24b8a4d_isrForCrosstalkSources_855450_13.stderr parsl.dataflow.dflow DEBUG: Task 2709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2709 has memoization hash 2d8fd22f55e7120c751e6690a58d02fc parsl.dataflow.memoization INFO: Task 2709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2709 try 0 launched on executor multi with executor id 2710 parsl.dataflow.dflow INFO: Standard output for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e730a62d-95d6-4995-b5f9-146160b2f83a_isrForCrosstalkSources_855396_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e730a62d-95d6-4995-b5f9-146160b2f83a_isrForCrosstalkSources_855396_28.stderr parsl.dataflow.dflow DEBUG: Task 2710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2710 has memoization hash d08950c486653cb6bb9d7886f1808c93 parsl.dataflow.memoization INFO: Task 2710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9e20836e414f4eeb825fea042cde1d with result from task 170 parsl.dataflow.dflow INFO: Parsl task 2710 try 0 launched on executor multi with executor id 2711 parsl.dataflow.dflow INFO: Standard output for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c0e3d1e3-57dc-4a2f-908b-8b51efdaf377_isrForCrosstalkSources_855400_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c0e3d1e3-57dc-4a2f-908b-8b51efdaf377_isrForCrosstalkSources_855400_18.stderr parsl.dataflow.dflow DEBUG: Task 2711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2711 has memoization hash e100bfb4d12b36757460c40dce3898cf parsl.dataflow.memoization INFO: Task 2711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/56733b72-7a28-4124-b6ca-6bb6b2497b1d_isrForCrosstalkSources_855382_10.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/56733b72-7a28-4124-b6ca-6bb6b2497b1d_isrForCrosstalkSources_855382_10.stderr parsl.dataflow.dflow INFO: Parsl task 2711 try 0 launched on executor multi with executor id 2712 parsl.dataflow.dflow INFO: Standard output for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/dc82d449-a567-4d41-9af7-b11ae3c1dae4_isrForCrosstalkSources_855422_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/dc82d449-a567-4d41-9af7-b11ae3c1dae4_isrForCrosstalkSources_855422_14.stderr parsl.dataflow.dflow DEBUG: Task 2712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2712 has memoization hash 90348297cd05aa5cadcfb356257b99ad parsl.dataflow.memoization INFO: Task 2712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2712 try 0 launched on executor multi with executor id 2713 parsl.dataflow.dflow INFO: Standard output for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/19eb876b-de49-40da-a680-f00bb229be51_isrForCrosstalkSources_855430_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/19eb876b-de49-40da-a680-f00bb229be51_isrForCrosstalkSources_855430_35.stderr parsl.dataflow.dflow DEBUG: Task 2713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2713 has memoization hash 242550a2dd6049bbf39cb545ff2106ed parsl.dataflow.memoization INFO: Task 2713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2713 try 0 launched on executor multi with executor id 2714 parsl.dataflow.dflow INFO: Standard output for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e3aa41c4-038e-48e3-a0c7-248f34ef7ddd_isrForCrosstalkSources_855433_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e3aa41c4-038e-48e3-a0c7-248f34ef7ddd_isrForCrosstalkSources_855433_4.stderr parsl.dataflow.dflow DEBUG: Task 2714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2714 has memoization hash d1aecf38f9d4099c827b0b3bff3ee643 parsl.dataflow.memoization INFO: Task 2714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2714 try 0 launched on executor multi with executor id 2715 parsl.dataflow.dflow INFO: Standard output for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fc2b0fbe-24d8-4680-802c-af69d8bf90f0_isrForCrosstalkSources_855440_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fc2b0fbe-24d8-4680-802c-af69d8bf90f0_isrForCrosstalkSources_855440_27.stderr parsl.dataflow.dflow DEBUG: Task 2715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2715 has memoization hash 550b47f0b070184325f769b9af3d71b6 parsl.dataflow.memoization INFO: Task 2715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2715 try 0 launched on executor multi with executor id 2716 parsl.dataflow.dflow INFO: Standard output for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/efbe6e01-125a-4090-bf13-27d72ec54073_isrForCrosstalkSources_855433_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/efbe6e01-125a-4090-bf13-27d72ec54073_isrForCrosstalkSources_855433_18.stderr parsl.dataflow.dflow DEBUG: Task 2716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2716 has memoization hash 6e90c3448a01776665a3b442ff1ce47d parsl.dataflow.memoization INFO: Task 2716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2716 try 0 launched on executor multi with executor id 2717 parsl.dataflow.dflow INFO: Standard output for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/646a5aba-7123-4da3-9424-65317f26d806_isrForCrosstalkSources_855459_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/646a5aba-7123-4da3-9424-65317f26d806_isrForCrosstalkSources_855459_56.stderr parsl.dataflow.dflow DEBUG: Task 2717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2717 has memoization hash 71bf552a2d0edb565e42be25344ca257 parsl.dataflow.memoization INFO: Task 2717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2717 try 0 launched on executor multi with executor id 2718 parsl.dataflow.dflow INFO: Standard output for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f74e9ded-9300-493a-9143-bb8b1d41fb2b_isrForCrosstalkSources_855458_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f74e9ded-9300-493a-9143-bb8b1d41fb2b_isrForCrosstalkSources_855458_38.stderr parsl.dataflow.dflow DEBUG: Task 2718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2718 has memoization hash 5586320a4b9c206bcb446c4ed8055625 parsl.dataflow.memoization INFO: Task 2718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2718 try 0 launched on executor multi with executor id 2719 parsl.dataflow.dflow INFO: Standard output for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/011bc451-d838-491e-8a54-b370a0f6fc52_isrForCrosstalkSources_855410_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/011bc451-d838-491e-8a54-b370a0f6fc52_isrForCrosstalkSources_855410_19.stderr parsl.dataflow.dflow DEBUG: Task 2719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2719 has memoization hash 67f84c5981bc66b78d9b938bff37eada parsl.dataflow.memoization INFO: Task 2719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2719 try 0 launched on executor multi with executor id 2720 parsl.dataflow.dflow INFO: Standard output for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/775057d3-c7fd-46a7-9e87-585c93ec8ff1_isrForCrosstalkSources_855382_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/775057d3-c7fd-46a7-9e87-585c93ec8ff1_isrForCrosstalkSources_855382_42.stderr parsl.dataflow.dflow DEBUG: Task 2720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2720 has memoization hash b2cb116fec73d0bba2efbf6c9217e4af parsl.dataflow.memoization INFO: Task 2720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2720 try 0 launched on executor multi with executor id 2721 parsl.dataflow.dflow INFO: Standard output for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab94639c-3e70-4999-b338-a635a4807858_isrForCrosstalkSources_855420_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab94639c-3e70-4999-b338-a635a4807858_isrForCrosstalkSources_855420_37.stderr parsl.dataflow.dflow DEBUG: Task 2721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2721 has memoization hash 80da686ebda16c44ef34b0da0cf9c222 parsl.dataflow.memoization INFO: Task 2721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2721 try 0 launched on executor multi with executor id 2722 parsl.dataflow.dflow INFO: Standard output for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/04dab376-e987-4c37-8387-78703602e39a_isrForCrosstalkSources_855390_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/04dab376-e987-4c37-8387-78703602e39a_isrForCrosstalkSources_855390_16.stderr parsl.dataflow.dflow DEBUG: Task 2722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2722 has memoization hash 18604c66e5be25f30c418f0f956fecd4 parsl.dataflow.memoization INFO: Task 2722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2722 try 0 launched on executor multi with executor id 2723 parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Standard output for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/494105fb-73fc-42c8-bcfc-ac62ad07af0c_isrForCrosstalkSources_855430_50.stdout parsl.dataflow.memoization DEBUG: Storing app cache entry 2723b9e70b8c3f853eecae403a0ea926 with result from task 459 parsl.dataflow.dflow INFO: Standard error for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/494105fb-73fc-42c8-bcfc-ac62ad07af0c_isrForCrosstalkSources_855430_50.stderr parsl.dataflow.dflow DEBUG: Task 2723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2723 has memoization hash 7085bc0c80e78759cc6ec65230151e7e parsl.dataflow.memoization INFO: Task 2723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6322de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2723 try 0 launched on executor multi with executor id 2724 parsl.dataflow.dflow INFO: Standard output for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/358da787-9135-4110-8339-c665dddc3cda_isrForCrosstalkSources_855391_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/358da787-9135-4110-8339-c665dddc3cda_isrForCrosstalkSources_855391_41.stderr parsl.dataflow.dflow DEBUG: Task 2724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2724 has memoization hash a366ed1a0fe02ac68668f72aa3f16be4 parsl.dataflow.memoization INFO: Task 2724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/00fadcdd-9f60-4d34-a9d5-b747fe1fa1b3_isrForCrosstalkSources_855433_11.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/00fadcdd-9f60-4d34-a9d5-b747fe1fa1b3_isrForCrosstalkSources_855433_11.stderr parsl.dataflow.dflow INFO: Parsl task 2724 try 0 launched on executor multi with executor id 2725 parsl.dataflow.dflow INFO: Standard output for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ddfe1634-c447-48f3-ad25-fe9b17a6fd9f_isrForCrosstalkSources_855398_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ddfe1634-c447-48f3-ad25-fe9b17a6fd9f_isrForCrosstalkSources_855398_40.stderr parsl.dataflow.dflow DEBUG: Task 2725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2725 has memoization hash 9dbc927d534fa28c752e2a8636ea0ffa parsl.dataflow.memoization INFO: Task 2725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2725 try 0 launched on executor multi with executor id 2726 parsl.dataflow.dflow INFO: Standard output for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00ef9d7c-d192-4b10-b2a3-b7376dec6cee_isrForCrosstalkSources_855390_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00ef9d7c-d192-4b10-b2a3-b7376dec6cee_isrForCrosstalkSources_855390_7.stderr parsl.dataflow.dflow DEBUG: Task 2726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2726 has memoization hash 4b84845d42faf65c13c2a2aa11ad475b parsl.dataflow.memoization INFO: Task 2726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2726 try 0 launched on executor multi with executor id 2727 parsl.dataflow.dflow INFO: Standard output for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4f5adbc2-0fe3-46a1-ad5a-18b16ddc105b_isrForCrosstalkSources_855379_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4f5adbc2-0fe3-46a1-ad5a-18b16ddc105b_isrForCrosstalkSources_855379_46.stderr parsl.dataflow.dflow DEBUG: Task 2727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2727 has memoization hash 14ddebc014a79ca57985f2a724a1f853 parsl.dataflow.memoization INFO: Task 2727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2727 try 0 launched on executor multi with executor id 2728 parsl.dataflow.dflow INFO: Standard output for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ae43a756-0a39-4d8a-8b10-467024235906_isrForCrosstalkSources_855452_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ae43a756-0a39-4d8a-8b10-467024235906_isrForCrosstalkSources_855452_38.stderr parsl.dataflow.dflow DEBUG: Task 2728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2728 has memoization hash 59c48c800abf10709c7fe16de9a2997c parsl.dataflow.memoization INFO: Task 2728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6320860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2728 try 0 launched on executor multi with executor id 2729 parsl.dataflow.dflow INFO: Standard output for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/95a500eb-90da-489a-9c5e-ba71613e808d_isrForCrosstalkSources_855420_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/95a500eb-90da-489a-9c5e-ba71613e808d_isrForCrosstalkSources_855420_9.stderr parsl.dataflow.dflow DEBUG: Task 2729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2729 has memoization hash 68e1a9f0a5955c5a813089198b625258 parsl.dataflow.memoization INFO: Task 2729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2729 try 0 launched on executor multi with executor id 2730 parsl.dataflow.dflow INFO: Standard output for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/07750463-170b-415e-bf48-119061fe3a91_isrForCrosstalkSources_855444_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/07750463-170b-415e-bf48-119061fe3a91_isrForCrosstalkSources_855444_31.stderr parsl.dataflow.dflow DEBUG: Task 2730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2730 has memoization hash 533bce797de5462a19ab607c86589766 parsl.dataflow.memoization INFO: Task 2730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:23 proc_lsst.multi:170 [INFO] [multi] job_id 16964152 proc_lsst.multi INFO: [multi] job_id 16964152 2024-03-09 14:51:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-09 14:51:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964152 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964152 parsl.executors.status_handling DEBUG: Launched block 126 on executor multi with job ID 16964152 parsl.executors.status_handling INFO: Allocated block ID 127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-127 2024-03-09 14:51:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-09 14:51:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2730 try 0 launched on executor multi with executor id 2731 parsl.dataflow.dflow INFO: Standard output for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2863dbf0-c968-4bb2-943e-30c69d4967eb_isrForCrosstalkSources_855453_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2863dbf0-c968-4bb2-943e-30c69d4967eb_isrForCrosstalkSources_855453_32.stderr parsl.dataflow.dflow DEBUG: Task 2731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2731 has memoization hash 55c98c28fd6c3ec1d223220e4f6be971 parsl.dataflow.memoization INFO: Task 2731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6321f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2731 try 0 launched on executor multi with executor id 2732 parsl.dataflow.dflow INFO: Standard output for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4dd33150-7f94-4b7e-9b1b-513567d8e982_isrForCrosstalkSources_855433_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4dd33150-7f94-4b7e-9b1b-513567d8e982_isrForCrosstalkSources_855433_45.stderr parsl.dataflow.dflow DEBUG: Task 2732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2732 has memoization hash 17438a9d0bd3f8a620af573116a5d1fc parsl.dataflow.memoization INFO: Task 2732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6323380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2732 try 0 launched on executor multi with executor id 2733 parsl.dataflow.dflow INFO: Standard output for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/4a8388b3-56ef-4971-85b6-7b3e979379ec_isrForCrosstalkSources_855458_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/4a8388b3-56ef-4971-85b6-7b3e979379ec_isrForCrosstalkSources_855458_14.stderr parsl.dataflow.dflow DEBUG: Task 2733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2733 has memoization hash f92637281284fea7f86fdbd361fddb08 parsl.dataflow.memoization INFO: Task 2733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2733 try 0 launched on executor multi with executor id 2734 parsl.dataflow.dflow INFO: Standard output for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a40d537b-3764-44f7-a85d-6461dece9db3_isrForCrosstalkSources_855386_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a40d537b-3764-44f7-a85d-6461dece9db3_isrForCrosstalkSources_855386_56.stderr parsl.dataflow.dflow DEBUG: Task 2734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2734 has memoization hash 4eccf0845d3fc5971e158010a65e0f54 parsl.dataflow.memoization INFO: Task 2734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2734 try 0 launched on executor multi with executor id 2735 parsl.dataflow.dflow INFO: Standard output for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d50066b5-a96c-4d69-844e-7b4eb707da23_isrForCrosstalkSources_855389_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d50066b5-a96c-4d69-844e-7b4eb707da23_isrForCrosstalkSources_855389_15.stderr parsl.dataflow.dflow DEBUG: Task 2735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2735 has memoization hash 271ba2d0a8d63fa71fe389eca52d47f5 parsl.dataflow.memoization INFO: Task 2735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63900e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2735 try 0 launched on executor multi with executor id 2736 parsl.dataflow.dflow INFO: Standard output for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1fc0a02b-cf46-404e-9968-b23c321ec330_isrForCrosstalkSources_855395_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1fc0a02b-cf46-404e-9968-b23c321ec330_isrForCrosstalkSources_855395_3.stderr parsl.dataflow.dflow DEBUG: Task 2736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2736 has memoization hash fd5a14f9f4335f889128d14e2af05fa2 parsl.dataflow.memoization INFO: Task 2736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2736 try 0 launched on executor multi with executor id 2737 parsl.dataflow.dflow INFO: Standard output for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3d9072e0-83e0-4b4d-b2fe-4e53fac26b31_isrForCrosstalkSources_855388_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3d9072e0-83e0-4b4d-b2fe-4e53fac26b31_isrForCrosstalkSources_855388_22.stderr parsl.dataflow.dflow DEBUG: Task 2737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2737 has memoization hash e61d4cf199ea83f3588d53f48c0cfd4a parsl.dataflow.memoization INFO: Task 2737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2737 try 0 launched on executor multi with executor id 2738 parsl.dataflow.dflow INFO: Standard output for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c556e707-edd8-4ee7-af82-655bd4f8ce3f_isrForCrosstalkSources_855384_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c556e707-edd8-4ee7-af82-655bd4f8ce3f_isrForCrosstalkSources_855384_21.stderr parsl.dataflow.dflow DEBUG: Task 2738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2738 has memoization hash 4a39d3e6bb3c889c4cf0f7e0cc5ebcfe parsl.dataflow.memoization INFO: Task 2738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2738 try 0 launched on executor multi with executor id 2739 parsl.dataflow.dflow INFO: Standard output for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/3310c853-831c-47c5-aa72-258a3120877e_isrForCrosstalkSources_855451_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/3310c853-831c-47c5-aa72-258a3120877e_isrForCrosstalkSources_855451_20.stderr parsl.dataflow.dflow DEBUG: Task 2739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2739 has memoization hash 4f8243406136e0898eecad0dacc15d40 parsl.dataflow.memoization INFO: Task 2739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63939c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2739 try 0 launched on executor multi with executor id 2740 parsl.dataflow.dflow INFO: Standard output for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/0bbeca93-5254-45b5-9bc9-d0ac8cd2980c_isrForCrosstalkSources_855440_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/0bbeca93-5254-45b5-9bc9-d0ac8cd2980c_isrForCrosstalkSources_855440_62.stderr parsl.dataflow.dflow DEBUG: Task 2740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2740 has memoization hash 093c19baf4ad29ab5827465c11e6c434 parsl.dataflow.memoization INFO: Task 2740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63905e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2740 try 0 launched on executor multi with executor id 2741 parsl.dataflow.dflow INFO: Standard output for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/42e2c45e-3ae2-4559-b090-31f306d91ae4_isrForCrosstalkSources_855425_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/42e2c45e-3ae2-4559-b090-31f306d91ae4_isrForCrosstalkSources_855425_32.stderr parsl.dataflow.dflow DEBUG: Task 2741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2741 has memoization hash f52d47d80cc25f8b4dd4af85e76c3893 parsl.dataflow.memoization INFO: Task 2741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2741 try 0 launched on executor multi with executor id 2742 parsl.dataflow.dflow INFO: Standard output for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3dd918a8-7a2b-4e66-bfca-08163dbcd28b_isrForCrosstalkSources_855401_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3dd918a8-7a2b-4e66-bfca-08163dbcd28b_isrForCrosstalkSources_855401_5.stderr parsl.dataflow.dflow DEBUG: Task 2742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2742 has memoization hash 36d76758f9c4621ef8c66962d3aafeb1 parsl.dataflow.memoization INFO: Task 2742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2742 try 0 launched on executor multi with executor id 2743 parsl.dataflow.dflow INFO: Standard output for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/eee5fb6d-ddf8-4eb9-a2c2-cdfec015cfcd_isrForCrosstalkSources_855406_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/eee5fb6d-ddf8-4eb9-a2c2-cdfec015cfcd_isrForCrosstalkSources_855406_6.stderr parsl.dataflow.dflow DEBUG: Task 2743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2743 has memoization hash 6d45c5eb1f1b0e7aa4df9f5d1f089eeb parsl.dataflow.memoization INFO: Task 2743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2743 try 0 launched on executor multi with executor id 2744 parsl.dataflow.dflow INFO: Standard output for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/3fda785f-277d-4ffb-892e-ac4a789445f7_isrForCrosstalkSources_855436_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/3fda785f-277d-4ffb-892e-ac4a789445f7_isrForCrosstalkSources_855436_9.stderr parsl.dataflow.dflow DEBUG: Task 2744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2744 has memoization hash 6fbe04009cc4599898617c3d82139fca parsl.dataflow.memoization INFO: Task 2744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2744 try 0 launched on executor multi with executor id 2745 parsl.dataflow.dflow INFO: Standard output for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/0141c2ec-7375-4b59-a711-6d8a87896b1a_isrForCrosstalkSources_855434_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/0141c2ec-7375-4b59-a711-6d8a87896b1a_isrForCrosstalkSources_855434_41.stderr parsl.dataflow.dflow DEBUG: Task 2745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2745 has memoization hash 410de272dfa866357857aca94c3f3566 parsl.dataflow.memoization INFO: Task 2745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63907c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2745 try 0 launched on executor multi with executor id 2746 parsl.dataflow.dflow INFO: Standard output for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/44358b84-b1e7-447f-af5f-358f0dcda1e2_isrForCrosstalkSources_855382_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/44358b84-b1e7-447f-af5f-358f0dcda1e2_isrForCrosstalkSources_855382_28.stderr parsl.dataflow.dflow DEBUG: Task 2746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2746 has memoization hash 292f758590864fdd2fd9e508ce09abe6 parsl.dataflow.memoization INFO: Task 2746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63909a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2746 try 0 launched on executor multi with executor id 2747 parsl.dataflow.dflow INFO: Standard output for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a1efe05e-0039-476d-b7a5-6da6f1078a75_isrForCrosstalkSources_855441_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a1efe05e-0039-476d-b7a5-6da6f1078a75_isrForCrosstalkSources_855441_36.stderr parsl.dataflow.dflow DEBUG: Task 2747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2747 has memoization hash 1593108cf60d50371157ecea2c4df33a parsl.dataflow.memoization INFO: Task 2747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2747 try 0 launched on executor multi with executor id 2748 parsl.dataflow.dflow INFO: Standard output for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6506b2f4-2629-454c-93c4-15bcd42d1b90_isrForCrosstalkSources_855443_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6506b2f4-2629-454c-93c4-15bcd42d1b90_isrForCrosstalkSources_855443_38.stderr parsl.dataflow.dflow DEBUG: Task 2748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2748 has memoization hash df94f9b4398e76f6b1d33d17f7738265 parsl.dataflow.memoization INFO: Task 2748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2748 try 0 launched on executor multi with executor id 2749 parsl.dataflow.dflow INFO: Standard output for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/16fcdc64-8a36-48df-a04f-cd17f9edff13_isrForCrosstalkSources_855452_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/16fcdc64-8a36-48df-a04f-cd17f9edff13_isrForCrosstalkSources_855452_23.stderr parsl.dataflow.dflow DEBUG: Task 2749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2749 has memoization hash 86a29a97a7da7e1ffe85bc9fbc84d3d5 parsl.dataflow.memoization INFO: Task 2749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2749 try 0 launched on executor multi with executor id 2750 parsl.dataflow.dflow INFO: Standard output for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/3bd70d76-a39c-4618-89df-97391a089542_isrForCrosstalkSources_855422_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/3bd70d76-a39c-4618-89df-97391a089542_isrForCrosstalkSources_855422_15.stderr parsl.dataflow.dflow DEBUG: Task 2750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2750 has memoization hash 2aa185bc5e87e7f78e9a548921b52bd9 parsl.dataflow.memoization INFO: Task 2750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2750 try 0 launched on executor multi with executor id 2751 parsl.dataflow.dflow INFO: Standard output for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/781e1ccc-62d7-45f3-8bf0-7d8bb7701661_isrForCrosstalkSources_855457_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/781e1ccc-62d7-45f3-8bf0-7d8bb7701661_isrForCrosstalkSources_855457_13.stderr parsl.dataflow.dflow DEBUG: Task 2751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2751 has memoization hash 4c31420da076aa9353231a3c50f63284 parsl.dataflow.memoization INFO: Task 2751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2751 try 0 launched on executor multi with executor id 2752 parsl.dataflow.dflow INFO: Standard output for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ac1ad1e5-72b0-48a7-b5b6-e18fdfbd00b5_isrForCrosstalkSources_855414_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ac1ad1e5-72b0-48a7-b5b6-e18fdfbd00b5_isrForCrosstalkSources_855414_40.stderr parsl.dataflow.dflow DEBUG: Task 2752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2752 has memoization hash a6f38090d86d20f1d2cec9cb1d748f05 parsl.dataflow.memoization INFO: Task 2752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2752 try 0 launched on executor multi with executor id 2753 parsl.dataflow.dflow INFO: Standard output for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/30afef6e-6c9b-4892-b24d-b2bf9e34017d_isrForCrosstalkSources_855397_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/30afef6e-6c9b-4892-b24d-b2bf9e34017d_isrForCrosstalkSources_855397_62.stderr parsl.dataflow.dflow DEBUG: Task 2753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2753 has memoization hash 3e7fb0c3aff144c48c554753cba178e5 parsl.dataflow.memoization INFO: Task 2753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f71c5fa7f36fdd08f6cc24564ecf393 with result from task 336 parsl.dataflow.dflow INFO: Parsl task 2753 try 0 launched on executor multi with executor id 2754 parsl.dataflow.dflow INFO: Standard output for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/67000ad8-0f88-4785-8035-321e9ad489d8_isrForCrosstalkSources_855415_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/67000ad8-0f88-4785-8035-321e9ad489d8_isrForCrosstalkSources_855415_53.stderr parsl.dataflow.dflow DEBUG: Task 2754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2754 has memoization hash 8ffed7d6544c78ff259e0d0a54481722 parsl.dataflow.memoization INFO: Task 2754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/6c7e4911-4864-4659-a2ee-c7f0bb47d57e_isrForCrosstalkSources_855435_15.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/6c7e4911-4864-4659-a2ee-c7f0bb47d57e_isrForCrosstalkSources_855435_15.stderr parsl.dataflow.dflow INFO: Parsl task 2754 try 0 launched on executor multi with executor id 2755 parsl.dataflow.dflow INFO: Standard output for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2fa9445c-1cf4-44f1-9bf5-272f31d99e88_isrForCrosstalkSources_855409_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2fa9445c-1cf4-44f1-9bf5-272f31d99e88_isrForCrosstalkSources_855409_35.stderr parsl.dataflow.dflow DEBUG: Task 2755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2755 has memoization hash b1749847f247ffb0f4cc31ff8a8b4d3b parsl.dataflow.memoization INFO: Task 2755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2755 try 0 launched on executor multi with executor id 2756 parsl.dataflow.dflow INFO: Standard output for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f2d4f20-a5af-4d59-8391-09b9be65db49_isrForCrosstalkSources_855456_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f2d4f20-a5af-4d59-8391-09b9be65db49_isrForCrosstalkSources_855456_37.stderr parsl.dataflow.dflow DEBUG: Task 2756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2756 has memoization hash 1afd13ecf3f34a97a2e877d6756348b0 parsl.dataflow.memoization INFO: Task 2756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2756 try 0 launched on executor multi with executor id 2757 parsl.dataflow.dflow INFO: Standard output for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eb70a809-4ee0-4c51-9ac2-51257e766367_isrForCrosstalkSources_855396_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eb70a809-4ee0-4c51-9ac2-51257e766367_isrForCrosstalkSources_855396_15.stderr parsl.dataflow.dflow DEBUG: Task 2757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2757 has memoization hash aab07132edd8a8ba24cb07f792217bc5 parsl.dataflow.memoization INFO: Task 2757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2757 try 0 launched on executor multi with executor id 2758 parsl.dataflow.dflow INFO: Standard output for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ed6c90e-73af-4837-83f2-0919ba93046f_isrForCrosstalkSources_855401_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ed6c90e-73af-4837-83f2-0919ba93046f_isrForCrosstalkSources_855401_47.stderr parsl.dataflow.dflow DEBUG: Task 2758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2758 has memoization hash 9ff235ed492fc1a6eba008d6f09c60cf parsl.dataflow.memoization INFO: Task 2758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63911c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2758 try 0 launched on executor multi with executor id 2759 parsl.dataflow.dflow INFO: Standard output for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3e08101f-ef19-489e-9a74-2ae1c7c2814c_isrForCrosstalkSources_855425_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3e08101f-ef19-489e-9a74-2ae1c7c2814c_isrForCrosstalkSources_855425_61.stderr parsl.dataflow.dflow DEBUG: Task 2759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2759 has memoization hash b3828cef5aefff63b61bea8245fc7eb2 parsl.dataflow.memoization INFO: Task 2759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2759 try 0 launched on executor multi with executor id 2760 parsl.dataflow.dflow INFO: Standard output for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a2cae7f-8afb-41a3-9436-ba274dd2b91d_isrForCrosstalkSources_855442_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a2cae7f-8afb-41a3-9436-ba274dd2b91d_isrForCrosstalkSources_855442_44.stderr parsl.dataflow.dflow DEBUG: Task 2760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2760 has memoization hash faf4645d2fdafc41cb56598a98c41810 parsl.dataflow.memoization INFO: Task 2760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2760 try 0 launched on executor multi with executor id 2761 parsl.dataflow.dflow INFO: Standard output for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f80e0e83-6e58-4101-9258-309f0a7cbe25_isrForCrosstalkSources_855395_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f80e0e83-6e58-4101-9258-309f0a7cbe25_isrForCrosstalkSources_855395_22.stderr parsl.dataflow.dflow DEBUG: Task 2761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2761 has memoization hash b72eac836997b6ba2dae904dc45ce2b7 parsl.dataflow.memoization INFO: Task 2761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2761 try 0 launched on executor multi with executor id 2762 parsl.dataflow.dflow INFO: Standard output for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/68e95841-b8e2-4797-b3a0-877c8a9a099a_isrForCrosstalkSources_855457_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/68e95841-b8e2-4797-b3a0-877c8a9a099a_isrForCrosstalkSources_855457_52.stderr parsl.dataflow.dflow DEBUG: Task 2762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2762 has memoization hash 0f4132900b638b010c2e18e51826877b parsl.dataflow.memoization INFO: Task 2762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2762 try 0 launched on executor multi with executor id 2763 parsl.dataflow.dflow INFO: Standard output for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4a5b9e27-e098-42f0-853f-f808407caedf_isrForCrosstalkSources_855411_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4a5b9e27-e098-42f0-853f-f808407caedf_isrForCrosstalkSources_855411_10.stderr parsl.dataflow.dflow DEBUG: Task 2763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2763 has memoization hash 70603fbe6ba3181871287ca82aab626c parsl.dataflow.memoization INFO: Task 2763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63914e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4956e3da7685a683faacddd19ed49a66 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/38440223-ee29-4cb5-af44-2d0e818c9ec3_isrForCrosstalkSources_855458_4.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/38440223-ee29-4cb5-af44-2d0e818c9ec3_isrForCrosstalkSources_855458_4.stderr 2024-03-09 14:51:24 proc_lsst.multi:170 [INFO] [multi] job_id 16964153 proc_lsst.multi INFO: [multi] job_id 16964153 2024-03-09 14:51:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-09 14:51:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964153 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964153 parsl.executors.status_handling DEBUG: Launched block 127 on executor multi with job ID 16964153 parsl.executors.status_handling INFO: Allocated block ID 128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-128 2024-03-09 14:51:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2763 try 0 launched on executor multi with executor id 2764 parsl.dataflow.dflow INFO: Standard output for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/326775bb-8cb0-4e53-adad-ed2ce200d3cc_isrForCrosstalkSources_855391_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/326775bb-8cb0-4e53-adad-ed2ce200d3cc_isrForCrosstalkSources_855391_43.stderr parsl.dataflow.dflow DEBUG: Task 2764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2764 has memoization hash 0a969c7a8386fc3f3765f9c642ebe25e parsl.dataflow.memoization INFO: Task 2764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2764 try 0 launched on executor multi with executor id 2765 parsl.dataflow.dflow INFO: Standard output for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/943b9dcc-18fe-4097-9c45-0353f350c196_isrForCrosstalkSources_855379_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/943b9dcc-18fe-4097-9c45-0353f350c196_isrForCrosstalkSources_855379_56.stderr parsl.dataflow.dflow DEBUG: Task 2765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2765 has memoization hash 57543f5f195b3711b112f59271aa38ca parsl.dataflow.memoization INFO: Task 2765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2765 try 0 launched on executor multi with executor id 2766 parsl.dataflow.dflow INFO: Standard output for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/86ebcf46-0d78-47e0-88fc-ca12ba2d8755_isrForCrosstalkSources_855449_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/86ebcf46-0d78-47e0-88fc-ca12ba2d8755_isrForCrosstalkSources_855449_1.stderr parsl.dataflow.dflow DEBUG: Task 2766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2766 has memoization hash 0dcc6f15a4e73ea93b47066f7ab9da5b parsl.dataflow.memoization INFO: Task 2766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2766 try 0 launched on executor multi with executor id 2767 parsl.dataflow.dflow INFO: Standard output for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/dce3c548-412a-45ac-aac2-5187a75e18f4_isrForCrosstalkSources_855439_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/dce3c548-412a-45ac-aac2-5187a75e18f4_isrForCrosstalkSources_855439_26.stderr parsl.dataflow.dflow DEBUG: Task 2767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2767 has memoization hash f959c0f3e41659a83a70a1f9045e2c2a parsl.dataflow.memoization INFO: Task 2767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63904a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffae716217ed1abfaa9b25572ac8591d with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b26adb54-6ff1-4062-b784-ef485a2c8671_isrForCrosstalkSources_855416_60.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b26adb54-6ff1-4062-b784-ef485a2c8671_isrForCrosstalkSources_855416_60.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65d40061bc617011dd77b76b060bf6d1 with result from task 283 parsl.dataflow.dflow INFO: Parsl task 2767 try 0 launched on executor multi with executor id 2768 parsl.dataflow.dflow INFO: Standard output for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/24080bc9-b19d-4df3-8a45-b6e073c956ed_isrForCrosstalkSources_855446_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/24080bc9-b19d-4df3-8a45-b6e073c956ed_isrForCrosstalkSources_855446_9.stderr parsl.dataflow.dflow DEBUG: Task 2768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2768 has memoization hash 388997b10e11dadceb7a06eefa9151e5 parsl.dataflow.memoization INFO: Task 2768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/698f92cb-cfc1-406a-bbf6-dbcebe9283c6_isrForCrosstalkSources_855445_37.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/698f92cb-cfc1-406a-bbf6-dbcebe9283c6_isrForCrosstalkSources_855445_37.stderr parsl.dataflow.dflow INFO: Parsl task 2768 try 0 launched on executor multi with executor id 2769 parsl.dataflow.dflow INFO: Standard output for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/92ff179d-a04c-43a5-802d-d777c1cba94c_isrForCrosstalkSources_855402_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/92ff179d-a04c-43a5-802d-d777c1cba94c_isrForCrosstalkSources_855402_45.stderr parsl.dataflow.dflow DEBUG: Task 2769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2769 has memoization hash 6e629c09d2737ef7c0016349798555b5 parsl.dataflow.memoization INFO: Task 2769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2769 try 0 launched on executor multi with executor id 2770 parsl.dataflow.dflow INFO: Standard output for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/25ba655c-d59c-4829-8895-895221c08550_isrForCrosstalkSources_855435_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/25ba655c-d59c-4829-8895-895221c08550_isrForCrosstalkSources_855435_36.stderr parsl.dataflow.dflow DEBUG: Task 2770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2770 has memoization hash 379bc75b463905db08e7fa95c1fedac0 parsl.dataflow.memoization INFO: Task 2770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2770 try 0 launched on executor multi with executor id 2771 parsl.dataflow.dflow INFO: Standard output for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3b8dcfcd-b378-4c27-b462-93952187e55c_isrForCrosstalkSources_855423_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3b8dcfcd-b378-4c27-b462-93952187e55c_isrForCrosstalkSources_855423_47.stderr parsl.dataflow.dflow DEBUG: Task 2771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2771 has memoization hash f206fa4c2c8b268a961aa29a2ad0da81 parsl.dataflow.memoization INFO: Task 2771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2771 try 0 launched on executor multi with executor id 2772 parsl.dataflow.dflow INFO: Standard output for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/398417d8-983c-4572-8812-0fddbc9a9b11_isrForCrosstalkSources_855454_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/398417d8-983c-4572-8812-0fddbc9a9b11_isrForCrosstalkSources_855454_10.stderr parsl.dataflow.dflow DEBUG: Task 2772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2772 has memoization hash 7a238c164c41fecd1408c57ee25f9353 parsl.dataflow.memoization INFO: Task 2772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2772 try 0 launched on executor multi with executor id 2773 parsl.dataflow.dflow INFO: Standard output for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/92b38801-16fd-4dcc-b238-2a6647f0d82a_isrForCrosstalkSources_855388_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/92b38801-16fd-4dcc-b238-2a6647f0d82a_isrForCrosstalkSources_855388_2.stderr parsl.dataflow.dflow DEBUG: Task 2773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2773 has memoization hash a0fd8b826fa3b54be6e998cb80b95b43 parsl.dataflow.memoization INFO: Task 2773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2773 try 0 launched on executor multi with executor id 2774 parsl.dataflow.dflow INFO: Standard output for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cb18bdc9-bb6d-49da-b71b-cbce772577b3_isrForCrosstalkSources_855382_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cb18bdc9-bb6d-49da-b71b-cbce772577b3_isrForCrosstalkSources_855382_9.stderr parsl.dataflow.dflow DEBUG: Task 2774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2774 has memoization hash 41ee7477d7a31979dcd8c0b9b6ea8148 parsl.dataflow.memoization INFO: Task 2774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63919e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2774 try 0 launched on executor multi with executor id 2775 parsl.dataflow.dflow INFO: Standard output for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b659be9-f8f7-4ad2-b269-c8fc3dfef01a_isrForCrosstalkSources_855406_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b659be9-f8f7-4ad2-b269-c8fc3dfef01a_isrForCrosstalkSources_855406_60.stderr parsl.dataflow.dflow DEBUG: Task 2775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2775 has memoization hash 61dc9a6bc64816fc05802ab7c9cc1b3e parsl.dataflow.memoization INFO: Task 2775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2775 try 0 launched on executor multi with executor id 2776 parsl.dataflow.dflow INFO: Standard output for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/192380fe-a315-450f-a0dd-fe5465e69e42_isrForCrosstalkSources_855416_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/192380fe-a315-450f-a0dd-fe5465e69e42_isrForCrosstalkSources_855416_23.stderr parsl.dataflow.dflow DEBUG: Task 2776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2776 has memoization hash 112932699dac0823ba07bff21793c0cb parsl.dataflow.memoization INFO: Task 2776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2776 try 0 launched on executor multi with executor id 2777 parsl.dataflow.dflow INFO: Standard output for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/86a6564a-1e32-4f00-80bc-6453d4654ea0_isrForCrosstalkSources_855390_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/86a6564a-1e32-4f00-80bc-6453d4654ea0_isrForCrosstalkSources_855390_2.stderr parsl.dataflow.dflow DEBUG: Task 2777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2777 has memoization hash 52534df7d3eb6df02d717c8dfb6dbe21 parsl.dataflow.memoization INFO: Task 2777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:24 proc_lsst.multi:170 [INFO] [multi] job_id 16964154 proc_lsst.multi INFO: [multi] job_id 16964154 2024-03-09 14:51:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-09 14:51:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964154 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964154 parsl.executors.status_handling DEBUG: Launched block 128 on executor multi with job ID 16964154 parsl.executors.status_handling INFO: Allocated block ID 129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-129 2024-03-09 14:51:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-09 14:51:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2777 try 0 launched on executor multi with executor id 2778 parsl.dataflow.dflow INFO: Standard output for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0588ecfa-0a9e-4eeb-83de-a4788059cf6e_isrForCrosstalkSources_855424_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0588ecfa-0a9e-4eeb-83de-a4788059cf6e_isrForCrosstalkSources_855424_61.stderr parsl.dataflow.dflow DEBUG: Task 2778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2778 has memoization hash 87e71021e298144ddd331031d4556ec6 parsl.dataflow.memoization INFO: Task 2778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2778 try 0 launched on executor multi with executor id 2779 parsl.dataflow.dflow INFO: Standard output for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f435e517-22f4-4c02-a2c0-c4f95f0ee67c_isrForCrosstalkSources_855390_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f435e517-22f4-4c02-a2c0-c4f95f0ee67c_isrForCrosstalkSources_855390_4.stderr parsl.dataflow.dflow DEBUG: Task 2779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2779 has memoization hash 478d47f274ebc9e854be2054d6980374 parsl.dataflow.memoization INFO: Task 2779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2779 try 0 launched on executor multi with executor id 2780 parsl.dataflow.dflow INFO: Standard output for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a235ac7e-0c43-4317-a518-d36a46172dee_isrForCrosstalkSources_855382_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a235ac7e-0c43-4317-a518-d36a46172dee_isrForCrosstalkSources_855382_3.stderr parsl.dataflow.dflow DEBUG: Task 2780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2780 has memoization hash 654775196b69d205475aa2c176651b27 parsl.dataflow.memoization INFO: Task 2780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2780 try 0 launched on executor multi with executor id 2781 parsl.dataflow.dflow INFO: Standard output for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/95d75a48-292d-4be9-b16e-448c5f88e768_isrForCrosstalkSources_855383_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/95d75a48-292d-4be9-b16e-448c5f88e768_isrForCrosstalkSources_855383_32.stderr parsl.dataflow.dflow DEBUG: Task 2781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2781 has memoization hash ac477c52bc87ff672eac40ad67a89a65 parsl.dataflow.memoization INFO: Task 2781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2781 try 0 launched on executor multi with executor id 2782 parsl.dataflow.dflow INFO: Standard output for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/210b038e-d4ee-4c96-8d79-17dbd25c621f_isrForCrosstalkSources_855387_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/210b038e-d4ee-4c96-8d79-17dbd25c621f_isrForCrosstalkSources_855387_19.stderr parsl.dataflow.dflow DEBUG: Task 2782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2782 has memoization hash b7ad13e8aab2cb750dc6b0ed05f7e0ea parsl.dataflow.memoization INFO: Task 2782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2782 try 0 launched on executor multi with executor id 2783 parsl.dataflow.dflow INFO: Standard output for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e9d614c7-0cc8-4858-8828-cf3374d103e1_isrForCrosstalkSources_855400_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e9d614c7-0cc8-4858-8828-cf3374d103e1_isrForCrosstalkSources_855400_50.stderr parsl.dataflow.dflow DEBUG: Task 2783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2783 has memoization hash 8e6200bd99d664d9b820ad7d6e5b7ff6 parsl.dataflow.memoization INFO: Task 2783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2783 try 0 launched on executor multi with executor id 2784 parsl.dataflow.dflow INFO: Standard output for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24c2bb7c-1931-4096-ae6b-0ddf818dc22b_isrForCrosstalkSources_855444_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24c2bb7c-1931-4096-ae6b-0ddf818dc22b_isrForCrosstalkSources_855444_11.stderr parsl.dataflow.dflow DEBUG: Task 2784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2784 has memoization hash 1db0d841389af03fe5ea136b4a942c0c parsl.dataflow.memoization INFO: Task 2784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2784 try 0 launched on executor multi with executor id 2785 parsl.dataflow.dflow INFO: Standard output for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e78ea31-b1bd-4e21-9efa-2a46a6ec55e7_isrForCrosstalkSources_855412_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e78ea31-b1bd-4e21-9efa-2a46a6ec55e7_isrForCrosstalkSources_855412_18.stderr parsl.dataflow.dflow DEBUG: Task 2785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2785 has memoization hash 820b54d098cd4f7b5d685ad0ed3d7791 parsl.dataflow.memoization INFO: Task 2785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63920c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2785 try 0 launched on executor multi with executor id 2786 parsl.dataflow.dflow INFO: Standard output for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/923d5cfd-e281-4e40-87db-01cb4f7fd4d0_isrForCrosstalkSources_855417_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/923d5cfd-e281-4e40-87db-01cb4f7fd4d0_isrForCrosstalkSources_855417_39.stderr parsl.dataflow.dflow DEBUG: Task 2786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2786 has memoization hash 81cd4befef90416aa876e42a06cca8dd parsl.dataflow.memoization INFO: Task 2786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63923e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2786 try 0 launched on executor multi with executor id 2787 parsl.dataflow.dflow INFO: Standard output for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b8d879de-f1e5-4e38-a58a-38de10bad514_isrForCrosstalkSources_855389_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b8d879de-f1e5-4e38-a58a-38de10bad514_isrForCrosstalkSources_855389_27.stderr parsl.dataflow.dflow DEBUG: Task 2787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2787 has memoization hash 1e9510228b68fa951e89b71c34e71d07 parsl.dataflow.memoization INFO: Task 2787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2787 try 0 launched on executor multi with executor id 2788 parsl.dataflow.dflow INFO: Standard output for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f2fa49f3-f380-40d4-a3d3-450b0455c027_isrForCrosstalkSources_855423_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f2fa49f3-f380-40d4-a3d3-450b0455c027_isrForCrosstalkSources_855423_58.stderr parsl.dataflow.dflow DEBUG: Task 2788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2788 has memoization hash d91bcd32b1daadd0f1f8276a1042f80c parsl.dataflow.memoization INFO: Task 2788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2788 try 0 launched on executor multi with executor id 2789 parsl.dataflow.dflow INFO: Standard output for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5d22c461-cee8-4ae6-874c-2948b0ceefc1_isrForCrosstalkSources_855433_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5d22c461-cee8-4ae6-874c-2948b0ceefc1_isrForCrosstalkSources_855433_56.stderr parsl.dataflow.dflow DEBUG: Task 2789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2789 has memoization hash c532a9b379d67928e3133c9598776d70 parsl.dataflow.memoization INFO: Task 2789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2789 try 0 launched on executor multi with executor id 2790 parsl.dataflow.dflow INFO: Standard output for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8466a2f4-fd86-472e-a019-850a689d0d84_isrForCrosstalkSources_855393_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8466a2f4-fd86-472e-a019-850a689d0d84_isrForCrosstalkSources_855393_59.stderr parsl.dataflow.dflow DEBUG: Task 2790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2790 has memoization hash e429050cbd5734ab2fe72ba3308636cd parsl.dataflow.memoization INFO: Task 2790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:25 proc_lsst.multi:170 [INFO] [multi] job_id 16964155 proc_lsst.multi INFO: [multi] job_id 16964155 2024-03-09 14:51:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-09 14:51:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964155 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964155 parsl.executors.status_handling DEBUG: Launched block 129 on executor multi with job ID 16964155 parsl.executors.status_handling INFO: Allocated block ID 130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-130 2024-03-09 14:51:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2790 try 0 launched on executor multi with executor id 2791 parsl.dataflow.dflow INFO: Standard output for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a00629d0-6432-49d8-908d-d09f4dde7837_isrForCrosstalkSources_855431_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a00629d0-6432-49d8-908d-d09f4dde7837_isrForCrosstalkSources_855431_32.stderr parsl.dataflow.dflow DEBUG: Task 2791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2791 has memoization hash 33edb3cc3e1752beab021e9018fb5deb parsl.dataflow.memoization INFO: Task 2791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2791 try 0 launched on executor multi with executor id 2792 parsl.dataflow.dflow INFO: Standard output for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7729f04b-dff2-43f1-9a59-18047668277b_isrForCrosstalkSources_855456_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7729f04b-dff2-43f1-9a59-18047668277b_isrForCrosstalkSources_855456_58.stderr parsl.dataflow.dflow DEBUG: Task 2792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2792 has memoization hash 61aef485714da8bf36af2dcd5589a30c parsl.dataflow.memoization INFO: Task 2792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2792 try 0 launched on executor multi with executor id 2793 parsl.dataflow.dflow INFO: Standard output for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/459e7dbf-4966-405a-86ba-3a37cc874444_isrForCrosstalkSources_855408_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/459e7dbf-4966-405a-86ba-3a37cc874444_isrForCrosstalkSources_855408_50.stderr parsl.dataflow.dflow DEBUG: Task 2793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2793 has memoization hash 195c6f3a9287088ec57641209f7600b9 parsl.dataflow.memoization INFO: Task 2793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2793 try 0 launched on executor multi with executor id 2794 parsl.dataflow.dflow INFO: Standard output for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1979cfec-9135-4b23-b597-2d1977ddee08_isrForCrosstalkSources_855417_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1979cfec-9135-4b23-b597-2d1977ddee08_isrForCrosstalkSources_855417_29.stderr parsl.dataflow.dflow DEBUG: Task 2794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2794 has memoization hash 86dcd28e7ff137035d7daf1032c27f7a parsl.dataflow.memoization INFO: Task 2794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2794 try 0 launched on executor multi with executor id 2795 parsl.dataflow.dflow INFO: Standard output for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/903add41-4790-4d6c-8f80-b31fbe6b5bb3_isrForCrosstalkSources_855431_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/903add41-4790-4d6c-8f80-b31fbe6b5bb3_isrForCrosstalkSources_855431_19.stderr parsl.dataflow.dflow DEBUG: Task 2795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2795 has memoization hash 4415fd1cf9180f62d1c7c05b5ea6061c parsl.dataflow.memoization INFO: Task 2795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2795 try 0 launched on executor multi with executor id 2796 parsl.dataflow.dflow INFO: Standard output for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/71951d8c-1640-49ec-b7d3-a9353ad8f7c6_isrForCrosstalkSources_855454_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/71951d8c-1640-49ec-b7d3-a9353ad8f7c6_isrForCrosstalkSources_855454_37.stderr parsl.dataflow.dflow DEBUG: Task 2796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2796 has memoization hash 359066d850f9575b2fdf10fabf9115e7 parsl.dataflow.memoization INFO: Task 2796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63936a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2796 try 0 launched on executor multi with executor id 2797 parsl.dataflow.dflow INFO: Standard output for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9f3085e7-574b-4e0a-bf5d-739cde62d6d0_isrForCrosstalkSources_855448_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9f3085e7-574b-4e0a-bf5d-739cde62d6d0_isrForCrosstalkSources_855448_43.stderr parsl.dataflow.dflow DEBUG: Task 2797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2797 has memoization hash 8f8be45f62b6ad45843d4fa6587a9c52 parsl.dataflow.memoization INFO: Task 2797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63937e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2797 try 0 launched on executor multi with executor id 2798 parsl.dataflow.dflow INFO: Standard output for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/eebbf966-ea4f-4931-b53e-da4b0c43fd8a_isrForCrosstalkSources_855410_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/eebbf966-ea4f-4931-b53e-da4b0c43fd8a_isrForCrosstalkSources_855410_34.stderr parsl.dataflow.dflow DEBUG: Task 2798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2798 has memoization hash 62af3486c32a4ae1c724be283b4518d8 parsl.dataflow.memoization INFO: Task 2798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2798 try 0 launched on executor multi with executor id 2799 parsl.dataflow.dflow INFO: Standard output for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/fb268a1a-9711-4d5e-bb2c-fe5002c2dc27_isrForCrosstalkSources_855426_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/fb268a1a-9711-4d5e-bb2c-fe5002c2dc27_isrForCrosstalkSources_855426_26.stderr parsl.dataflow.dflow DEBUG: Task 2799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2799 has memoization hash 751ff1a0f841ca4ce0387253c2be116b parsl.dataflow.memoization INFO: Task 2799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63934c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2799 try 0 launched on executor multi with executor id 2800 parsl.dataflow.dflow INFO: Standard output for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1d0addc1-3cb0-43d7-8a59-a5e8f29b289c_isrForCrosstalkSources_855384_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1d0addc1-3cb0-43d7-8a59-a5e8f29b289c_isrForCrosstalkSources_855384_42.stderr parsl.dataflow.dflow DEBUG: Task 2800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2800 has memoization hash 7adeccca22b0ed2af015b1fe29aa7a96 parsl.dataflow.memoization INFO: Task 2800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2800 try 0 launched on executor multi with executor id 2801 parsl.dataflow.dflow INFO: Standard output for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cfc1b682-f5a2-4cee-8c8e-1f9079e85fc0_isrForCrosstalkSources_855422_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cfc1b682-f5a2-4cee-8c8e-1f9079e85fc0_isrForCrosstalkSources_855422_48.stderr parsl.dataflow.dflow DEBUG: Task 2801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2801 has memoization hash 5d96041f0d993e2eab0cf5c678f36c92 parsl.dataflow.memoization INFO: Task 2801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2801 try 0 launched on executor multi with executor id 2802 parsl.dataflow.dflow INFO: Standard output for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1c165b9c-962d-4c8d-9137-ff317465e2f7_isrForCrosstalkSources_855401_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1c165b9c-962d-4c8d-9137-ff317465e2f7_isrForCrosstalkSources_855401_61.stderr parsl.dataflow.dflow DEBUG: Task 2802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2802 has memoization hash 9ae93aa02133e3e76f31836879af3a0b parsl.dataflow.memoization INFO: Task 2802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2802 try 0 launched on executor multi with executor id 2803 parsl.dataflow.dflow INFO: Standard output for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18330a6e-976a-4c76-b5b3-b78a5e2de3bc_isrForCrosstalkSources_855423_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18330a6e-976a-4c76-b5b3-b78a5e2de3bc_isrForCrosstalkSources_855423_49.stderr parsl.dataflow.dflow DEBUG: Task 2803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2803 has memoization hash 33bbbce85dbe21bf8a536b5bba62381d parsl.dataflow.memoization INFO: Task 2803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63931a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2803 try 0 launched on executor multi with executor id 2804 parsl.dataflow.dflow INFO: Standard output for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cc51daa7-0ffd-40fa-ba55-fccaca423f7b_isrForCrosstalkSources_855446_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cc51daa7-0ffd-40fa-ba55-fccaca423f7b_isrForCrosstalkSources_855446_43.stderr parsl.dataflow.dflow DEBUG: Task 2804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2804 has memoization hash 2cfef73152e98236c0b4987459b475c5 parsl.dataflow.memoization INFO: Task 2804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2804 try 0 launched on executor multi with executor id 2805 parsl.dataflow.dflow INFO: Standard output for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/37a3e70f-bf4a-4456-bbf6-516065f3a4c2_isrForCrosstalkSources_855387_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/37a3e70f-bf4a-4456-bbf6-516065f3a4c2_isrForCrosstalkSources_855387_26.stderr parsl.dataflow.dflow DEBUG: Task 2805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2805 has memoization hash ee92425b7ee2a342fe7c8ec49c70b13d parsl.dataflow.memoization INFO: Task 2805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2805 try 0 launched on executor multi with executor id 2806 parsl.dataflow.dflow INFO: Standard output for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/37a635bb-86d7-449e-8ffa-04523940a286_isrForCrosstalkSources_855413_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/37a635bb-86d7-449e-8ffa-04523940a286_isrForCrosstalkSources_855413_48.stderr parsl.dataflow.dflow DEBUG: Task 2806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2806 has memoization hash 640dd16bd6c6289bd335f0d846e590b5 parsl.dataflow.memoization INFO: Task 2806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2806 try 0 launched on executor multi with executor id 2807 parsl.dataflow.dflow INFO: Standard output for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a07f0796-bc6d-4662-8b67-1a7c795c144e_isrForCrosstalkSources_855450_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a07f0796-bc6d-4662-8b67-1a7c795c144e_isrForCrosstalkSources_855450_18.stderr parsl.dataflow.dflow DEBUG: Task 2807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2807 has memoization hash be477d5e64cea48e8f1d78a2fd23e3c8 parsl.dataflow.memoization INFO: Task 2807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2807 try 0 launched on executor multi with executor id 2808 parsl.dataflow.dflow INFO: Standard output for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/20674411-c058-4567-8c53-a4ea33ad1c50_isrForCrosstalkSources_855416_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/20674411-c058-4567-8c53-a4ea33ad1c50_isrForCrosstalkSources_855416_22.stderr parsl.dataflow.dflow DEBUG: Task 2808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2808 has memoization hash 7036544b23551901d5cfb583665fd598 parsl.dataflow.memoization INFO: Task 2808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2808 try 0 launched on executor multi with executor id 2809 parsl.dataflow.dflow INFO: Standard output for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/10e859ba-dd62-4c79-9d2a-537b860063c3_isrForCrosstalkSources_855389_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/10e859ba-dd62-4c79-9d2a-537b860063c3_isrForCrosstalkSources_855389_57.stderr parsl.dataflow.dflow DEBUG: Task 2809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2809 has memoization hash 3e30b6dbbdc3fbc85d3b9c221e93da43 parsl.dataflow.memoization INFO: Task 2809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2809 try 0 launched on executor multi with executor id 2810 parsl.dataflow.dflow INFO: Standard output for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f932ec1a-1da7-4563-b7d7-148789509c3b_isrForCrosstalkSources_855426_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f932ec1a-1da7-4563-b7d7-148789509c3b_isrForCrosstalkSources_855426_15.stderr parsl.dataflow.dflow DEBUG: Task 2810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2810 has memoization hash d16e6a95bf415f9aa843513f877d223c parsl.dataflow.memoization INFO: Task 2810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2810 try 0 launched on executor multi with executor id 2811 parsl.dataflow.dflow INFO: Standard output for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54de433c-1777-4f66-8007-2c667fcc4c50_isrForCrosstalkSources_855382_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54de433c-1777-4f66-8007-2c667fcc4c50_isrForCrosstalkSources_855382_33.stderr parsl.dataflow.dflow DEBUG: Task 2811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2811 has memoization hash 06684c92afd1092152cdfa26666c51f7 parsl.dataflow.memoization INFO: Task 2811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2811 try 0 launched on executor multi with executor id 2812 parsl.dataflow.dflow INFO: Standard output for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a840a83d-29a9-424f-a23a-72205b11b4ff_isrForCrosstalkSources_855444_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a840a83d-29a9-424f-a23a-72205b11b4ff_isrForCrosstalkSources_855444_58.stderr parsl.dataflow.dflow DEBUG: Task 2812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2812 has memoization hash 2f97eb982190a67ff7cd03c754403dfe parsl.dataflow.memoization INFO: Task 2812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2812 try 0 launched on executor multi with executor id 2813 parsl.dataflow.dflow INFO: Standard output for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6fc45cd7-3f2d-48a3-9cce-ccb643d68279_isrForCrosstalkSources_855415_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6fc45cd7-3f2d-48a3-9cce-ccb643d68279_isrForCrosstalkSources_855415_60.stderr parsl.dataflow.dflow DEBUG: Task 2813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2813 has memoization hash 5ea7cdce98f0b9978bc72e5df33c0711 parsl.dataflow.memoization INFO: Task 2813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2813 try 0 launched on executor multi with executor id 2814 parsl.dataflow.dflow INFO: Standard output for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4c953f22-dbfb-49ca-ae9a-1515781ef94a_isrForCrosstalkSources_855392_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4c953f22-dbfb-49ca-ae9a-1515781ef94a_isrForCrosstalkSources_855392_11.stderr parsl.dataflow.dflow DEBUG: Task 2814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2814 has memoization hash 7e0cb0abe3f393f456f5f392f9cc5cbc parsl.dataflow.memoization INFO: Task 2814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2814 try 0 launched on executor multi with executor id 2815 parsl.dataflow.dflow INFO: Standard output for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f56f7bd0-6536-4917-be64-4f8657927ebd_isrForCrosstalkSources_855452_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f56f7bd0-6536-4917-be64-4f8657927ebd_isrForCrosstalkSources_855452_34.stderr parsl.dataflow.dflow DEBUG: Task 2815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2815 has memoization hash 638cea57068d4a1ab7ed65b7391021b6 parsl.dataflow.memoization INFO: Task 2815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2815 try 0 launched on executor multi with executor id 2816 parsl.dataflow.dflow INFO: Standard output for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f4ee715c-12e4-4b48-9fce-8714659aed37_isrForCrosstalkSources_855444_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f4ee715c-12e4-4b48-9fce-8714659aed37_isrForCrosstalkSources_855444_37.stderr parsl.dataflow.dflow DEBUG: Task 2816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2816 has memoization hash 610df0c427627dc004f70e09c9ed0f70 parsl.dataflow.memoization INFO: Task 2816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63927a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2816 try 0 launched on executor multi with executor id 2817 parsl.dataflow.dflow INFO: Standard output for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f6b8b7d5-26ea-4c7f-8a05-0572b3d46381_isrForCrosstalkSources_855379_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f6b8b7d5-26ea-4c7f-8a05-0572b3d46381_isrForCrosstalkSources_855379_25.stderr parsl.dataflow.dflow DEBUG: Task 2817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2817 has memoization hash 4bc3f24d4a21c5ad377d1353fd58a15a parsl.dataflow.memoization INFO: Task 2817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2817 try 0 launched on executor multi with executor id 2818 parsl.dataflow.dflow INFO: Standard output for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/062a5d7f-d415-4eb7-94f6-40e68d5f5202_isrForCrosstalkSources_855414_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/062a5d7f-d415-4eb7-94f6-40e68d5f5202_isrForCrosstalkSources_855414_50.stderr parsl.dataflow.dflow DEBUG: Task 2818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2818 has memoization hash ea6eab2aef734387304ad25916df89a1 parsl.dataflow.memoization INFO: Task 2818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:25 proc_lsst.multi:170 [INFO] [multi] job_id 16964156 proc_lsst.multi INFO: [multi] job_id 16964156 2024-03-09 14:51:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-09 14:51:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964156 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964156 parsl.executors.status_handling DEBUG: Launched block 130 on executor multi with job ID 16964156 parsl.executors.status_handling INFO: Allocated block ID 131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-131 2024-03-09 14:51:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-09 14:51:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2818 try 0 launched on executor multi with executor id 2819 parsl.dataflow.dflow INFO: Standard output for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c2403b27-0c1d-4dd1-a7a9-878c6cff5903_isrForCrosstalkSources_855388_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c2403b27-0c1d-4dd1-a7a9-878c6cff5903_isrForCrosstalkSources_855388_1.stderr parsl.dataflow.dflow DEBUG: Task 2819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2819 has memoization hash 99ba37d3d8a13ad4491f4f5913d0b9ba parsl.dataflow.memoization INFO: Task 2819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2819 try 0 launched on executor multi with executor id 2820 parsl.dataflow.dflow INFO: Standard output for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4def6a6e-7775-4aa9-abcf-f5f76b50726a_isrForCrosstalkSources_855384_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4def6a6e-7775-4aa9-abcf-f5f76b50726a_isrForCrosstalkSources_855384_6.stderr parsl.dataflow.dflow DEBUG: Task 2820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2820 has memoization hash 200264c9c8b558f394ac5c74daa73ada parsl.dataflow.memoization INFO: Task 2820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2820 try 0 launched on executor multi with executor id 2821 parsl.dataflow.dflow INFO: Standard output for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b102af4f-00f6-42cb-9134-75df1f7fb151_isrForCrosstalkSources_855448_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b102af4f-00f6-42cb-9134-75df1f7fb151_isrForCrosstalkSources_855448_51.stderr parsl.dataflow.dflow DEBUG: Task 2821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2821 has memoization hash f24c9bf436a0b24f02e86582d0433073 parsl.dataflow.memoization INFO: Task 2821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2821 try 0 launched on executor multi with executor id 2822 parsl.dataflow.dflow INFO: Standard output for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f6ab3741-dd04-48dd-a104-7c6654b480b1_isrForCrosstalkSources_855426_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f6ab3741-dd04-48dd-a104-7c6654b480b1_isrForCrosstalkSources_855426_36.stderr parsl.dataflow.dflow DEBUG: Task 2822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2822 has memoization hash af545b58de9a0072897080e3df0b5720 parsl.dataflow.memoization INFO: Task 2822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2822 try 0 launched on executor multi with executor id 2823 parsl.dataflow.dflow INFO: Standard output for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/960c354f-80c8-4f13-b3c0-3160f3f24f1b_isrForCrosstalkSources_855418_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/960c354f-80c8-4f13-b3c0-3160f3f24f1b_isrForCrosstalkSources_855418_39.stderr parsl.dataflow.dflow DEBUG: Task 2823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2823 has memoization hash 5edd9d83973567bc843b06833ae38506 parsl.dataflow.memoization INFO: Task 2823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2823 try 0 launched on executor multi with executor id 2824 parsl.dataflow.dflow INFO: Standard output for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/349614ff-a290-4379-8cbc-f8d4f49fac06_isrForCrosstalkSources_855447_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/349614ff-a290-4379-8cbc-f8d4f49fac06_isrForCrosstalkSources_855447_51.stderr parsl.dataflow.dflow DEBUG: Task 2824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2824 has memoization hash e43fcb2a439d80d078bddd4bcfb036da parsl.dataflow.memoization INFO: Task 2824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2824 try 0 launched on executor multi with executor id 2825 parsl.dataflow.dflow INFO: Standard output for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ed6cc0ef-552f-4828-98fc-fd9c5f800667_isrForCrosstalkSources_855415_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ed6cc0ef-552f-4828-98fc-fd9c5f800667_isrForCrosstalkSources_855415_45.stderr parsl.dataflow.dflow DEBUG: Task 2825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2825 has memoization hash 2a112f1e67940b2128039877207e0934 parsl.dataflow.memoization INFO: Task 2825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2825 try 0 launched on executor multi with executor id 2826 parsl.dataflow.dflow INFO: Standard output for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/694edef1-f8ca-43ad-8143-a4ed1dba2a65_isrForCrosstalkSources_855401_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/694edef1-f8ca-43ad-8143-a4ed1dba2a65_isrForCrosstalkSources_855401_30.stderr parsl.dataflow.dflow DEBUG: Task 2826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2826 has memoization hash a45e55cfcffb23441f62af3865f6564a parsl.dataflow.memoization INFO: Task 2826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63902c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2826 try 0 launched on executor multi with executor id 2827 parsl.dataflow.dflow INFO: Standard output for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0081ccb0-10e4-4d0b-a892-52c8978f5f91_isrForCrosstalkSources_855428_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0081ccb0-10e4-4d0b-a892-52c8978f5f91_isrForCrosstalkSources_855428_25.stderr parsl.dataflow.dflow DEBUG: Task 2827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2827 has memoization hash 2af95c8399fc72c71fd9f5b36e97b9ea parsl.dataflow.memoization INFO: Task 2827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2827 try 0 launched on executor multi with executor id 2828 parsl.dataflow.dflow INFO: Standard output for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d0d900a8-0177-40fa-a06e-36310ad93fe5_isrForCrosstalkSources_855422_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d0d900a8-0177-40fa-a06e-36310ad93fe5_isrForCrosstalkSources_855422_19.stderr parsl.dataflow.dflow DEBUG: Task 2828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2828 has memoization hash d576bdaeefdebcc0555a4ce488030e72 parsl.dataflow.memoization INFO: Task 2828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2828 try 0 launched on executor multi with executor id 2829 parsl.dataflow.dflow INFO: Standard output for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78a61bd8-0633-43ca-abc1-6c7264b12980_isrForCrosstalkSources_855387_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78a61bd8-0633-43ca-abc1-6c7264b12980_isrForCrosstalkSources_855387_24.stderr parsl.dataflow.dflow DEBUG: Task 2829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2829 has memoization hash c5e224ad893a2a15816adeba97e4df9f parsl.dataflow.memoization INFO: Task 2829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6390680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2829 try 0 launched on executor multi with executor id 2830 parsl.dataflow.dflow INFO: Standard output for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2017614-add4-42cf-b273-2402dc5bb13e_isrForCrosstalkSources_855382_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2017614-add4-42cf-b273-2402dc5bb13e_isrForCrosstalkSources_855382_54.stderr parsl.dataflow.dflow DEBUG: Task 2830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2830 has memoization hash ddf97cdde4595ed2d5739cf130b3e338 parsl.dataflow.memoization INFO: Task 2830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6391f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2830 try 0 launched on executor multi with executor id 2831 parsl.dataflow.dflow INFO: Standard output for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1c50c8f0-bcfc-40a2-9984-2ab8c03fc2e1_isrForCrosstalkSources_855403_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1c50c8f0-bcfc-40a2-9984-2ab8c03fc2e1_isrForCrosstalkSources_855403_29.stderr parsl.dataflow.dflow DEBUG: Task 2831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2831 has memoization hash 7e19099225bfb83ea508df39e4a6b3af parsl.dataflow.memoization INFO: Task 2831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2831 try 0 launched on executor multi with executor id 2832 parsl.dataflow.dflow INFO: Standard output for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d8dd7792-3120-402d-8c11-ff8fc7e44661_isrForCrosstalkSources_855385_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d8dd7792-3120-402d-8c11-ff8fc7e44661_isrForCrosstalkSources_855385_56.stderr parsl.dataflow.dflow DEBUG: Task 2832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2832 has memoization hash b4251d799cf0df07be99e7a95f79b0d6 parsl.dataflow.memoization INFO: Task 2832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2832 try 0 launched on executor multi with executor id 2833 parsl.dataflow.dflow INFO: Standard output for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0029f1ea-259f-407e-8d1e-1e1dd25ac411_isrForCrosstalkSources_855425_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0029f1ea-259f-407e-8d1e-1e1dd25ac411_isrForCrosstalkSources_855425_42.stderr parsl.dataflow.dflow DEBUG: Task 2833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2833 has memoization hash f808aa30c124a72f7e0875666ff85caa parsl.dataflow.memoization INFO: Task 2833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2833 try 0 launched on executor multi with executor id 2834 parsl.dataflow.dflow INFO: Standard output for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e636e68a-4ebd-402d-af07-9aebbed7465e_isrForCrosstalkSources_855428_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e636e68a-4ebd-402d-af07-9aebbed7465e_isrForCrosstalkSources_855428_35.stderr parsl.dataflow.dflow DEBUG: Task 2834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2834 has memoization hash abd4c7da40921fa8b41872cd06b3fdc0 parsl.dataflow.memoization INFO: Task 2834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2834 try 0 launched on executor multi with executor id 2835 parsl.dataflow.dflow INFO: Standard output for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c93d5bb2-e0cb-4621-8703-70a9939a16d1_isrForCrosstalkSources_855420_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c93d5bb2-e0cb-4621-8703-70a9939a16d1_isrForCrosstalkSources_855420_58.stderr parsl.dataflow.dflow DEBUG: Task 2835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2835 has memoization hash f3697f074ab31043b8c771ce7627fee6 parsl.dataflow.memoization INFO: Task 2835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2835 try 0 launched on executor multi with executor id 2836 parsl.dataflow.dflow INFO: Standard output for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/de2a6eeb-8c52-4656-a557-04e89e5a952d_isrForCrosstalkSources_855405_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/de2a6eeb-8c52-4656-a557-04e89e5a952d_isrForCrosstalkSources_855405_11.stderr parsl.dataflow.dflow DEBUG: Task 2836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2836 has memoization hash e840735b30a1f723db706e6c97a42e83 parsl.dataflow.memoization INFO: Task 2836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2836 try 0 launched on executor multi with executor id 2837 parsl.dataflow.dflow INFO: Standard output for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/cf926bef-e33b-4946-ade9-db3de9f17dba_isrForCrosstalkSources_855421_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/cf926bef-e33b-4946-ade9-db3de9f17dba_isrForCrosstalkSources_855421_19.stderr parsl.dataflow.dflow DEBUG: Task 2837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2837 has memoization hash 66279b0cb733e6a2f2f55eeea55901ee parsl.dataflow.memoization INFO: Task 2837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2837 try 0 launched on executor multi with executor id 2838 parsl.dataflow.dflow INFO: Standard output for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ed70679f-13c5-4711-ba7e-498a15398f61_isrForCrosstalkSources_855413_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ed70679f-13c5-4711-ba7e-498a15398f61_isrForCrosstalkSources_855413_29.stderr parsl.dataflow.dflow DEBUG: Task 2838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2838 has memoization hash 1ad37d40d8aa1d52761be95d126931b5 parsl.dataflow.memoization INFO: Task 2838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2838 try 0 launched on executor multi with executor id 2839 parsl.dataflow.dflow INFO: Standard output for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b74124e0-a748-4799-9482-f30d6f5c0b34_isrForCrosstalkSources_855381_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b74124e0-a748-4799-9482-f30d6f5c0b34_isrForCrosstalkSources_855381_58.stderr parsl.dataflow.dflow DEBUG: Task 2839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2839 has memoization hash 4a5faca61ff6932bdf1166677db018c6 parsl.dataflow.memoization INFO: Task 2839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2839 try 0 launched on executor multi with executor id 2840 parsl.dataflow.dflow INFO: Standard output for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/dc4d5441-a444-4674-b634-d6b2f675145a_isrForCrosstalkSources_855457_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/dc4d5441-a444-4674-b634-d6b2f675145a_isrForCrosstalkSources_855457_2.stderr parsl.dataflow.dflow DEBUG: Task 2840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2840 has memoization hash ef031e80590bb3a6b57aef03de158b55 parsl.dataflow.memoization INFO: Task 2840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2840 try 0 launched on executor multi with executor id 2841 parsl.dataflow.dflow INFO: Standard output for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65c89a88-e7f6-4f74-a0cf-d5017d17cc7c_isrForCrosstalkSources_855413_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65c89a88-e7f6-4f74-a0cf-d5017d17cc7c_isrForCrosstalkSources_855413_28.stderr parsl.dataflow.dflow DEBUG: Task 2841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2841 has memoization hash 3b563fb60267afe29bedd96a74bc1564 parsl.dataflow.memoization INFO: Task 2841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2841 try 0 launched on executor multi with executor id 2842 parsl.dataflow.dflow INFO: Standard output for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9973acd3-24db-454a-aaad-090390849dbf_isrForCrosstalkSources_855394_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9973acd3-24db-454a-aaad-090390849dbf_isrForCrosstalkSources_855394_16.stderr parsl.dataflow.dflow DEBUG: Task 2842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2842 has memoization hash 1cadf3d1208ab60e7e4954716d2452be parsl.dataflow.memoization INFO: Task 2842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2842 try 0 launched on executor multi with executor id 2843 parsl.dataflow.dflow INFO: Standard output for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe43df-d108-4b33-b260-30a56f2bee93_isrForCrosstalkSources_855423_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe43df-d108-4b33-b260-30a56f2bee93_isrForCrosstalkSources_855423_42.stderr parsl.dataflow.dflow DEBUG: Task 2843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2843 has memoization hash 3d547a978c470c533f7f774745274fb2 parsl.dataflow.memoization INFO: Task 2843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2843 try 0 launched on executor multi with executor id 2844 parsl.dataflow.dflow INFO: Standard output for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/267cf525-fc71-45d0-b4a2-4e30697e99b7_isrForCrosstalkSources_855448_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/267cf525-fc71-45d0-b4a2-4e30697e99b7_isrForCrosstalkSources_855448_12.stderr parsl.dataflow.dflow DEBUG: Task 2844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2844 has memoization hash 22daf464ee21aa24ef1b803589eedbed parsl.dataflow.memoization INFO: Task 2844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2844 try 0 launched on executor multi with executor id 2845 parsl.dataflow.dflow INFO: Standard output for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0eb18e47-d636-4e38-9272-a9ff15e65682_isrForCrosstalkSources_855427_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0eb18e47-d636-4e38-9272-a9ff15e65682_isrForCrosstalkSources_855427_26.stderr parsl.dataflow.dflow DEBUG: Task 2845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2845 has memoization hash 82b8c21122b7dc06c73ca45c337a34a6 parsl.dataflow.memoization INFO: Task 2845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2845 try 0 launched on executor multi with executor id 2846 parsl.dataflow.dflow INFO: Standard output for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/46ff46b7-4095-4b81-a319-53abf2406c2d_isrForCrosstalkSources_855444_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/46ff46b7-4095-4b81-a319-53abf2406c2d_isrForCrosstalkSources_855444_57.stderr parsl.dataflow.dflow DEBUG: Task 2846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2846 has memoization hash 4cf873635871b8139a267bdbd3b72e01 parsl.dataflow.memoization INFO: Task 2846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2846 try 0 launched on executor multi with executor id 2847 parsl.dataflow.dflow INFO: Standard output for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fc0b5ef9-fd1c-4d28-9643-729298565ecf_isrForCrosstalkSources_855449_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fc0b5ef9-fd1c-4d28-9643-729298565ecf_isrForCrosstalkSources_855449_48.stderr parsl.dataflow.dflow DEBUG: Task 2847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2847 has memoization hash 29036cf427b33f85e2f512f675271e0b parsl.dataflow.memoization INFO: Task 2847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2847 try 0 launched on executor multi with executor id 2848 parsl.dataflow.dflow INFO: Standard output for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cd09ea79-4a1f-41c7-a5bb-ae76ea5139e5_isrForCrosstalkSources_855407_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cd09ea79-4a1f-41c7-a5bb-ae76ea5139e5_isrForCrosstalkSources_855407_2.stderr parsl.dataflow.dflow DEBUG: Task 2848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2848 has memoization hash b9f3c560c2527d76f43c476358c6e1f6 parsl.dataflow.memoization INFO: Task 2848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2848 try 0 launched on executor multi with executor id 2849 parsl.dataflow.dflow INFO: Standard output for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6a6fe69-ae55-4e46-b0c1-af79673dd5ef_isrForCrosstalkSources_855457_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6a6fe69-ae55-4e46-b0c1-af79673dd5ef_isrForCrosstalkSources_855457_48.stderr parsl.dataflow.dflow DEBUG: Task 2849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2849 has memoization hash 2ae06f3d7be67c1b11dcb1bc989e7b83 parsl.dataflow.memoization INFO: Task 2849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:26 proc_lsst.multi:170 [INFO] [multi] job_id 16964157 proc_lsst.multi INFO: [multi] job_id 16964157 2024-03-09 14:51:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-09 14:51:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964157 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964157 parsl.executors.status_handling DEBUG: Launched block 131 on executor multi with job ID 16964157 parsl.executors.status_handling INFO: Allocated block ID 132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-132 2024-03-09 14:51:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2849 try 0 launched on executor multi with executor id 2850 parsl.dataflow.dflow INFO: Standard output for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5e2d4615-c189-466e-b537-b4f0e16614d5_isrForCrosstalkSources_855428_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5e2d4615-c189-466e-b537-b4f0e16614d5_isrForCrosstalkSources_855428_59.stderr parsl.dataflow.dflow DEBUG: Task 2850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2850 has memoization hash 85dfbf3721ec19f4fdb360e816b611aa parsl.dataflow.memoization INFO: Task 2850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6392700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2850 try 0 launched on executor multi with executor id 2851 parsl.dataflow.dflow INFO: Standard output for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/36d8fd98-45d9-48f6-915c-254eee6c00a9_isrForCrosstalkSources_855384_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/36d8fd98-45d9-48f6-915c-254eee6c00a9_isrForCrosstalkSources_855384_2.stderr parsl.dataflow.dflow DEBUG: Task 2851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2851 has memoization hash c7bf179d50101dd940251135137e4a5f parsl.dataflow.memoization INFO: Task 2851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2851 try 0 launched on executor multi with executor id 2852 parsl.dataflow.dflow INFO: Standard output for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d5ab5060-31b7-4f42-9031-43ed2b3dce22_isrForCrosstalkSources_855381_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d5ab5060-31b7-4f42-9031-43ed2b3dce22_isrForCrosstalkSources_855381_49.stderr parsl.dataflow.dflow DEBUG: Task 2852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2852 has memoization hash f9953a3e24946ab4faf2fa39a3875c17 parsl.dataflow.memoization INFO: Task 2852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2852 try 0 launched on executor multi with executor id 2853 parsl.dataflow.dflow INFO: Standard output for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/add8afd4-1065-47a5-b5e5-eafc50fb8b1f_isrForCrosstalkSources_855419_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/add8afd4-1065-47a5-b5e5-eafc50fb8b1f_isrForCrosstalkSources_855419_14.stderr parsl.dataflow.dflow DEBUG: Task 2853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2853 has memoization hash ab506ba26088dabbd8d04e40ef8b8be4 parsl.dataflow.memoization INFO: Task 2853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2853 try 0 launched on executor multi with executor id 2854 parsl.dataflow.dflow INFO: Standard output for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/51c10c10-07e6-4a43-ab78-208bbeae764c_isrForCrosstalkSources_855442_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/51c10c10-07e6-4a43-ab78-208bbeae764c_isrForCrosstalkSources_855442_16.stderr parsl.dataflow.dflow DEBUG: Task 2854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2854 has memoization hash c0d657d857d445e7b3f8e29e339f4dd7 parsl.dataflow.memoization INFO: Task 2854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2854 try 0 launched on executor multi with executor id 2855 parsl.dataflow.dflow INFO: Standard output for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/af5a5ac1-6b17-46a4-96e2-f75fe72b132f_isrForCrosstalkSources_855450_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/af5a5ac1-6b17-46a4-96e2-f75fe72b132f_isrForCrosstalkSources_855450_29.stderr parsl.dataflow.dflow DEBUG: Task 2855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2855 has memoization hash 915f05eee926de2687be6673071a7b36 parsl.dataflow.memoization INFO: Task 2855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2855 try 0 launched on executor multi with executor id 2856 parsl.dataflow.dflow INFO: Standard output for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1dff4c23-c6c6-4d99-8349-a3bc1bba5baf_isrForCrosstalkSources_855409_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1dff4c23-c6c6-4d99-8349-a3bc1bba5baf_isrForCrosstalkSources_855409_18.stderr parsl.dataflow.dflow DEBUG: Task 2856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2856 has memoization hash b508afd838e0225d88cac92a0a89543c parsl.dataflow.memoization INFO: Task 2856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2856 try 0 launched on executor multi with executor id 2857 parsl.dataflow.dflow INFO: Standard output for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/be814532-e83a-4a9b-b4a1-2d48e20c7382_isrForCrosstalkSources_855405_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/be814532-e83a-4a9b-b4a1-2d48e20c7382_isrForCrosstalkSources_855405_19.stderr parsl.dataflow.dflow DEBUG: Task 2857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2857 has memoization hash b103833e2af2dfd7c0ed8937fb2b5905 parsl.dataflow.memoization INFO: Task 2857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2857 try 0 launched on executor multi with executor id 2858 parsl.dataflow.dflow INFO: Standard output for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/9e441014-4a40-46a9-8503-cef7db0a1335_isrForCrosstalkSources_855405_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/9e441014-4a40-46a9-8503-cef7db0a1335_isrForCrosstalkSources_855405_8.stderr parsl.dataflow.dflow DEBUG: Task 2858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2858 has memoization hash 8b162c1e9aa5b9be3ee362dc5c80de0c parsl.dataflow.memoization INFO: Task 2858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2858 try 0 launched on executor multi with executor id 2859 parsl.dataflow.dflow INFO: Standard output for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9e1b533a-7e6a-449a-9674-5f7e5e0223ad_isrForCrosstalkSources_855380_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9e1b533a-7e6a-449a-9674-5f7e5e0223ad_isrForCrosstalkSources_855380_32.stderr parsl.dataflow.dflow DEBUG: Task 2859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2859 has memoization hash f19afaced834a5b0ae3d2b78807ddba0 parsl.dataflow.memoization INFO: Task 2859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2859 try 0 launched on executor multi with executor id 2860 parsl.dataflow.dflow INFO: Standard output for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918ec45-b20f-49b2-b1bd-6fc770db7ae2_isrForCrosstalkSources_855404_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918ec45-b20f-49b2-b1bd-6fc770db7ae2_isrForCrosstalkSources_855404_38.stderr parsl.dataflow.dflow DEBUG: Task 2860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2860 has memoization hash 4ced147320ca8b4e5766e8d5faab9e5a parsl.dataflow.memoization INFO: Task 2860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2860 try 0 launched on executor multi with executor id 2861 parsl.dataflow.dflow INFO: Standard output for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2b819dd0-ad9f-456a-a33a-78afd4359d53_isrForCrosstalkSources_855419_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2b819dd0-ad9f-456a-a33a-78afd4359d53_isrForCrosstalkSources_855419_38.stderr parsl.dataflow.dflow DEBUG: Task 2861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2861 has memoization hash 8112de04f122a03301822847117ad9dc parsl.dataflow.memoization INFO: Task 2861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2861 try 0 launched on executor multi with executor id 2862 parsl.dataflow.dflow INFO: Standard output for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6553d294-b8e7-4869-9ec8-39c29660854d_isrForCrosstalkSources_855392_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6553d294-b8e7-4869-9ec8-39c29660854d_isrForCrosstalkSources_855392_54.stderr parsl.dataflow.dflow DEBUG: Task 2862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2862 has memoization hash 3963d77f9e48a13f844566b2c00fe2db parsl.dataflow.memoization INFO: Task 2862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2862 try 0 launched on executor multi with executor id 2863 parsl.dataflow.dflow INFO: Standard output for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9f18aa49-7710-4427-b661-d3e67b662762_isrForCrosstalkSources_855396_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9f18aa49-7710-4427-b661-d3e67b662762_isrForCrosstalkSources_855396_18.stderr parsl.dataflow.dflow DEBUG: Task 2863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2863 has memoization hash 11ea269b2ba72c9481f734e4ac66a5d8 parsl.dataflow.memoization INFO: Task 2863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2863 try 0 launched on executor multi with executor id 2864 parsl.dataflow.dflow INFO: Standard output for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ae236b45-ea49-4d79-94ba-2018c602dcd5_isrForCrosstalkSources_855406_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ae236b45-ea49-4d79-94ba-2018c602dcd5_isrForCrosstalkSources_855406_11.stderr parsl.dataflow.dflow DEBUG: Task 2864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2864 has memoization hash fb089537aabb7644c44b4f762d247cc1 parsl.dataflow.memoization INFO: Task 2864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2864 try 0 launched on executor multi with executor id 2865 parsl.dataflow.dflow INFO: Standard output for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/30cd0134-2e33-4dd9-96a7-fcaff743d9f6_isrForCrosstalkSources_855440_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/30cd0134-2e33-4dd9-96a7-fcaff743d9f6_isrForCrosstalkSources_855440_59.stderr parsl.dataflow.dflow DEBUG: Task 2865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2865 has memoization hash 4a539c74529da61e2e5488922fb689aa parsl.dataflow.memoization INFO: Task 2865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2865 try 0 launched on executor multi with executor id 2866 parsl.dataflow.dflow INFO: Standard output for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/544d056e-9d5d-40b8-98f4-67660e544302_isrForCrosstalkSources_855419_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/544d056e-9d5d-40b8-98f4-67660e544302_isrForCrosstalkSources_855419_33.stderr parsl.dataflow.dflow DEBUG: Task 2866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2866 has memoization hash a452ff72b94d0a3bbbd2fc66ea953cfc parsl.dataflow.memoization INFO: Task 2866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2866 try 0 launched on executor multi with executor id 2867 parsl.dataflow.dflow INFO: Standard output for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/90f59eb0-d3b1-4075-a17d-12c3c49dc50e_isrForCrosstalkSources_855418_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/90f59eb0-d3b1-4075-a17d-12c3c49dc50e_isrForCrosstalkSources_855418_48.stderr parsl.dataflow.dflow DEBUG: Task 2867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2867 has memoization hash d01ff256d845dd632ceb43833a148bea parsl.dataflow.memoization INFO: Task 2867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2867 try 0 launched on executor multi with executor id 2868 parsl.dataflow.dflow INFO: Standard output for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d16b6c0-526e-4257-9912-7a3cb5ec35ff_isrForCrosstalkSources_855410_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d16b6c0-526e-4257-9912-7a3cb5ec35ff_isrForCrosstalkSources_855410_43.stderr parsl.dataflow.dflow DEBUG: Task 2868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2868 has memoization hash afa779c3df9802dc656b2d8f39d4bef1 parsl.dataflow.memoization INFO: Task 2868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2868 try 0 launched on executor multi with executor id 2869 parsl.dataflow.dflow INFO: Standard output for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bea83dc5-67f0-4e35-b846-cf420a92570c_isrForCrosstalkSources_855410_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bea83dc5-67f0-4e35-b846-cf420a92570c_isrForCrosstalkSources_855410_4.stderr parsl.dataflow.dflow DEBUG: Task 2869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2869 has memoization hash e653d54556889ef8a652a53db5a2622c parsl.dataflow.memoization INFO: Task 2869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2869 try 0 launched on executor multi with executor id 2870 parsl.dataflow.dflow INFO: Standard output for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ba004dae-c62a-4692-976a-7f8b96ee9429_isrForCrosstalkSources_855442_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ba004dae-c62a-4692-976a-7f8b96ee9429_isrForCrosstalkSources_855442_4.stderr parsl.dataflow.dflow DEBUG: Task 2870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2870 has memoization hash 58b63ea1353cecf1f1a2d7bd7c3878d0 parsl.dataflow.memoization INFO: Task 2870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2870 try 0 launched on executor multi with executor id 2871 parsl.dataflow.dflow INFO: Standard output for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3e9d0e05-ad1a-4b88-88fa-84161be46c99_isrForCrosstalkSources_855381_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3e9d0e05-ad1a-4b88-88fa-84161be46c99_isrForCrosstalkSources_855381_14.stderr parsl.dataflow.dflow DEBUG: Task 2871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2871 has memoization hash bbdae664948b786da6ba7dce62b6dc8e parsl.dataflow.memoization INFO: Task 2871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2871 try 0 launched on executor multi with executor id 2872 parsl.dataflow.dflow INFO: Standard output for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/21bc1823-f416-4d1c-ae26-212f97c2e007_isrForCrosstalkSources_855382_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/21bc1823-f416-4d1c-ae26-212f97c2e007_isrForCrosstalkSources_855382_16.stderr parsl.dataflow.dflow DEBUG: Task 2872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2872 has memoization hash fd7dcf654eb80a60f48ed9f2184d20ff parsl.dataflow.memoization INFO: Task 2872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2872 try 0 launched on executor multi with executor id 2873 parsl.dataflow.dflow INFO: Standard output for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b148d249-40ea-4b7e-aa65-2dce07456cd4_isrForCrosstalkSources_855415_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b148d249-40ea-4b7e-aa65-2dce07456cd4_isrForCrosstalkSources_855415_23.stderr parsl.dataflow.dflow DEBUG: Task 2873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2873 has memoization hash 9ec0cff7f336a42fb8ea3602349adea9 parsl.dataflow.memoization INFO: Task 2873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2873 try 0 launched on executor multi with executor id 2874 parsl.dataflow.dflow INFO: Standard output for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b27957e2-fe69-4234-b534-c065d42d3724_isrForCrosstalkSources_855381_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b27957e2-fe69-4234-b534-c065d42d3724_isrForCrosstalkSources_855381_59.stderr parsl.dataflow.dflow DEBUG: Task 2874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2874 has memoization hash 2a2381f3a7d4fafd93e6da3ab5c012a8 parsl.dataflow.memoization INFO: Task 2874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2874 try 0 launched on executor multi with executor id 2875 parsl.dataflow.dflow INFO: Standard output for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5853d7ed-8e7e-47e9-8d47-0cfa3abc0994_isrForCrosstalkSources_855433_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5853d7ed-8e7e-47e9-8d47-0cfa3abc0994_isrForCrosstalkSources_855433_26.stderr parsl.dataflow.dflow DEBUG: Task 2875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2875 has memoization hash 62e073283693ec78fbf5d4954d83816b parsl.dataflow.memoization INFO: Task 2875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2875 try 0 launched on executor multi with executor id 2876 parsl.dataflow.dflow INFO: Standard output for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/bad00a8f-ce24-4c96-98d0-237877bf641e_isrForCrosstalkSources_855423_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/bad00a8f-ce24-4c96-98d0-237877bf641e_isrForCrosstalkSources_855423_62.stderr parsl.dataflow.dflow DEBUG: Task 2876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2876 has memoization hash a6fb7f0f386c09ee3f6cce595ed3c582 parsl.dataflow.memoization INFO: Task 2876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2876 try 0 launched on executor multi with executor id 2877 parsl.dataflow.dflow INFO: Standard output for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/eec39f82-2fd6-4bc3-8663-66ecc95aeaba_isrForCrosstalkSources_855393_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/eec39f82-2fd6-4bc3-8663-66ecc95aeaba_isrForCrosstalkSources_855393_26.stderr parsl.dataflow.dflow DEBUG: Task 2877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2877 has memoization hash 0d51a3095fa17bda92d0dae3206dc785 parsl.dataflow.memoization INFO: Task 2877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2877 try 0 launched on executor multi with executor id 2878 parsl.dataflow.dflow INFO: Standard output for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cae5085c-dc87-4090-8941-ed76bc64c463_isrForCrosstalkSources_855422_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cae5085c-dc87-4090-8941-ed76bc64c463_isrForCrosstalkSources_855422_31.stderr parsl.dataflow.dflow DEBUG: Task 2878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2878 has memoization hash 5d147774503dd51a9bb737fdb03d98a7 parsl.dataflow.memoization INFO: Task 2878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2878 try 0 launched on executor multi with executor id 2879 parsl.dataflow.dflow INFO: Standard output for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c8db900d-95b7-46a3-a139-04a2281493cd_isrForCrosstalkSources_855385_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c8db900d-95b7-46a3-a139-04a2281493cd_isrForCrosstalkSources_855385_42.stderr parsl.dataflow.dflow DEBUG: Task 2879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2879 has memoization hash 8f772a80afc3c80e8a241c72c7ea1b3a parsl.dataflow.memoization INFO: Task 2879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2879 try 0 launched on executor multi with executor id 2880 parsl.dataflow.dflow INFO: Standard output for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/611f4f07-86ad-4548-b64b-7df7d7599ec5_isrForCrosstalkSources_855441_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/611f4f07-86ad-4548-b64b-7df7d7599ec5_isrForCrosstalkSources_855441_52.stderr parsl.dataflow.dflow DEBUG: Task 2880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2880 has memoization hash 80095bd666b15cfd33fd4237386b60f3 parsl.dataflow.memoization INFO: Task 2880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2880 try 0 launched on executor multi with executor id 2881 parsl.dataflow.dflow INFO: Standard output for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/32c6c38e-9725-48fa-91b5-007684518147_isrForCrosstalkSources_855411_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/32c6c38e-9725-48fa-91b5-007684518147_isrForCrosstalkSources_855411_40.stderr parsl.dataflow.dflow DEBUG: Task 2881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2881 has memoization hash 309b44b4a2c0dce3b53f0d7ab0bd992f parsl.dataflow.memoization INFO: Task 2881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2881 try 0 launched on executor multi with executor id 2882 parsl.dataflow.dflow INFO: Standard output for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dc4c5788-1d84-4304-875e-9bffe02430da_isrForCrosstalkSources_855446_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dc4c5788-1d84-4304-875e-9bffe02430da_isrForCrosstalkSources_855446_57.stderr parsl.dataflow.dflow DEBUG: Task 2882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2882 has memoization hash 1f34fc8a7754ad58432517901bc290f1 parsl.dataflow.memoization INFO: Task 2882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2882 try 0 launched on executor multi with executor id 2883 parsl.dataflow.dflow INFO: Standard output for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db48e072-fbe4-4516-a547-bdd46b28204d_isrForCrosstalkSources_855388_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db48e072-fbe4-4516-a547-bdd46b28204d_isrForCrosstalkSources_855388_44.stderr parsl.dataflow.dflow DEBUG: Task 2883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2883 has memoization hash 32365d0cacfa6d0e0a9b13c755815626 parsl.dataflow.memoization INFO: Task 2883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2883 try 0 launched on executor multi with executor id 2884 parsl.dataflow.dflow INFO: Standard output for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e743353d-eca1-4960-af85-3ac62de17e02_isrForCrosstalkSources_855400_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e743353d-eca1-4960-af85-3ac62de17e02_isrForCrosstalkSources_855400_36.stderr parsl.dataflow.dflow DEBUG: Task 2884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2884 has memoization hash 1e98cb39b008fcce5603438a58fa03ac parsl.dataflow.memoization INFO: Task 2884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2884 try 0 launched on executor multi with executor id 2885 parsl.dataflow.dflow INFO: Standard output for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/8489eddc-131e-4659-94b1-db07e6cd9656_isrForCrosstalkSources_855417_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/8489eddc-131e-4659-94b1-db07e6cd9656_isrForCrosstalkSources_855417_19.stderr parsl.dataflow.dflow DEBUG: Task 2885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2885 has memoization hash 5bcc208c540cd6c7ad29aaf82bdf04e0 parsl.dataflow.memoization INFO: Task 2885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8fe6273aab45ad5000179dcf36da704 with result from task 175 parsl.dataflow.dflow INFO: Parsl task 2885 try 0 launched on executor multi with executor id 2886 parsl.dataflow.dflow INFO: Standard output for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/eebd0392-76cf-4f72-a784-d94cf65ccd7b_isrForCrosstalkSources_855434_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/eebd0392-76cf-4f72-a784-d94cf65ccd7b_isrForCrosstalkSources_855434_22.stderr parsl.dataflow.dflow DEBUG: Task 2886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2886 has memoization hash 765f80ab6fa032f3367849e19788a1ec parsl.dataflow.memoization INFO: Task 2886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/370c6fbc-2b39-43a3-b9b8-101b56e76746_isrForCrosstalkSources_855422_9.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/370c6fbc-2b39-43a3-b9b8-101b56e76746_isrForCrosstalkSources_855422_9.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e36ac5dbb114f745f6991021b8eaf4b with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e2a2e292-dc77-488a-a796-14f58d8ec03b_isrForCrosstalkSources_855424_32.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e2a2e292-dc77-488a-a796-14f58d8ec03b_isrForCrosstalkSources_855424_32.stderr parsl.dataflow.dflow INFO: Parsl task 2886 try 0 launched on executor multi with executor id 2887 parsl.dataflow.dflow INFO: Standard output for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/daf3188a-915d-4dca-b572-f924e47d5ac3_isrForCrosstalkSources_855449_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/daf3188a-915d-4dca-b572-f924e47d5ac3_isrForCrosstalkSources_855449_50.stderr parsl.dataflow.dflow DEBUG: Task 2887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2887 has memoization hash b0b97bdd2fa01a5a33a816cd8944613b parsl.dataflow.memoization INFO: Task 2887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27638ebe8ccaf02cff667900efaf1323 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/80d98ee8-6872-4446-a050-fe6c562fe69d_isrForCrosstalkSources_855428_20.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/80d98ee8-6872-4446-a050-fe6c562fe69d_isrForCrosstalkSources_855428_20.stderr 2024-03-09 14:51:26 proc_lsst.multi:170 [INFO] [multi] job_id 16964158 proc_lsst.multi INFO: [multi] job_id 16964158 2024-03-09 14:51:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-09 14:51:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964158 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964158 parsl.executors.status_handling DEBUG: Launched block 132 on executor multi with job ID 16964158 parsl.executors.status_handling INFO: Allocated block ID 133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-133 2024-03-09 14:51:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-09 14:51:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2887 try 0 launched on executor multi with executor id 2888 parsl.dataflow.dflow INFO: Standard output for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ecada91b-e567-4404-976f-a9357ec728f4_isrForCrosstalkSources_855384_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ecada91b-e567-4404-976f-a9357ec728f4_isrForCrosstalkSources_855384_46.stderr parsl.dataflow.dflow DEBUG: Task 2888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2888 has memoization hash 31bee9a13cd130fe9a6025afa9e974ed parsl.dataflow.memoization INFO: Task 2888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2888 try 0 launched on executor multi with executor id 2889 parsl.dataflow.dflow INFO: Standard output for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8b066a2f-ea8f-4d47-a6f3-0d267ab6def8_isrForCrosstalkSources_855395_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8b066a2f-ea8f-4d47-a6f3-0d267ab6def8_isrForCrosstalkSources_855395_11.stderr parsl.dataflow.dflow DEBUG: Task 2889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2889 has memoization hash 8e9459ce03dd103a679499699d6f5cb3 parsl.dataflow.memoization INFO: Task 2889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2889 try 0 launched on executor multi with executor id 2890 parsl.dataflow.dflow INFO: Standard output for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d1acbe6c-7b74-4283-ab09-75a423a0b516_isrForCrosstalkSources_855410_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d1acbe6c-7b74-4283-ab09-75a423a0b516_isrForCrosstalkSources_855410_47.stderr parsl.dataflow.dflow DEBUG: Task 2890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2890 has memoization hash 731fc5682d5d04dbd11312f18ab8916c parsl.dataflow.memoization INFO: Task 2890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2890 try 0 launched on executor multi with executor id 2891 parsl.dataflow.dflow INFO: Standard output for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0e27612a-7b67-4021-bca6-c402ce8ac85a_isrForCrosstalkSources_855446_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0e27612a-7b67-4021-bca6-c402ce8ac85a_isrForCrosstalkSources_855446_14.stderr parsl.dataflow.dflow DEBUG: Task 2891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2891 has memoization hash 4f10f7045df775d326efeff26c3798c4 parsl.dataflow.memoization INFO: Task 2891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2891 try 0 launched on executor multi with executor id 2892 parsl.dataflow.dflow INFO: Standard output for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e21f2282-c2eb-4396-89b1-eedfee0aac81_isrForCrosstalkSources_855414_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e21f2282-c2eb-4396-89b1-eedfee0aac81_isrForCrosstalkSources_855414_42.stderr parsl.dataflow.dflow DEBUG: Task 2892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2892 has memoization hash 32e465211302cafe8d6b3fcbdb2ede36 parsl.dataflow.memoization INFO: Task 2892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2892 try 0 launched on executor multi with executor id 2893 parsl.dataflow.dflow INFO: Standard output for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/d9727d62-4663-409c-81a5-e9069608445c_isrForCrosstalkSources_855424_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/d9727d62-4663-409c-81a5-e9069608445c_isrForCrosstalkSources_855424_43.stderr parsl.dataflow.dflow DEBUG: Task 2893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2893 has memoization hash 868d38a7912c146d6350d7c060fbb458 parsl.dataflow.memoization INFO: Task 2893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2893 try 0 launched on executor multi with executor id 2894 parsl.dataflow.dflow INFO: Standard output for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/6d15e703-ff5d-4399-a1f2-200fb6a0ac94_isrForCrosstalkSources_855388_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/6d15e703-ff5d-4399-a1f2-200fb6a0ac94_isrForCrosstalkSources_855388_11.stderr parsl.dataflow.dflow DEBUG: Task 2894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2894 has memoization hash 51b8226a43c66684ffc7e83ea2e87a75 parsl.dataflow.memoization INFO: Task 2894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4abd4312d6d148ba4b96760270538273 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a29152d3-845a-4cc9-b368-079f2165a343_isrForCrosstalkSources_855452_4.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a29152d3-845a-4cc9-b368-079f2165a343_isrForCrosstalkSources_855452_4.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2894 try 0 launched on executor multi with executor id 2895 parsl.dataflow.dflow INFO: Standard output for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7d2a2699-0883-480f-a326-cdce309b08ea_isrForCrosstalkSources_855424_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7d2a2699-0883-480f-a326-cdce309b08ea_isrForCrosstalkSources_855424_37.stderr parsl.dataflow.dflow DEBUG: Task 2895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2895 has memoization hash c5ceb6b479dfdf0963282b8f112eadda parsl.dataflow.memoization INFO: Task 2895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2895 try 0 launched on executor multi with executor id 2896 parsl.dataflow.dflow INFO: Standard output for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/647a6e96-ddf3-46e9-bfd6-404a82359225_isrForCrosstalkSources_855391_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/647a6e96-ddf3-46e9-bfd6-404a82359225_isrForCrosstalkSources_855391_14.stderr parsl.dataflow.dflow DEBUG: Task 2896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2896 has memoization hash a4188931e808cda5d9be15ef76f2cd0f parsl.dataflow.memoization INFO: Task 2896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2896 try 0 launched on executor multi with executor id 2897 parsl.dataflow.dflow INFO: Standard output for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ee4653b1-5ff1-4061-ac0a-cb1eb1f3c655_isrForCrosstalkSources_855450_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ee4653b1-5ff1-4061-ac0a-cb1eb1f3c655_isrForCrosstalkSources_855450_8.stderr parsl.dataflow.dflow DEBUG: Task 2897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2897 has memoization hash c7bd4aa437e6331d3c1962d54555aca3 parsl.dataflow.memoization INFO: Task 2897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2897 try 0 launched on executor multi with executor id 2898 parsl.dataflow.dflow INFO: Standard output for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/137c4698-e6f6-4ae1-9f75-266db303f298_isrForCrosstalkSources_855420_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/137c4698-e6f6-4ae1-9f75-266db303f298_isrForCrosstalkSources_855420_6.stderr parsl.dataflow.dflow DEBUG: Task 2898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2898 has memoization hash cfcc5d8ffd75e8de553f80140e614fcc parsl.dataflow.memoization INFO: Task 2898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c610e18cd185ff14d8d8e19eb76901c8 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/053f7b10-d523-4cc3-8226-3ed5a5851e1c_isrForCrosstalkSources_855406_26.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/053f7b10-d523-4cc3-8226-3ed5a5851e1c_isrForCrosstalkSources_855406_26.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc14f2952ee33110f2ebbe4513327455 with result from task 160 parsl.dataflow.dflow INFO: Parsl task 2898 try 0 launched on executor multi with executor id 2899 parsl.dataflow.dflow INFO: Standard output for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/380aa4aa-b58b-4a27-b910-4db0d80e7860_isrForCrosstalkSources_855424_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/380aa4aa-b58b-4a27-b910-4db0d80e7860_isrForCrosstalkSources_855424_62.stderr parsl.dataflow.dflow DEBUG: Task 2899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2899 has memoization hash ea0d88f7e40582a9f328622d29c9183c parsl.dataflow.memoization INFO: Task 2899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2899 try 0 launched on executor multi with executor id 2900 parsl.dataflow.dflow INFO: Standard output for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eec25275-1e5a-477d-b959-a004185d0d85_isrForCrosstalkSources_855446_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eec25275-1e5a-477d-b959-a004185d0d85_isrForCrosstalkSources_855446_17.stderr parsl.dataflow.dflow DEBUG: Task 2900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2900 has memoization hash d3b1c6b296c316bdbb33cfa8eed85d1e parsl.dataflow.memoization INFO: Task 2900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2900 try 0 launched on executor multi with executor id 2901 parsl.dataflow.dflow INFO: Standard output for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b738176c-b4de-4090-b55b-90d7120944fd_isrForCrosstalkSources_855420_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b738176c-b4de-4090-b55b-90d7120944fd_isrForCrosstalkSources_855420_47.stderr parsl.dataflow.dflow DEBUG: Task 2901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2901 has memoization hash 13e68560d0c88bc447ffad820d8074fe parsl.dataflow.memoization INFO: Task 2901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/773e3c0d-b1d5-4a87-88eb-80c19858afb3_isrForCrosstalkSources_855410_25.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/773e3c0d-b1d5-4a87-88eb-80c19858afb3_isrForCrosstalkSources_855410_25.stderr parsl.dataflow.dflow INFO: Parsl task 2901 try 0 launched on executor multi with executor id 2902 parsl.dataflow.dflow INFO: Standard output for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/258910da-2ca9-4fbe-9e34-b71bc747cbb6_isrForCrosstalkSources_855437_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/258910da-2ca9-4fbe-9e34-b71bc747cbb6_isrForCrosstalkSources_855437_43.stderr parsl.dataflow.dflow DEBUG: Task 2902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2902 has memoization hash 62cf501432cc95b3c0a56eba67a4af9d parsl.dataflow.memoization INFO: Task 2902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2902 try 0 launched on executor multi with executor id 2903 parsl.dataflow.dflow INFO: Standard output for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d272b08f-60cb-4fec-b92c-f8d289134a1e_isrForCrosstalkSources_855428_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d272b08f-60cb-4fec-b92c-f8d289134a1e_isrForCrosstalkSources_855428_21.stderr parsl.dataflow.dflow DEBUG: Task 2903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2903 has memoization hash 928da7059e0c5464997f285a4feabb36 parsl.dataflow.memoization INFO: Task 2903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2903 try 0 launched on executor multi with executor id 2904 parsl.dataflow.dflow INFO: Standard output for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/fb880062-15e9-48e5-9133-0996e05e8258_isrForCrosstalkSources_855422_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/fb880062-15e9-48e5-9133-0996e05e8258_isrForCrosstalkSources_855422_28.stderr parsl.dataflow.dflow DEBUG: Task 2904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2904 has memoization hash 9faa2d46a80dc4758c59045fd06373dc parsl.dataflow.memoization INFO: Task 2904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2904 try 0 launched on executor multi with executor id 2905 parsl.dataflow.dflow INFO: Standard output for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/6b11495c-6372-48e4-805f-a0364c108295_isrForCrosstalkSources_855442_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/6b11495c-6372-48e4-805f-a0364c108295_isrForCrosstalkSources_855442_39.stderr parsl.dataflow.dflow DEBUG: Task 2905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2905 has memoization hash 20027197983cc0d79b6a437d8d9d4990 parsl.dataflow.memoization INFO: Task 2905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2905 try 0 launched on executor multi with executor id 2906 parsl.dataflow.dflow INFO: Standard output for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aea0dc2-3386-4e39-b9dc-d675adeed7f5_isrForCrosstalkSources_855411_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aea0dc2-3386-4e39-b9dc-d675adeed7f5_isrForCrosstalkSources_855411_11.stderr parsl.dataflow.dflow DEBUG: Task 2906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2906 has memoization hash 8d3a44b2d537867e1d4f7d514bce4a99 parsl.dataflow.memoization INFO: Task 2906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2906 try 0 launched on executor multi with executor id 2907 parsl.dataflow.dflow INFO: Standard output for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccaddd3a-6150-4004-b5ff-71d5ad63497f_isrForCrosstalkSources_855433_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccaddd3a-6150-4004-b5ff-71d5ad63497f_isrForCrosstalkSources_855433_27.stderr parsl.dataflow.dflow DEBUG: Task 2907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2907 has memoization hash 4518ff85902d298c34376e189e0809b9 parsl.dataflow.memoization INFO: Task 2907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2907 try 0 launched on executor multi with executor id 2908 parsl.dataflow.dflow INFO: Standard output for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/dac43149-1333-4b22-b991-b36945f0f092_isrForCrosstalkSources_855411_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/dac43149-1333-4b22-b991-b36945f0f092_isrForCrosstalkSources_855411_35.stderr parsl.dataflow.dflow DEBUG: Task 2908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2908 has memoization hash d73b8b7fb9565ab07a092fd2071f3d28 parsl.dataflow.memoization INFO: Task 2908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2908 try 0 launched on executor multi with executor id 2909 parsl.dataflow.dflow INFO: Standard output for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3e57a946-3749-470f-a889-34fab3eb9158_isrForCrosstalkSources_855380_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3e57a946-3749-470f-a889-34fab3eb9158_isrForCrosstalkSources_855380_36.stderr parsl.dataflow.dflow DEBUG: Task 2909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2909 has memoization hash 1374e220450430f7787ee2f1467a1bcc parsl.dataflow.memoization INFO: Task 2909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2909 try 0 launched on executor multi with executor id 2910 parsl.dataflow.dflow INFO: Standard output for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/74c707f4-011e-4362-9e31-1b996f252ebb_isrForCrosstalkSources_855451_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/74c707f4-011e-4362-9e31-1b996f252ebb_isrForCrosstalkSources_855451_55.stderr parsl.dataflow.dflow DEBUG: Task 2910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2910 has memoization hash 24ed55a8448d00b21fb280e34caee793 parsl.dataflow.memoization INFO: Task 2910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2910 try 0 launched on executor multi with executor id 2911 parsl.dataflow.dflow INFO: Standard output for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/33354eb1-a860-446f-9adf-b8e4209ea673_isrForCrosstalkSources_855395_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/33354eb1-a860-446f-9adf-b8e4209ea673_isrForCrosstalkSources_855395_52.stderr parsl.dataflow.dflow DEBUG: Task 2911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2911 has memoization hash f39749e69fd6381bba8744f9ff15b779 parsl.dataflow.memoization INFO: Task 2911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2911 try 0 launched on executor multi with executor id 2912 parsl.dataflow.dflow INFO: Standard output for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/4bd27203-1214-4651-b0ee-0d614c4d175f_isrForCrosstalkSources_855420_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/4bd27203-1214-4651-b0ee-0d614c4d175f_isrForCrosstalkSources_855420_53.stderr parsl.dataflow.dflow DEBUG: Task 2912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2912 has memoization hash b0a4889eb3c8590dd12e4ea44733ec71 parsl.dataflow.memoization INFO: Task 2912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2912 try 0 launched on executor multi with executor id 2913 parsl.dataflow.dflow INFO: Standard output for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a88d2115-8738-46a7-9d44-59627189149a_isrForCrosstalkSources_855383_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a88d2115-8738-46a7-9d44-59627189149a_isrForCrosstalkSources_855383_10.stderr parsl.dataflow.dflow DEBUG: Task 2913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2913 has memoization hash 9b187dc14b7e98d809235feebc91caed parsl.dataflow.memoization INFO: Task 2913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2913 try 0 launched on executor multi with executor id 2914 parsl.dataflow.dflow INFO: Standard output for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/be3f152b-722e-4cec-bf81-a29cb251e274_isrForCrosstalkSources_855400_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/be3f152b-722e-4cec-bf81-a29cb251e274_isrForCrosstalkSources_855400_62.stderr parsl.dataflow.dflow DEBUG: Task 2914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2914 has memoization hash 1c2d121892002538a255f395e7e45c62 parsl.dataflow.memoization INFO: Task 2914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2914 try 0 launched on executor multi with executor id 2915 parsl.dataflow.dflow INFO: Standard output for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1d75fac3-806d-42ea-b2f2-e92c949b32b4_isrForCrosstalkSources_855427_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1d75fac3-806d-42ea-b2f2-e92c949b32b4_isrForCrosstalkSources_855427_43.stderr parsl.dataflow.dflow DEBUG: Task 2915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2915 has memoization hash 785f181f8bacda485d34aec1fa56fba9 parsl.dataflow.memoization INFO: Task 2915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2915 try 0 launched on executor multi with executor id 2916 parsl.dataflow.dflow INFO: Standard output for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6a38255e-8ac8-456c-a75f-a3d1bbc41e92_isrForCrosstalkSources_855412_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6a38255e-8ac8-456c-a75f-a3d1bbc41e92_isrForCrosstalkSources_855412_12.stderr parsl.dataflow.dflow DEBUG: Task 2916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2916 has memoization hash ad474e0d8747f5e330dde9a3235a7bcf parsl.dataflow.memoization INFO: Task 2916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2916 try 0 launched on executor multi with executor id 2917 parsl.dataflow.dflow INFO: Standard output for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fd3e1c6a-5489-4d23-a7ec-8a907b8d60b8_isrForCrosstalkSources_855458_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fd3e1c6a-5489-4d23-a7ec-8a907b8d60b8_isrForCrosstalkSources_855458_24.stderr parsl.dataflow.dflow DEBUG: Task 2917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2917 has memoization hash ae1928f1ac5eba0d17f7ed7cfef00747 parsl.dataflow.memoization INFO: Task 2917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2917 try 0 launched on executor multi with executor id 2918 parsl.dataflow.dflow INFO: Standard output for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8a8ff100-a2ec-4214-bcbe-79820791394d_isrForCrosstalkSources_855407_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8a8ff100-a2ec-4214-bcbe-79820791394d_isrForCrosstalkSources_855407_27.stderr parsl.dataflow.dflow DEBUG: Task 2918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2918 has memoization hash e379fb19e32d13519721543f7f7c3da1 parsl.dataflow.memoization INFO: Task 2918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:27 proc_lsst.multi:170 [INFO] [multi] job_id 16964159 proc_lsst.multi INFO: [multi] job_id 16964159 2024-03-09 14:51:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-09 14:51:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964159 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964159 parsl.executors.status_handling DEBUG: Launched block 133 on executor multi with job ID 16964159 parsl.executors.status_handling INFO: Allocated block ID 134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-134 2024-03-09 14:51:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 2024-03-09 14:51:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-09 14:51:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2918 try 0 launched on executor multi with executor id 2919 parsl.dataflow.dflow INFO: Standard output for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/10b52193-a551-420b-b5c9-0c5076af39c0_isrForCrosstalkSources_855400_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/10b52193-a551-420b-b5c9-0c5076af39c0_isrForCrosstalkSources_855400_47.stderr parsl.dataflow.dflow DEBUG: Task 2919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2919 has memoization hash 8b406b198ed72584112d64cd68e951e9 parsl.dataflow.memoization INFO: Task 2919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2919 try 0 launched on executor multi with executor id 2920 parsl.dataflow.dflow INFO: Standard output for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2c982f0f-6d82-4f2e-a589-3ec56c55d0a7_isrForCrosstalkSources_855447_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2c982f0f-6d82-4f2e-a589-3ec56c55d0a7_isrForCrosstalkSources_855447_47.stderr parsl.dataflow.dflow DEBUG: Task 2920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2920 has memoization hash e947bfa8dd5ff9bccaa273b3a9cf0924 parsl.dataflow.memoization INFO: Task 2920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2920 try 0 launched on executor multi with executor id 2921 parsl.dataflow.dflow INFO: Standard output for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aa8fff54-1be5-430e-8ce0-ff95d520b18b_isrForCrosstalkSources_855426_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aa8fff54-1be5-430e-8ce0-ff95d520b18b_isrForCrosstalkSources_855426_62.stderr parsl.dataflow.dflow DEBUG: Task 2921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2921 has memoization hash 96c23fee7ca3368abad57a31c7b2d07b parsl.dataflow.memoization INFO: Task 2921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2921 try 0 launched on executor multi with executor id 2922 parsl.dataflow.dflow INFO: Standard output for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e55cfff1-1657-402a-9a19-007feeb4489c_isrForCrosstalkSources_855445_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e55cfff1-1657-402a-9a19-007feeb4489c_isrForCrosstalkSources_855445_41.stderr parsl.dataflow.dflow DEBUG: Task 2922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2922 has memoization hash bd17d01bf41166e1410c4a6b59667571 parsl.dataflow.memoization INFO: Task 2922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2922 try 0 launched on executor multi with executor id 2923 parsl.dataflow.dflow INFO: Standard output for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ca3b1ed9-3a0f-4536-8c9c-02fbeca6a046_isrForCrosstalkSources_855411_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ca3b1ed9-3a0f-4536-8c9c-02fbeca6a046_isrForCrosstalkSources_855411_33.stderr parsl.dataflow.dflow DEBUG: Task 2923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2923 has memoization hash cae145e8d794415b7f71139f82daccac parsl.dataflow.memoization INFO: Task 2923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2923 try 0 launched on executor multi with executor id 2924 parsl.dataflow.dflow INFO: Standard output for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ee5ca1a4-085f-455f-82bb-3e596ad01efa_isrForCrosstalkSources_855433_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ee5ca1a4-085f-455f-82bb-3e596ad01efa_isrForCrosstalkSources_855433_57.stderr parsl.dataflow.dflow DEBUG: Task 2924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2924 has memoization hash b780f2aef3a188c3e8754efcc6c61002 parsl.dataflow.memoization INFO: Task 2924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2924 try 0 launched on executor multi with executor id 2925 parsl.dataflow.dflow INFO: Standard output for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/230a11b2-2718-4e49-8e05-4209600d57ec_isrForCrosstalkSources_855379_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/230a11b2-2718-4e49-8e05-4209600d57ec_isrForCrosstalkSources_855379_41.stderr parsl.dataflow.dflow DEBUG: Task 2925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2925 has memoization hash dcb537105d89085645a428447d962dca parsl.dataflow.memoization INFO: Task 2925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2925 try 0 launched on executor multi with executor id 2926 parsl.dataflow.dflow INFO: Standard output for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d1d3fba8-bbca-4b78-a1e7-22282812ee4c_isrForCrosstalkSources_855420_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d1d3fba8-bbca-4b78-a1e7-22282812ee4c_isrForCrosstalkSources_855420_36.stderr parsl.dataflow.dflow DEBUG: Task 2926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2926 has memoization hash 4335e4775fdfcf29622be9fd468d67bc parsl.dataflow.memoization INFO: Task 2926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2926 try 0 launched on executor multi with executor id 2927 parsl.dataflow.dflow INFO: Standard output for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c5273657-e651-4517-8c76-6735290b2160_isrForCrosstalkSources_855427_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c5273657-e651-4517-8c76-6735290b2160_isrForCrosstalkSources_855427_59.stderr parsl.dataflow.dflow DEBUG: Task 2927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2927 has memoization hash 6da267f29b3507543b4e63790b2269cf parsl.dataflow.memoization INFO: Task 2927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2927 try 0 launched on executor multi with executor id 2928 parsl.dataflow.dflow INFO: Standard output for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/89b04792-8137-4e13-a249-01df559f5993_isrForCrosstalkSources_855383_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/89b04792-8137-4e13-a249-01df559f5993_isrForCrosstalkSources_855383_27.stderr parsl.dataflow.dflow DEBUG: Task 2928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2928 has memoization hash 0f035e9d09ce01d5c9743c9a33a581a3 parsl.dataflow.memoization INFO: Task 2928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2928 try 0 launched on executor multi with executor id 2929 parsl.dataflow.dflow INFO: Standard output for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78cfe66b-30a8-43e2-9174-121f2f31ab27_isrForCrosstalkSources_855433_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78cfe66b-30a8-43e2-9174-121f2f31ab27_isrForCrosstalkSources_855433_52.stderr parsl.dataflow.dflow DEBUG: Task 2929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2929 has memoization hash d9f880c265237715f7cea9209a6f481f parsl.dataflow.memoization INFO: Task 2929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2929 try 0 launched on executor multi with executor id 2930 parsl.dataflow.dflow INFO: Standard output for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/15be9895-4cfc-40c4-8b05-adb7a8f64434_isrForCrosstalkSources_855394_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/15be9895-4cfc-40c4-8b05-adb7a8f64434_isrForCrosstalkSources_855394_7.stderr parsl.dataflow.dflow DEBUG: Task 2930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2930 has memoization hash e2935a72c9bc1fee36a53f0a3a0d7f8f parsl.dataflow.memoization INFO: Task 2930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2930 try 0 launched on executor multi with executor id 2931 parsl.dataflow.dflow INFO: Standard output for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/328f7bab-0af5-4403-9cf3-5da21f02aa35_isrForCrosstalkSources_855439_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/328f7bab-0af5-4403-9cf3-5da21f02aa35_isrForCrosstalkSources_855439_19.stderr parsl.dataflow.dflow DEBUG: Task 2931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2931 has memoization hash 301722810146c3ec994aa680bb49bb92 parsl.dataflow.memoization INFO: Task 2931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2931 try 0 launched on executor multi with executor id 2932 parsl.dataflow.dflow INFO: Standard output for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/84c2b62d-d718-40fa-ac38-1e51f073c27e_isrForCrosstalkSources_855443_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/84c2b62d-d718-40fa-ac38-1e51f073c27e_isrForCrosstalkSources_855443_13.stderr parsl.dataflow.dflow DEBUG: Task 2932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2932 has memoization hash 8890cc9593db6aeb04939e5f1a40a997 parsl.dataflow.memoization INFO: Task 2932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2932 try 0 launched on executor multi with executor id 2933 parsl.dataflow.dflow INFO: Standard output for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2cfe14d7-365e-43f1-a539-c2ecba14aadd_isrForCrosstalkSources_855434_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2cfe14d7-365e-43f1-a539-c2ecba14aadd_isrForCrosstalkSources_855434_46.stderr parsl.dataflow.dflow DEBUG: Task 2933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2933 has memoization hash bc00911ea837a209e580059c602bf1c4 parsl.dataflow.memoization INFO: Task 2933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2933 try 0 launched on executor multi with executor id 2934 parsl.dataflow.dflow INFO: Standard output for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf0f8caa-c4e9-4790-956d-b1911e7c0dca_isrForCrosstalkSources_855402_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf0f8caa-c4e9-4790-956d-b1911e7c0dca_isrForCrosstalkSources_855402_38.stderr parsl.dataflow.dflow DEBUG: Task 2934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2934 has memoization hash 6b0574cde876f5022faf13a4edd7525e parsl.dataflow.memoization INFO: Task 2934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2934 try 0 launched on executor multi with executor id 2935 parsl.dataflow.dflow INFO: Standard output for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1b1cea2d-a9f2-4151-a3e8-d5d264ccbff2_isrForCrosstalkSources_855459_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1b1cea2d-a9f2-4151-a3e8-d5d264ccbff2_isrForCrosstalkSources_855459_11.stderr parsl.dataflow.dflow DEBUG: Task 2935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2935 has memoization hash c2fe2af1b367b7d9e94636c79f2a1ef1 parsl.dataflow.memoization INFO: Task 2935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2935 try 0 launched on executor multi with executor id 2936 parsl.dataflow.dflow INFO: Standard output for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/412509dc-3a20-42f0-8aa8-bf0e2983b2f5_isrForCrosstalkSources_855394_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/412509dc-3a20-42f0-8aa8-bf0e2983b2f5_isrForCrosstalkSources_855394_46.stderr parsl.dataflow.dflow DEBUG: Task 2936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2936 has memoization hash 3887bf7be31b646c4ddebc68b3c87fee parsl.dataflow.memoization INFO: Task 2936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2936 try 0 launched on executor multi with executor id 2937 parsl.dataflow.dflow INFO: Standard output for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5445774a-ebc0-4762-86b5-bbd58a9c8f6b_isrForCrosstalkSources_855395_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5445774a-ebc0-4762-86b5-bbd58a9c8f6b_isrForCrosstalkSources_855395_54.stderr parsl.dataflow.dflow DEBUG: Task 2937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2937 has memoization hash bafd1fa5e4a95ff998db3234a085d45c parsl.dataflow.memoization INFO: Task 2937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6393420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2937 try 0 launched on executor multi with executor id 2938 parsl.dataflow.dflow INFO: Standard output for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6ec7b5fa-29ae-4eca-a098-3d4e7ab05433_isrForCrosstalkSources_855402_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6ec7b5fa-29ae-4eca-a098-3d4e7ab05433_isrForCrosstalkSources_855402_41.stderr parsl.dataflow.dflow DEBUG: Task 2938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2938 has memoization hash 1ceb17d6ddfc59fcc600302f33586e59 parsl.dataflow.memoization INFO: Task 2938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2938 try 0 launched on executor multi with executor id 2939 parsl.dataflow.dflow INFO: Standard output for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e7522906-dbba-480a-9be0-579d55840a1b_isrForCrosstalkSources_855380_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e7522906-dbba-480a-9be0-579d55840a1b_isrForCrosstalkSources_855380_58.stderr parsl.dataflow.dflow DEBUG: Task 2939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2939 has memoization hash 132ec103eca99007b02f58562959300f parsl.dataflow.memoization INFO: Task 2939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a09740dce4ab9860819ba967fdd5e43e with result from task 49 parsl.dataflow.dflow INFO: Parsl task 2939 try 0 launched on executor multi with executor id 2940 parsl.dataflow.dflow INFO: Standard output for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/af4e2486-9ed7-4f5e-abbb-43589408cf88_isrForCrosstalkSources_855395_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/af4e2486-9ed7-4f5e-abbb-43589408cf88_isrForCrosstalkSources_855395_16.stderr parsl.dataflow.dflow DEBUG: Task 2940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2940 has memoization hash da7dbe9c0ff3a89102afffc3c40131c3 parsl.dataflow.memoization INFO: Task 2940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2940 try 0 launched on executor multi with executor id 2941 parsl.dataflow.dflow INFO: Standard output for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0d6a8ec2-c008-4447-94fb-8388dc06a86b_isrForCrosstalkSources_855395_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0d6a8ec2-c008-4447-94fb-8388dc06a86b_isrForCrosstalkSources_855395_53.stderr parsl.dataflow.dflow DEBUG: Task 2941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2941 has memoization hash 8947771e8339584e38c7d18720e61177 parsl.dataflow.memoization INFO: Task 2941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cece0a50-f82b-4939-bffd-6a69d4d9b76f_isrForCrosstalkSources_855409_21.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cece0a50-f82b-4939-bffd-6a69d4d9b76f_isrForCrosstalkSources_855409_21.stderr parsl.dataflow.dflow INFO: Parsl task 2941 try 0 launched on executor multi with executor id 2942 parsl.dataflow.dflow INFO: Standard output for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/395a77d8-11f5-49eb-9fae-719d22e1fb41_isrForCrosstalkSources_855414_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/395a77d8-11f5-49eb-9fae-719d22e1fb41_isrForCrosstalkSources_855414_17.stderr parsl.dataflow.dflow DEBUG: Task 2942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2942 has memoization hash 8adf1b68cd72316d7d896a61210e9e00 parsl.dataflow.memoization INFO: Task 2942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2942 try 0 launched on executor multi with executor id 2943 parsl.dataflow.dflow INFO: Standard output for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/854677dc-76bf-46a1-86fb-74aad2d9958a_isrForCrosstalkSources_855381_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/854677dc-76bf-46a1-86fb-74aad2d9958a_isrForCrosstalkSources_855381_9.stderr parsl.dataflow.dflow DEBUG: Task 2943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2943 has memoization hash 3e94b6111863732f2d400afa0888f4bc parsl.dataflow.memoization INFO: Task 2943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2943 try 0 launched on executor multi with executor id 2944 parsl.dataflow.dflow INFO: Standard output for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5e688c1c-eb8f-4c26-8506-890275866973_isrForCrosstalkSources_855394_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5e688c1c-eb8f-4c26-8506-890275866973_isrForCrosstalkSources_855394_49.stderr parsl.dataflow.dflow DEBUG: Task 2944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2944 has memoization hash e7bf679287721b167f895586b39b1c2c parsl.dataflow.memoization INFO: Task 2944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2944 try 0 launched on executor multi with executor id 2945 parsl.dataflow.dflow INFO: Standard output for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/919c6312-7078-4649-aa86-ebe5ab689852_isrForCrosstalkSources_855422_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/919c6312-7078-4649-aa86-ebe5ab689852_isrForCrosstalkSources_855422_5.stderr parsl.dataflow.dflow DEBUG: Task 2945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2945 has memoization hash 398915a9622cc054cde096153f394731 parsl.dataflow.memoization INFO: Task 2945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2945 try 0 launched on executor multi with executor id 2946 parsl.dataflow.dflow INFO: Standard output for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/768d5dd4-bd31-4491-afca-d7c9d619261d_isrForCrosstalkSources_855417_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/768d5dd4-bd31-4491-afca-d7c9d619261d_isrForCrosstalkSources_855417_22.stderr parsl.dataflow.dflow DEBUG: Task 2946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2946 has memoization hash 420669f63c3c48ee48040f593587897d parsl.dataflow.memoization INFO: Task 2946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:28 proc_lsst.multi:170 [INFO] [multi] job_id 16964160 proc_lsst.multi INFO: [multi] job_id 16964160 2024-03-09 14:51:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-09 14:51:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964160 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964160 parsl.executors.status_handling DEBUG: Launched block 134 on executor multi with job ID 16964160 parsl.executors.status_handling INFO: Allocated block ID 135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-135 2024-03-09 14:51:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2946 try 0 launched on executor multi with executor id 2947 parsl.dataflow.dflow INFO: Standard output for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/157f80ef-9795-49a6-b14d-e8179dc7fd8e_isrForCrosstalkSources_855402_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/157f80ef-9795-49a6-b14d-e8179dc7fd8e_isrForCrosstalkSources_855402_49.stderr parsl.dataflow.dflow DEBUG: Task 2947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2947 has memoization hash d89b54c641bc83aea430db92a9cc0d6e parsl.dataflow.memoization INFO: Task 2947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2947 try 0 launched on executor multi with executor id 2948 parsl.dataflow.dflow INFO: Standard output for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9dbabf65-ed80-4c6c-9851-0ea70618673e_isrForCrosstalkSources_855387_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9dbabf65-ed80-4c6c-9851-0ea70618673e_isrForCrosstalkSources_855387_9.stderr parsl.dataflow.dflow DEBUG: Task 2948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2948 has memoization hash ddfeb91b0b38d5b26ee7677d2150b28f parsl.dataflow.memoization INFO: Task 2948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2948 try 0 launched on executor multi with executor id 2949 parsl.dataflow.dflow INFO: Standard output for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8834c3b7-707b-4c8b-9370-0e3692ede66b_isrForCrosstalkSources_855446_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8834c3b7-707b-4c8b-9370-0e3692ede66b_isrForCrosstalkSources_855446_31.stderr parsl.dataflow.dflow DEBUG: Task 2949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2949 has memoization hash 1e64f9a121654a2d02a7696692132df5 parsl.dataflow.memoization INFO: Task 2949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2949 try 0 launched on executor multi with executor id 2950 parsl.dataflow.dflow INFO: Standard output for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ebb3823-6de7-483c-ab49-1af9ba3488ec_isrForCrosstalkSources_855442_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ebb3823-6de7-483c-ab49-1af9ba3488ec_isrForCrosstalkSources_855442_9.stderr parsl.dataflow.dflow DEBUG: Task 2950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2950 has memoization hash 185483fa71579cad8536a8ef44a08101 parsl.dataflow.memoization INFO: Task 2950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2950 try 0 launched on executor multi with executor id 2951 parsl.dataflow.dflow INFO: Standard output for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ecdd6c6e-6c2d-483e-8c6e-36ac6dac08cc_isrForCrosstalkSources_855449_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ecdd6c6e-6c2d-483e-8c6e-36ac6dac08cc_isrForCrosstalkSources_855449_36.stderr parsl.dataflow.dflow DEBUG: Task 2951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2951 has memoization hash 14512675b6ef6f1b75b154a3cde2fd65 parsl.dataflow.memoization INFO: Task 2951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2951 try 0 launched on executor multi with executor id 2952 parsl.dataflow.dflow INFO: Standard output for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0a6841e4-e8c2-426d-9694-4a5f4b7e378c_isrForCrosstalkSources_855425_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0a6841e4-e8c2-426d-9694-4a5f4b7e378c_isrForCrosstalkSources_855425_23.stderr parsl.dataflow.dflow DEBUG: Task 2952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2952 has memoization hash decc5c9fc4024026dc74922084c658b8 parsl.dataflow.memoization INFO: Task 2952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2952 try 0 launched on executor multi with executor id 2953 parsl.dataflow.dflow INFO: Standard output for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/779cfd83-f8f1-40b2-8632-3ef3813ec54f_isrForCrosstalkSources_855391_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/779cfd83-f8f1-40b2-8632-3ef3813ec54f_isrForCrosstalkSources_855391_12.stderr parsl.dataflow.dflow DEBUG: Task 2953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2953 has memoization hash 5ae2f34bfea5a3eda44fc921523e9a84 parsl.dataflow.memoization INFO: Task 2953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2953 try 0 launched on executor multi with executor id 2954 parsl.dataflow.dflow INFO: Standard output for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3e32847b-bdf0-4f1a-b1b9-7f0e649fd223_isrForCrosstalkSources_855453_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3e32847b-bdf0-4f1a-b1b9-7f0e649fd223_isrForCrosstalkSources_855453_54.stderr parsl.dataflow.dflow DEBUG: Task 2954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2954 has memoization hash c4b1edd54d04d44f26ae31e7d52c2169 parsl.dataflow.memoization INFO: Task 2954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2954 try 0 launched on executor multi with executor id 2955 parsl.dataflow.dflow INFO: Standard output for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/135e9e23-ff9d-4204-98a4-dcacfd683ae2_isrForCrosstalkSources_855455_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/135e9e23-ff9d-4204-98a4-dcacfd683ae2_isrForCrosstalkSources_855455_55.stderr parsl.dataflow.dflow DEBUG: Task 2955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2955 has memoization hash f57faeb51cbe0ac24928b9b462103e11 parsl.dataflow.memoization INFO: Task 2955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2955 try 0 launched on executor multi with executor id 2956 parsl.dataflow.dflow INFO: Standard output for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9b3b37f3-5bcb-49fc-b849-98679d4bfb02_isrForCrosstalkSources_855399_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9b3b37f3-5bcb-49fc-b849-98679d4bfb02_isrForCrosstalkSources_855399_10.stderr parsl.dataflow.dflow DEBUG: Task 2956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2956 has memoization hash b585222d6432239e7e4fe9429e0ee3ef parsl.dataflow.memoization INFO: Task 2956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2956 try 0 launched on executor multi with executor id 2957 parsl.dataflow.dflow INFO: Standard output for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d1b73dc2-16c7-4537-9aba-450cb1eca11e_isrForCrosstalkSources_855407_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d1b73dc2-16c7-4537-9aba-450cb1eca11e_isrForCrosstalkSources_855407_41.stderr parsl.dataflow.dflow DEBUG: Task 2957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2957 has memoization hash 6c472ac20a3f160dce154a385824d2df parsl.dataflow.memoization INFO: Task 2957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2957 try 0 launched on executor multi with executor id 2958 parsl.dataflow.dflow INFO: Standard output for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7ed1e183-1cac-4d8d-9a60-85c04d8ade87_isrForCrosstalkSources_855403_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7ed1e183-1cac-4d8d-9a60-85c04d8ade87_isrForCrosstalkSources_855403_14.stderr parsl.dataflow.dflow DEBUG: Task 2958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2958 has memoization hash eaafaaef37c6a82f153ff12996cfdfbc parsl.dataflow.memoization INFO: Task 2958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2958 try 0 launched on executor multi with executor id 2959 parsl.dataflow.dflow INFO: Standard output for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b760b8ca-9965-483d-aeaa-ff4252f4890e_isrForCrosstalkSources_855381_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b760b8ca-9965-483d-aeaa-ff4252f4890e_isrForCrosstalkSources_855381_56.stderr parsl.dataflow.dflow DEBUG: Task 2959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2959 has memoization hash 2ea3a3548a5e814d8ef376311c1d62a8 parsl.dataflow.memoization INFO: Task 2959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2959 try 0 launched on executor multi with executor id 2960 parsl.dataflow.dflow INFO: Standard output for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1b3a3a89-fea9-43ba-865f-2f26ec30ad52_isrForCrosstalkSources_855391_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1b3a3a89-fea9-43ba-865f-2f26ec30ad52_isrForCrosstalkSources_855391_49.stderr parsl.dataflow.dflow DEBUG: Task 2960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2960 has memoization hash ce179678849ad4f462f1d341f1089b5b parsl.dataflow.memoization INFO: Task 2960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2960 try 0 launched on executor multi with executor id 2961 parsl.dataflow.dflow INFO: Standard output for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc62d57c-94ca-44ba-8a75-274d2be52533_isrForCrosstalkSources_855421_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc62d57c-94ca-44ba-8a75-274d2be52533_isrForCrosstalkSources_855421_52.stderr parsl.dataflow.dflow DEBUG: Task 2961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2961 has memoization hash c0e430829c303141dcaf80a9118c3e8b parsl.dataflow.memoization INFO: Task 2961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2961 try 0 launched on executor multi with executor id 2962 parsl.dataflow.dflow INFO: Standard output for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/7160b75d-6518-4fe2-9635-d558c3414c16_isrForCrosstalkSources_855447_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/7160b75d-6518-4fe2-9635-d558c3414c16_isrForCrosstalkSources_855447_56.stderr parsl.dataflow.dflow DEBUG: Task 2962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2962 has memoization hash ebe8f93c8b7610d9a2d81c8161c1b85b parsl.dataflow.memoization INFO: Task 2962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2962 try 0 launched on executor multi with executor id 2963 parsl.dataflow.dflow INFO: Standard output for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/2d92fe30-906e-4d00-9e8f-9f3c81f2e33a_isrForCrosstalkSources_855381_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/2d92fe30-906e-4d00-9e8f-9f3c81f2e33a_isrForCrosstalkSources_855381_41.stderr parsl.dataflow.dflow DEBUG: Task 2963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2963 has memoization hash 8e47eae6c2cd72b45415429fc6ee2c48 parsl.dataflow.memoization INFO: Task 2963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2963 try 0 launched on executor multi with executor id 2964 parsl.dataflow.dflow INFO: Standard output for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/c99ba92a-ab84-4634-94f0-4368472f9975_isrForCrosstalkSources_855453_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/c99ba92a-ab84-4634-94f0-4368472f9975_isrForCrosstalkSources_855453_10.stderr parsl.dataflow.dflow DEBUG: Task 2964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2964 has memoization hash 3b69e7bae784c602fb659a2cb049b259 parsl.dataflow.memoization INFO: Task 2964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2964 try 0 launched on executor multi with executor id 2965 parsl.dataflow.dflow INFO: Standard output for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e73f48bf-78b4-4ebc-a4f0-acf180985535_isrForCrosstalkSources_855450_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e73f48bf-78b4-4ebc-a4f0-acf180985535_isrForCrosstalkSources_855450_61.stderr parsl.dataflow.dflow DEBUG: Task 2965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2965 has memoization hash d0b829077de0b285965cc4cc1c48c284 parsl.dataflow.memoization INFO: Task 2965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2965 try 0 launched on executor multi with executor id 2966 parsl.dataflow.dflow INFO: Standard output for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/72d6aec5-9905-4c70-b318-9a588c74e7c5_isrForCrosstalkSources_855413_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/72d6aec5-9905-4c70-b318-9a588c74e7c5_isrForCrosstalkSources_855413_42.stderr parsl.dataflow.dflow DEBUG: Task 2966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2966 has memoization hash a97c37f5b5b65ec79a25bfcf742fa249 parsl.dataflow.memoization INFO: Task 2966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2966 try 0 launched on executor multi with executor id 2967 parsl.dataflow.dflow INFO: Standard output for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8298ac3a-4b01-4fee-90e8-30bddd974bd6_isrForCrosstalkSources_855386_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8298ac3a-4b01-4fee-90e8-30bddd974bd6_isrForCrosstalkSources_855386_19.stderr parsl.dataflow.dflow DEBUG: Task 2967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2967 has memoization hash f04c80c9e92eae34bc7e5b8eb366fe05 parsl.dataflow.memoization INFO: Task 2967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2967 try 0 launched on executor multi with executor id 2968 parsl.dataflow.dflow INFO: Standard output for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5caf12e0-d2f9-4109-8837-aa02d946430e_isrForCrosstalkSources_855458_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5caf12e0-d2f9-4109-8837-aa02d946430e_isrForCrosstalkSources_855458_26.stderr parsl.dataflow.dflow DEBUG: Task 2968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2968 has memoization hash 366a10ba2701153538cd355d3a289769 parsl.dataflow.memoization INFO: Task 2968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2968 try 0 launched on executor multi with executor id 2969 parsl.dataflow.dflow INFO: Standard output for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c55c0281-c57a-4d8d-b831-0f4c6478dee0_isrForCrosstalkSources_855419_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c55c0281-c57a-4d8d-b831-0f4c6478dee0_isrForCrosstalkSources_855419_54.stderr parsl.dataflow.dflow DEBUG: Task 2969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2969 has memoization hash f55c49fda00ecc228452fa0160279246 parsl.dataflow.memoization INFO: Task 2969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2969 try 0 launched on executor multi with executor id 2970 parsl.dataflow.dflow INFO: Standard output for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a3e84d5a-8ee1-4d02-afc0-e794d86ce346_isrForCrosstalkSources_855393_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a3e84d5a-8ee1-4d02-afc0-e794d86ce346_isrForCrosstalkSources_855393_20.stderr parsl.dataflow.dflow DEBUG: Task 2970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2970 has memoization hash 12b193b2c7300e18df7622bdec009fd1 parsl.dataflow.memoization INFO: Task 2970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2970 try 0 launched on executor multi with executor id 2971 parsl.dataflow.dflow INFO: Standard output for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4770957c-70f2-459e-8279-20b97fc09e81_isrForCrosstalkSources_855409_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4770957c-70f2-459e-8279-20b97fc09e81_isrForCrosstalkSources_855409_11.stderr parsl.dataflow.dflow DEBUG: Task 2971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2971 has memoization hash ba4340abe471fd451f033bdae7d98a69 parsl.dataflow.memoization INFO: Task 2971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2971 try 0 launched on executor multi with executor id 2972 parsl.dataflow.dflow INFO: Standard output for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1e1e089d-ab1a-4ce4-bc91-077b2cbfe45a_isrForCrosstalkSources_855388_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1e1e089d-ab1a-4ce4-bc91-077b2cbfe45a_isrForCrosstalkSources_855388_23.stderr parsl.dataflow.dflow DEBUG: Task 2972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2972 has memoization hash f19024fca1923a417ed642922c479c0b parsl.dataflow.memoization INFO: Task 2972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:28 proc_lsst.multi:170 [INFO] [multi] job_id 16964161 proc_lsst.multi INFO: [multi] job_id 16964161 2024-03-09 14:51:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-09 14:51:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964161 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964161 parsl.executors.status_handling DEBUG: Launched block 135 on executor multi with job ID 16964161 parsl.executors.status_handling INFO: Allocated block ID 136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-136 2024-03-09 14:51:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-09 14:51:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2972 try 0 launched on executor multi with executor id 2973 parsl.dataflow.dflow INFO: Standard output for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/5ad4ca0d-90e6-4f43-b837-de4d493ad15e_isrForCrosstalkSources_855409_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/5ad4ca0d-90e6-4f43-b837-de4d493ad15e_isrForCrosstalkSources_855409_2.stderr parsl.dataflow.dflow DEBUG: Task 2973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2973 has memoization hash b55afbc971876c71ecab64e9025ed34b parsl.dataflow.memoization INFO: Task 2973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2973 try 0 launched on executor multi with executor id 2974 parsl.dataflow.dflow INFO: Standard output for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7126994c-f289-4405-b3ac-49e5cedf2b54_isrForCrosstalkSources_855422_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7126994c-f289-4405-b3ac-49e5cedf2b54_isrForCrosstalkSources_855422_33.stderr parsl.dataflow.dflow DEBUG: Task 2974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2974 has memoization hash 4c262ba0631f1d30df9a237484de20f5 parsl.dataflow.memoization INFO: Task 2974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2974 try 0 launched on executor multi with executor id 2975 parsl.dataflow.dflow INFO: Standard output for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6504951e-ec01-48af-8716-11ecc9d20617_isrForCrosstalkSources_855394_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6504951e-ec01-48af-8716-11ecc9d20617_isrForCrosstalkSources_855394_15.stderr parsl.dataflow.dflow DEBUG: Task 2975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2975 has memoization hash 26f719029d66513d9cdc05338ffefeca parsl.dataflow.memoization INFO: Task 2975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2975 try 0 launched on executor multi with executor id 2976 parsl.dataflow.dflow INFO: Standard output for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9ecb1418-23cf-4132-bbce-ba8f77abc1f4_isrForCrosstalkSources_855456_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9ecb1418-23cf-4132-bbce-ba8f77abc1f4_isrForCrosstalkSources_855456_17.stderr parsl.dataflow.dflow DEBUG: Task 2976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2976 has memoization hash f3adf83b8b4ed109ad1e3123464a0516 parsl.dataflow.memoization INFO: Task 2976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2976 try 0 launched on executor multi with executor id 2977 parsl.dataflow.dflow INFO: Standard output for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c4dff29b-9be6-4783-8ac7-798a25c20f64_isrForCrosstalkSources_855450_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c4dff29b-9be6-4783-8ac7-798a25c20f64_isrForCrosstalkSources_855450_35.stderr parsl.dataflow.dflow DEBUG: Task 2977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2977 has memoization hash 9d9e4317a964f6ec4624ef829092466b parsl.dataflow.memoization INFO: Task 2977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2977 try 0 launched on executor multi with executor id 2978 parsl.dataflow.dflow INFO: Standard output for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/54622395-6534-4852-8456-fd1028d0890c_isrForCrosstalkSources_855429_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/54622395-6534-4852-8456-fd1028d0890c_isrForCrosstalkSources_855429_33.stderr parsl.dataflow.dflow DEBUG: Task 2978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2978 has memoization hash 93faebb3426f12b3fbfdbcf0c8e6fc6f parsl.dataflow.memoization INFO: Task 2978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2978 try 0 launched on executor multi with executor id 2979 parsl.dataflow.dflow INFO: Standard output for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/1767a961-3366-4aff-af8c-d0d0409573be_isrForCrosstalkSources_855405_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/1767a961-3366-4aff-af8c-d0d0409573be_isrForCrosstalkSources_855405_62.stderr parsl.dataflow.dflow DEBUG: Task 2979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2979 has memoization hash 5171e8ddba465d8e5667ecf9547fc79a parsl.dataflow.memoization INFO: Task 2979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2979 try 0 launched on executor multi with executor id 2980 parsl.dataflow.dflow INFO: Standard output for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/75c96385-80d4-47b9-9b86-0f653e7e8ddc_isrForCrosstalkSources_855426_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/75c96385-80d4-47b9-9b86-0f653e7e8ddc_isrForCrosstalkSources_855426_3.stderr parsl.dataflow.dflow DEBUG: Task 2980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2980 has memoization hash 470ac77564bfdfa03612515dc8dfcb19 parsl.dataflow.memoization INFO: Task 2980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2980 try 0 launched on executor multi with executor id 2981 parsl.dataflow.dflow INFO: Standard output for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/49b81e15-1149-4fe0-aba4-bb33824f4caf_isrForCrosstalkSources_855384_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/49b81e15-1149-4fe0-aba4-bb33824f4caf_isrForCrosstalkSources_855384_14.stderr parsl.dataflow.dflow DEBUG: Task 2981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2981 has memoization hash ad328c45d33726a542d0deb99e7071dd parsl.dataflow.memoization INFO: Task 2981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2981 try 0 launched on executor multi with executor id 2982 parsl.dataflow.dflow INFO: Standard output for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/31249e03-035f-4469-a977-6d1b7579a871_isrForCrosstalkSources_855435_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/31249e03-035f-4469-a977-6d1b7579a871_isrForCrosstalkSources_855435_23.stderr parsl.dataflow.dflow DEBUG: Task 2982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2982 has memoization hash 27d19a841fb1b889298ef056dd9ea373 parsl.dataflow.memoization INFO: Task 2982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2982 try 0 launched on executor multi with executor id 2983 parsl.dataflow.dflow INFO: Standard output for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b87205ec-f445-43ff-8096-cbe6c5dc2ba4_isrForCrosstalkSources_855411_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b87205ec-f445-43ff-8096-cbe6c5dc2ba4_isrForCrosstalkSources_855411_39.stderr parsl.dataflow.dflow DEBUG: Task 2983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2983 has memoization hash d11dfdbc42a944131865cfb3f295e525 parsl.dataflow.memoization INFO: Task 2983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2983 try 0 launched on executor multi with executor id 2984 parsl.dataflow.dflow INFO: Standard output for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/410d8ba8-b803-4dbb-8bfd-c2b78a0dddf1_isrForCrosstalkSources_855437_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/410d8ba8-b803-4dbb-8bfd-c2b78a0dddf1_isrForCrosstalkSources_855437_51.stderr parsl.dataflow.dflow DEBUG: Task 2984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2984 has memoization hash 9021bc3745ff772e0adf581c9bf68dcd parsl.dataflow.memoization INFO: Task 2984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2984 try 0 launched on executor multi with executor id 2985 parsl.dataflow.dflow INFO: Standard output for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87284f1d-6afc-47c2-9c50-43cdc0d08dad_isrForCrosstalkSources_855391_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87284f1d-6afc-47c2-9c50-43cdc0d08dad_isrForCrosstalkSources_855391_61.stderr parsl.dataflow.dflow DEBUG: Task 2985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2985 has memoization hash 32ec218f1bf6b136a6b01ddca1103f09 parsl.dataflow.memoization INFO: Task 2985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2985 try 0 launched on executor multi with executor id 2986 parsl.dataflow.dflow INFO: Standard output for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/54c490ab-91ea-4372-9903-f9dbc5f0b040_isrForCrosstalkSources_855413_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/54c490ab-91ea-4372-9903-f9dbc5f0b040_isrForCrosstalkSources_855413_40.stderr parsl.dataflow.dflow DEBUG: Task 2986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2986 has memoization hash 1f486f6b3456ef79142ad7d64f2bc706 parsl.dataflow.memoization INFO: Task 2986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2986 try 0 launched on executor multi with executor id 2987 parsl.dataflow.dflow INFO: Standard output for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/5e87eeaf-fed3-446f-8593-78ee602f2856_isrForCrosstalkSources_855399_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/5e87eeaf-fed3-446f-8593-78ee602f2856_isrForCrosstalkSources_855399_48.stderr parsl.dataflow.dflow DEBUG: Task 2987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2987 has memoization hash ccb37dc46ce7e1ff4691046dada28f68 parsl.dataflow.memoization INFO: Task 2987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2987 try 0 launched on executor multi with executor id 2988 parsl.dataflow.dflow INFO: Standard output for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/88082731-6430-4387-85af-8dc36b53afbd_isrForCrosstalkSources_855432_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/88082731-6430-4387-85af-8dc36b53afbd_isrForCrosstalkSources_855432_54.stderr parsl.dataflow.dflow DEBUG: Task 2988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2988 has memoization hash 401308f2fd1c7e1f0e8070ea0c3b4269 parsl.dataflow.memoization INFO: Task 2988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2988 try 0 launched on executor multi with executor id 2989 parsl.dataflow.dflow INFO: Standard output for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f2474901-45d6-44f4-9ec8-816b27512454_isrForCrosstalkSources_855451_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f2474901-45d6-44f4-9ec8-816b27512454_isrForCrosstalkSources_855451_27.stderr parsl.dataflow.dflow DEBUG: Task 2989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2989 has memoization hash d43fe0159f3ff8e78a0b100e93d1d121 parsl.dataflow.memoization INFO: Task 2989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2989 try 0 launched on executor multi with executor id 2990 parsl.dataflow.dflow INFO: Standard output for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7d768068-8c89-4f8d-9bff-25bd6ac84fad_isrForCrosstalkSources_855406_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7d768068-8c89-4f8d-9bff-25bd6ac84fad_isrForCrosstalkSources_855406_33.stderr parsl.dataflow.dflow DEBUG: Task 2990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2990 has memoization hash 35d38a94df68f9c20877ec8d86ca7b42 parsl.dataflow.memoization INFO: Task 2990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2990 try 0 launched on executor multi with executor id 2991 parsl.dataflow.dflow INFO: Standard output for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/17c56097-a798-48d1-9d30-6aedff8a85af_isrForCrosstalkSources_855428_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/17c56097-a798-48d1-9d30-6aedff8a85af_isrForCrosstalkSources_855428_57.stderr parsl.dataflow.dflow DEBUG: Task 2991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2991 has memoization hash d8bfc583a9c6b8363268a74ebd019b04 parsl.dataflow.memoization INFO: Task 2991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2991 try 0 launched on executor multi with executor id 2992 parsl.dataflow.dflow INFO: Standard output for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f726129a-d77b-430b-b6e6-5756945fb3a9_isrForCrosstalkSources_855448_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f726129a-d77b-430b-b6e6-5756945fb3a9_isrForCrosstalkSources_855448_32.stderr parsl.dataflow.dflow DEBUG: Task 2992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2992 has memoization hash 9a9fb8a8cecdf5d51d7cf5eec1386ef4 parsl.dataflow.memoization INFO: Task 2992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2992 try 0 launched on executor multi with executor id 2993 parsl.dataflow.dflow INFO: Standard output for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/6c023183-91b4-4da7-bf5f-02214072def7_isrForCrosstalkSources_855396_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/6c023183-91b4-4da7-bf5f-02214072def7_isrForCrosstalkSources_855396_58.stderr parsl.dataflow.dflow DEBUG: Task 2993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2993 has memoization hash c8b7f188df7e1b444cc310875aa93c59 parsl.dataflow.memoization INFO: Task 2993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2993 try 0 launched on executor multi with executor id 2994 parsl.dataflow.dflow INFO: Standard output for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ba611f8-094b-4f3f-9755-b777c007d241_isrForCrosstalkSources_855427_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ba611f8-094b-4f3f-9755-b777c007d241_isrForCrosstalkSources_855427_46.stderr parsl.dataflow.dflow DEBUG: Task 2994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2994 has memoization hash 9a0fc5374d266f673edf1d38ca7d76fb parsl.dataflow.memoization INFO: Task 2994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2994 try 0 launched on executor multi with executor id 2995 parsl.dataflow.dflow INFO: Standard output for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d5bef3e7-218c-4904-8d04-ff77fd118408_isrForCrosstalkSources_855402_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d5bef3e7-218c-4904-8d04-ff77fd118408_isrForCrosstalkSources_855402_24.stderr parsl.dataflow.dflow DEBUG: Task 2995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2995 has memoization hash 65c575abbf9a9b2b5d81c48c4269ef17 parsl.dataflow.memoization INFO: Task 2995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2995 try 0 launched on executor multi with executor id 2996 parsl.dataflow.dflow INFO: Standard output for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/36bbd2b6-63f7-4ea4-b126-5b756e45d733_isrForCrosstalkSources_855379_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/36bbd2b6-63f7-4ea4-b126-5b756e45d733_isrForCrosstalkSources_855379_19.stderr parsl.dataflow.dflow DEBUG: Task 2996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2996 has memoization hash 2db9ba6fdb6d1d174edcb3cbd30d6e95 parsl.dataflow.memoization INFO: Task 2996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2996 try 0 launched on executor multi with executor id 2997 parsl.dataflow.dflow INFO: Standard output for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a1aa9702-1190-4398-8263-c680d5adaf7b_isrForCrosstalkSources_855404_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a1aa9702-1190-4398-8263-c680d5adaf7b_isrForCrosstalkSources_855404_37.stderr parsl.dataflow.dflow DEBUG: Task 2997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2997 has memoization hash 540e7e2ddde767f6274ecdb983eb25ab parsl.dataflow.memoization INFO: Task 2997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2997 try 0 launched on executor multi with executor id 2998 parsl.dataflow.dflow INFO: Standard output for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e473d6b0-e22e-4dd1-a0a8-f5b275175da0_isrForCrosstalkSources_855423_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e473d6b0-e22e-4dd1-a0a8-f5b275175da0_isrForCrosstalkSources_855423_13.stderr parsl.dataflow.dflow DEBUG: Task 2998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2998 has memoization hash c1f7daa23812b239b13b93c0137da651 parsl.dataflow.memoization INFO: Task 2998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2998 try 0 launched on executor multi with executor id 2999 parsl.dataflow.dflow INFO: Standard output for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/560a825b-1099-4b20-8164-9e8424f27361_isrForCrosstalkSources_855385_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/560a825b-1099-4b20-8164-9e8424f27361_isrForCrosstalkSources_855385_44.stderr parsl.dataflow.dflow DEBUG: Task 2999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2999 has memoization hash d583aa3698416dd369072ee843931a13 parsl.dataflow.memoization INFO: Task 2999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2999 try 0 launched on executor multi with executor id 3000 parsl.dataflow.dflow INFO: Standard output for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e51ca827-eeaf-4b4c-92a3-057d5190f1e0_isrForCrosstalkSources_855441_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e51ca827-eeaf-4b4c-92a3-057d5190f1e0_isrForCrosstalkSources_855441_47.stderr parsl.dataflow.dflow DEBUG: Task 3000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3000 has memoization hash 5240e2478a29ee9104bda4059a93c531 parsl.dataflow.memoization INFO: Task 3000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:29 proc_lsst.multi:170 [INFO] [multi] job_id 16964162 parsl.dataflow.dflow INFO: Parsl task 3000 try 0 launched on executor multi with executor id 3001 parsl.dataflow.dflow INFO: Standard output for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/598a075d-6945-43cf-9c42-c366b12eadec_isrForCrosstalkSources_855429_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/598a075d-6945-43cf-9c42-c366b12eadec_isrForCrosstalkSources_855429_44.stderr parsl.dataflow.dflow DEBUG: Task 3001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3001 has memoization hash e7d5a6fecb1f22a07b80e7294d5446c0 parsl.dataflow.memoization INFO: Task 3001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a65f2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3001 try 0 launched on executor multi with executor id 3002 parsl.dataflow.dflow INFO: Standard output for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e65b7292-de32-4a54-8f9b-2101861179a7_isrForCrosstalkSources_855418_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e65b7292-de32-4a54-8f9b-2101861179a7_isrForCrosstalkSources_855418_46.stderr parsl.dataflow.dflow DEBUG: Task 3002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3002 has memoization hash 08fe447f905058c7a22b71b699caea06 parsl.dataflow.memoization INFO: Task 3002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] job_id 16964162 2024-03-09 14:51:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-09 14:51:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964162 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964162 parsl.executors.status_handling DEBUG: Launched block 136 on executor multi with job ID 16964162 parsl.executors.status_handling INFO: Allocated block ID 137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-137 2024-03-09 14:51:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3002 try 0 launched on executor multi with executor id 3003 parsl.dataflow.dflow INFO: Standard output for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e95c16f4-2502-40d9-bcc3-1bdda882c587_isrForCrosstalkSources_855451_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e95c16f4-2502-40d9-bcc3-1bdda882c587_isrForCrosstalkSources_855451_53.stderr parsl.dataflow.dflow DEBUG: Task 3003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3003 has memoization hash aeff74643f4c5172d8e5d15dac88add5 parsl.dataflow.memoization INFO: Task 3003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3003 try 0 launched on executor multi with executor id 3004 parsl.dataflow.dflow INFO: Standard output for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0539a862-ffc1-4f8b-b9f8-1e628a685037_isrForCrosstalkSources_855386_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0539a862-ffc1-4f8b-b9f8-1e628a685037_isrForCrosstalkSources_855386_61.stderr parsl.dataflow.dflow DEBUG: Task 3004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3004 has memoization hash eb0bbd936f6b1eed8ec78ea4705b0541 parsl.dataflow.memoization INFO: Task 3004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3004 try 0 launched on executor multi with executor id 3005 parsl.dataflow.dflow INFO: Standard output for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ad78ceb9-ebed-4d73-b2b1-d81a0219e0d0_isrForCrosstalkSources_855392_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ad78ceb9-ebed-4d73-b2b1-d81a0219e0d0_isrForCrosstalkSources_855392_43.stderr parsl.dataflow.dflow DEBUG: Task 3005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3005 has memoization hash 478924f814f570344ea8f1385e9d1030 parsl.dataflow.memoization INFO: Task 3005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3005 try 0 launched on executor multi with executor id 3006 parsl.dataflow.dflow INFO: Standard output for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0db106a-e486-4b1a-8021-c69e5fcdf399_isrForCrosstalkSources_855388_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0db106a-e486-4b1a-8021-c69e5fcdf399_isrForCrosstalkSources_855388_19.stderr parsl.dataflow.dflow DEBUG: Task 3006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3006 has memoization hash 1c6931f18b7fd5d2e7823f0581429dd6 parsl.dataflow.memoization INFO: Task 3006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3006 try 0 launched on executor multi with executor id 3007 parsl.dataflow.dflow INFO: Standard output for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9f48430c-a8e3-43f1-860b-d96f56822cec_isrForCrosstalkSources_855403_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9f48430c-a8e3-43f1-860b-d96f56822cec_isrForCrosstalkSources_855403_43.stderr parsl.dataflow.dflow DEBUG: Task 3007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3007 has memoization hash 493f769e5452351a831a1dc4c064c6d1 parsl.dataflow.memoization INFO: Task 3007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3007 try 0 launched on executor multi with executor id 3008 parsl.dataflow.dflow INFO: Standard output for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/19731c85-63eb-4f80-86fa-fa50552bd097_isrForCrosstalkSources_855397_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/19731c85-63eb-4f80-86fa-fa50552bd097_isrForCrosstalkSources_855397_15.stderr parsl.dataflow.dflow DEBUG: Task 3008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3008 has memoization hash 134792514594d9b039b23d9450d9a2c8 parsl.dataflow.memoization INFO: Task 3008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3008 try 0 launched on executor multi with executor id 3009 parsl.dataflow.dflow INFO: Standard output for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6e1325de-38a4-453d-bce4-8093d76c2299_isrForCrosstalkSources_855429_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6e1325de-38a4-453d-bce4-8093d76c2299_isrForCrosstalkSources_855429_40.stderr parsl.dataflow.dflow DEBUG: Task 3009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3009 has memoization hash 3714875f82b4940488b971164b2a6dd5 parsl.dataflow.memoization INFO: Task 3009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3009 try 0 launched on executor multi with executor id 3010 parsl.dataflow.dflow INFO: Standard output for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a1ce4a50-3e33-4e73-b299-e1bd957858e1_isrForCrosstalkSources_855451_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a1ce4a50-3e33-4e73-b299-e1bd957858e1_isrForCrosstalkSources_855451_25.stderr parsl.dataflow.dflow DEBUG: Task 3010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3010 has memoization hash 42fbbe474386b2c0bcfb147b3ed2b4db parsl.dataflow.memoization INFO: Task 3010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3010 try 0 launched on executor multi with executor id 3011 parsl.dataflow.dflow INFO: Standard output for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/342966e1-c1b1-4f44-8976-c0468ed9d280_isrForCrosstalkSources_855457_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/342966e1-c1b1-4f44-8976-c0468ed9d280_isrForCrosstalkSources_855457_6.stderr parsl.dataflow.dflow DEBUG: Task 3011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3011 has memoization hash 56098997cadf41d4b08780d6fedc4b15 parsl.dataflow.memoization INFO: Task 3011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3011 try 0 launched on executor multi with executor id 3012 parsl.dataflow.dflow INFO: Standard output for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/15dfe727-2d57-47fd-b18a-d3f3421c8cc1_isrForCrosstalkSources_855397_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/15dfe727-2d57-47fd-b18a-d3f3421c8cc1_isrForCrosstalkSources_855397_60.stderr parsl.dataflow.dflow DEBUG: Task 3012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3012 has memoization hash 9f74cf77aa2d8f89f56369a6e6c4b4b6 parsl.dataflow.memoization INFO: Task 3012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3012 try 0 launched on executor multi with executor id 3013 parsl.dataflow.dflow INFO: Standard output for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/341e1c34-b787-4806-ad65-e070cc908c73_isrForCrosstalkSources_855459_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/341e1c34-b787-4806-ad65-e070cc908c73_isrForCrosstalkSources_855459_59.stderr parsl.dataflow.dflow DEBUG: Task 3013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3013 has memoization hash a5b6ca0ae327f85e2ae61c15cf33a9e4 parsl.dataflow.memoization INFO: Task 3013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3013 try 0 launched on executor multi with executor id 3014 parsl.dataflow.dflow INFO: Standard output for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/19bb55c9-d851-4454-9010-62bfda3b2a0c_isrForCrosstalkSources_855412_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/19bb55c9-d851-4454-9010-62bfda3b2a0c_isrForCrosstalkSources_855412_15.stderr parsl.dataflow.dflow DEBUG: Task 3014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3014 has memoization hash db5c5ebf83fb6f2ed33e0473d09228c6 parsl.dataflow.memoization INFO: Task 3014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3014 try 0 launched on executor multi with executor id 3015 parsl.dataflow.dflow INFO: Standard output for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/79bb7e94-f181-423a-8aac-89e2a4cebac4_isrForCrosstalkSources_855388_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/79bb7e94-f181-423a-8aac-89e2a4cebac4_isrForCrosstalkSources_855388_4.stderr parsl.dataflow.dflow DEBUG: Task 3015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3015 has memoization hash e0169be1623424f1e8596c9cbc5e0103 parsl.dataflow.memoization INFO: Task 3015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3015 try 0 launched on executor multi with executor id 3016 parsl.dataflow.dflow INFO: Standard output for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/993efff5-3c71-44fd-931a-5ccc92c524a8_isrForCrosstalkSources_855451_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/993efff5-3c71-44fd-931a-5ccc92c524a8_isrForCrosstalkSources_855451_46.stderr parsl.dataflow.dflow DEBUG: Task 3016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3016 has memoization hash 9886ae1ea83c30fffb5661306ac92a00 parsl.dataflow.memoization INFO: Task 3016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3016 try 0 launched on executor multi with executor id 3017 parsl.dataflow.dflow INFO: Standard output for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/75ac1539-149d-4131-8ea3-3b460ddc4313_isrForCrosstalkSources_855438_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/75ac1539-149d-4131-8ea3-3b460ddc4313_isrForCrosstalkSources_855438_58.stderr parsl.dataflow.dflow DEBUG: Task 3017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3017 has memoization hash 3b629752f76951a064445ac0f2f6b252 parsl.dataflow.memoization INFO: Task 3017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3017 try 0 launched on executor multi with executor id 3018 parsl.dataflow.dflow INFO: Standard output for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5ca4db1c-7c84-4d4c-b307-e149f6456243_isrForCrosstalkSources_855453_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5ca4db1c-7c84-4d4c-b307-e149f6456243_isrForCrosstalkSources_855453_43.stderr parsl.dataflow.dflow DEBUG: Task 3018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3018 has memoization hash f8b06653e8aa91552f11d0f59d06fd1c parsl.dataflow.memoization INFO: Task 3018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3018 try 0 launched on executor multi with executor id 3019 parsl.dataflow.dflow INFO: Standard output for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/35196dc1-5402-438d-bbbe-66324897304b_isrForCrosstalkSources_855425_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/35196dc1-5402-438d-bbbe-66324897304b_isrForCrosstalkSources_855425_19.stderr parsl.dataflow.dflow DEBUG: Task 3019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3019 has memoization hash 13166cd6d493476bd14d7b200ab0ce67 parsl.dataflow.memoization INFO: Task 3019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3019 try 0 launched on executor multi with executor id 3020 parsl.dataflow.dflow INFO: Standard output for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e253d5fd-3fc2-4bac-8c77-37c4c21c6f2b_isrForCrosstalkSources_855396_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e253d5fd-3fc2-4bac-8c77-37c4c21c6f2b_isrForCrosstalkSources_855396_34.stderr parsl.dataflow.dflow DEBUG: Task 3020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3020 has memoization hash a5984e63d1c791bacd7e185fd024c01d parsl.dataflow.memoization INFO: Task 3020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3020 try 0 launched on executor multi with executor id 3021 parsl.dataflow.dflow INFO: Standard output for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5ee77ab5-9f6b-4c30-bb51-2b96045a6b2f_isrForCrosstalkSources_855458_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5ee77ab5-9f6b-4c30-bb51-2b96045a6b2f_isrForCrosstalkSources_855458_33.stderr parsl.dataflow.dflow DEBUG: Task 3021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3021 has memoization hash 6e9c59af61053a400c0f64e139b47935 parsl.dataflow.memoization INFO: Task 3021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3021 try 0 launched on executor multi with executor id 3022 parsl.dataflow.dflow INFO: Standard output for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dc05e594-40e8-4b53-a411-53b3274671f9_isrForCrosstalkSources_855434_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dc05e594-40e8-4b53-a411-53b3274671f9_isrForCrosstalkSources_855434_47.stderr parsl.dataflow.dflow DEBUG: Task 3022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3022 has memoization hash 72e0642068784724ac890e2a06c21296 parsl.dataflow.memoization INFO: Task 3022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3022 try 0 launched on executor multi with executor id 3023 parsl.dataflow.dflow INFO: Standard output for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5d40fde9-54d7-4d92-be5e-cfa567d0602f_isrForCrosstalkSources_855400_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5d40fde9-54d7-4d92-be5e-cfa567d0602f_isrForCrosstalkSources_855400_32.stderr parsl.dataflow.dflow DEBUG: Task 3023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3023 has memoization hash 8314a4fb94aac76b54ee9e49b5a7b0ef parsl.dataflow.memoization INFO: Task 3023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3023 try 0 launched on executor multi with executor id 3024 parsl.dataflow.dflow INFO: Standard output for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4988558f-8e83-42a9-8cd4-501210093783_isrForCrosstalkSources_855451_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4988558f-8e83-42a9-8cd4-501210093783_isrForCrosstalkSources_855451_12.stderr parsl.dataflow.dflow DEBUG: Task 3024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3024 has memoization hash 035b387e3133fb2cbc0860a09ef563cb parsl.dataflow.memoization INFO: Task 3024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3024 try 0 launched on executor multi with executor id 3025 parsl.dataflow.dflow INFO: Standard output for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c0f3b3f1-7ab4-4675-93ea-115cff4d1907_isrForCrosstalkSources_855410_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c0f3b3f1-7ab4-4675-93ea-115cff4d1907_isrForCrosstalkSources_855410_51.stderr parsl.dataflow.dflow DEBUG: Task 3025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3025 has memoization hash 9d420aad593d6125897d42366bae2403 parsl.dataflow.memoization INFO: Task 3025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3025 try 0 launched on executor multi with executor id 3026 parsl.dataflow.dflow INFO: Standard output for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/048db880-43d0-4a01-a4d2-0478f1937cdd_isrForCrosstalkSources_855411_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/048db880-43d0-4a01-a4d2-0478f1937cdd_isrForCrosstalkSources_855411_20.stderr parsl.dataflow.dflow DEBUG: Task 3026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3026 has memoization hash 76882c1a92b68a7ff46cf8c579fa6c86 parsl.dataflow.memoization INFO: Task 3026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3026 try 0 launched on executor multi with executor id 3027 parsl.dataflow.dflow INFO: Standard output for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1b849368-eb3f-4781-8f19-867f251810d8_isrForCrosstalkSources_855417_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1b849368-eb3f-4781-8f19-867f251810d8_isrForCrosstalkSources_855417_41.stderr parsl.dataflow.dflow DEBUG: Task 3027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3027 has memoization hash e70fb28a48ea0a657c86f38380e36325 parsl.dataflow.memoization INFO: Task 3027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3027 try 0 launched on executor multi with executor id 3028 parsl.dataflow.dflow INFO: Standard output for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/32fe50a5-8f99-444a-bcba-67b0044daa99_isrForCrosstalkSources_855426_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/32fe50a5-8f99-444a-bcba-67b0044daa99_isrForCrosstalkSources_855426_30.stderr parsl.dataflow.dflow DEBUG: Task 3028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3028 has memoization hash 687c3f522a2aec79a4f29f0223c5e9db parsl.dataflow.memoization INFO: Task 3028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3028 try 0 launched on executor multi with executor id 3029 parsl.dataflow.dflow INFO: Standard output for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5d6117fe-a165-4372-8c7c-3870b78eb169_isrForCrosstalkSources_855428_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5d6117fe-a165-4372-8c7c-3870b78eb169_isrForCrosstalkSources_855428_44.stderr parsl.dataflow.dflow DEBUG: Task 3029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3029 has memoization hash 1c4a75f618117a751e6bb281906a2176 parsl.dataflow.memoization INFO: Task 3029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3029 try 0 launched on executor multi with executor id 3030 parsl.dataflow.dflow INFO: Standard output for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4793ef6f-3ccd-4526-bc62-c74d17f6a3d3_isrForCrosstalkSources_855415_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4793ef6f-3ccd-4526-bc62-c74d17f6a3d3_isrForCrosstalkSources_855415_25.stderr parsl.dataflow.dflow DEBUG: Task 3030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3030 has memoization hash bed110c612109faf78d14f3a7008bc79 parsl.dataflow.memoization INFO: Task 3030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3030 try 0 launched on executor multi with executor id 3031 parsl.dataflow.dflow INFO: Standard output for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/236ee426-c51d-499b-8097-ee1523ea8038_isrForCrosstalkSources_855389_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/236ee426-c51d-499b-8097-ee1523ea8038_isrForCrosstalkSources_855389_41.stderr parsl.dataflow.dflow DEBUG: Task 3031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3031 has memoization hash 1682c115e558b1e42a6000309a286eb4 parsl.dataflow.memoization INFO: Task 3031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da8ec5cf3fd873e71730003cf34a0c33 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3be627f4-d673-4b63-a510-034c1158ae8b_isrForCrosstalkSources_855397_34.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3be627f4-d673-4b63-a510-034c1158ae8b_isrForCrosstalkSources_855397_34.stderr parsl.dataflow.dflow INFO: Parsl task 3031 try 0 launched on executor multi with executor id 3032 parsl.dataflow.dflow INFO: Standard output for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f1f80d01-9acd-4169-bf75-58fcc53fc19a_isrForCrosstalkSources_855429_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f1f80d01-9acd-4169-bf75-58fcc53fc19a_isrForCrosstalkSources_855429_9.stderr 2024-03-09 14:51:29 proc_lsst.multi:170 [INFO] [multi] job_id 16964163 proc_lsst.multi INFO: [multi] job_id 16964163 2024-03-09 14:51:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-09 14:51:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964163 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964163 parsl.dataflow.dflow DEBUG: Task 3032 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 137 on executor multi with job ID 16964163 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling INFO: Allocated block ID 138 parsl.dataflow.dflow INFO: Task 3032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-138 2024-03-09 14:51:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-09 14:51:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow DEBUG: Task 3032 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3032 has memoization hash 2d8fa9e97fc0df771838b0507a07710d parsl.dataflow.memoization INFO: Task 3032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3032 try 0 launched on executor multi with executor id 3033 parsl.dataflow.dflow INFO: Standard output for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3ec1bdeb-e03e-46b0-bdc6-0a28ae329796_isrForCrosstalkSources_855440_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3ec1bdeb-e03e-46b0-bdc6-0a28ae329796_isrForCrosstalkSources_855440_5.stderr parsl.dataflow.dflow DEBUG: Task 3033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3033 has memoization hash 4888494a8f1fe0beb04eece98cd452cc parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization INFO: Task 3033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3033 try 0 launched on executor multi with executor id 3034 parsl.dataflow.dflow INFO: Standard output for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/51358ccd-b027-4dca-8094-e8aa54ea5ec4_isrForCrosstalkSources_855427_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/51358ccd-b027-4dca-8094-e8aa54ea5ec4_isrForCrosstalkSources_855427_49.stderr parsl.dataflow.dflow DEBUG: Task 3034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3034 has memoization hash f1da30869b5f1aa2d27c2da2d901a251 parsl.dataflow.memoization INFO: Task 3034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3034 try 0 launched on executor multi with executor id 3035 parsl.dataflow.dflow INFO: Standard output for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ec44760d-ab13-4d8d-a5b8-9cd2a70ae087_isrForCrosstalkSources_855424_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ec44760d-ab13-4d8d-a5b8-9cd2a70ae087_isrForCrosstalkSources_855424_30.stderr parsl.dataflow.dflow DEBUG: Task 3035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3035 has memoization hash b76c808f1383d8406332262573efc9ad parsl.dataflow.memoization INFO: Task 3035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3035 try 0 launched on executor multi with executor id 3036 parsl.dataflow.dflow INFO: Standard output for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/183ae7fc-dec9-44bb-972b-8f2bfffda74e_isrForCrosstalkSources_855452_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/183ae7fc-dec9-44bb-972b-8f2bfffda74e_isrForCrosstalkSources_855452_8.stderr parsl.dataflow.dflow DEBUG: Task 3036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3036 has memoization hash 02f876e308e4d72330a1c0a92c2e1f82 parsl.dataflow.memoization INFO: Task 3036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3036 try 0 launched on executor multi with executor id 3037 parsl.dataflow.dflow INFO: Standard output for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5338b710-1b6f-4810-aabf-1f55a9960c97_isrForCrosstalkSources_855418_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5338b710-1b6f-4810-aabf-1f55a9960c97_isrForCrosstalkSources_855418_27.stderr parsl.dataflow.dflow DEBUG: Task 3037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3037 has memoization hash baf8e453e233ffee0a3ea4bcd9c67283 parsl.dataflow.memoization INFO: Task 3037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3037 try 0 launched on executor multi with executor id 3038 parsl.dataflow.dflow INFO: Standard output for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/2816fc43-1de6-4532-bb22-5a3cd6fda399_isrForCrosstalkSources_855387_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/2816fc43-1de6-4532-bb22-5a3cd6fda399_isrForCrosstalkSources_855387_1.stderr parsl.dataflow.dflow DEBUG: Task 3038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3038 has memoization hash 4a1b94e2dc9d198cbb7fe1d2dfb99842 parsl.dataflow.memoization INFO: Task 3038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3038 try 0 launched on executor multi with executor id 3039 parsl.dataflow.dflow INFO: Standard output for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9b5291b9-dcd0-4016-befd-b99d76434c5e_isrForCrosstalkSources_855395_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9b5291b9-dcd0-4016-befd-b99d76434c5e_isrForCrosstalkSources_855395_2.stderr parsl.dataflow.dflow DEBUG: Task 3039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3039 has memoization hash 0b3cbd13f880d3100f77555e5375f005 parsl.dataflow.memoization INFO: Task 3039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3039 try 0 launched on executor multi with executor id 3040 parsl.dataflow.dflow INFO: Standard output for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5ffe6500-bf47-454b-9848-9ec1bc0ea2cc_isrForCrosstalkSources_855449_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5ffe6500-bf47-454b-9848-9ec1bc0ea2cc_isrForCrosstalkSources_855449_26.stderr parsl.dataflow.dflow DEBUG: Task 3040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3040 has memoization hash 97a3bd7d921b596ef169d12143721c3a parsl.dataflow.memoization INFO: Task 3040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61584a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0acdfeb00193ea9e99bb5ec6de950463 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/ce6b283e-443b-4500-b8da-c45e6dd861fb_isrForCrosstalkSources_855393_13.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/ce6b283e-443b-4500-b8da-c45e6dd861fb_isrForCrosstalkSources_855393_13.stderr parsl.dataflow.dflow INFO: Parsl task 3040 try 0 launched on executor multi with executor id 3041 parsl.dataflow.dflow INFO: Standard output for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/779162ff-1f82-45f4-8865-538f03369d81_isrForCrosstalkSources_855404_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/779162ff-1f82-45f4-8865-538f03369d81_isrForCrosstalkSources_855404_34.stderr parsl.dataflow.dflow DEBUG: Task 3041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3041 has memoization hash 3a9af6e12b21a5cc586c7bc8b1223bf2 parsl.dataflow.memoization INFO: Task 3041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3041 try 0 launched on executor multi with executor id 3042 parsl.dataflow.dflow INFO: Standard output for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e120e213-b551-4c37-a4e5-6d21d2746b5a_isrForCrosstalkSources_855379_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e120e213-b551-4c37-a4e5-6d21d2746b5a_isrForCrosstalkSources_855379_59.stderr parsl.dataflow.dflow DEBUG: Task 3042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3042 has memoization hash 077b75bd9b891cdaebc35fccc5317e0f parsl.dataflow.memoization INFO: Task 3042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3042 try 0 launched on executor multi with executor id 3043 parsl.dataflow.dflow INFO: Standard output for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9a2d515a-36a4-428c-9ff8-1916cf2738c8_isrForCrosstalkSources_855389_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9a2d515a-36a4-428c-9ff8-1916cf2738c8_isrForCrosstalkSources_855389_55.stderr parsl.dataflow.dflow DEBUG: Task 3043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3043 has memoization hash 61c686e91e588339c7a23e06a65c7180 parsl.dataflow.memoization INFO: Task 3043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3043 try 0 launched on executor multi with executor id 3044 parsl.dataflow.dflow INFO: Standard output for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b93a3721-abc7-4a7a-a79e-d3338b9ff956_isrForCrosstalkSources_855454_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b93a3721-abc7-4a7a-a79e-d3338b9ff956_isrForCrosstalkSources_855454_25.stderr parsl.dataflow.dflow DEBUG: Task 3044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3044 has memoization hash dec58745516050297b434317c1cfd4de parsl.dataflow.memoization INFO: Task 3044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61585e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3044 try 0 launched on executor multi with executor id 3045 parsl.dataflow.dflow INFO: Standard output for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/20cdfae8-1767-4971-b5f5-d36a449781cc_isrForCrosstalkSources_855397_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/20cdfae8-1767-4971-b5f5-d36a449781cc_isrForCrosstalkSources_855397_53.stderr parsl.dataflow.dflow DEBUG: Task 3045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3045 has memoization hash 7f22558f71b186273d766509a8c5ab51 parsl.dataflow.memoization INFO: Task 3045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3045 try 0 launched on executor multi with executor id 3046 parsl.dataflow.dflow INFO: Standard output for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a47e2378-1c64-4e59-a553-ad57df705a72_isrForCrosstalkSources_855432_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a47e2378-1c64-4e59-a553-ad57df705a72_isrForCrosstalkSources_855432_7.stderr parsl.dataflow.dflow DEBUG: Task 3046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3046 has memoization hash a1434c3cfb5eeab0482ed58e7c62bb04 parsl.dataflow.memoization INFO: Task 3046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3046 try 0 launched on executor multi with executor id 3047 parsl.dataflow.dflow INFO: Standard output for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/9d1cf79e-cb91-4a8b-9f55-7f42a0a278e6_isrForCrosstalkSources_855443_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/9d1cf79e-cb91-4a8b-9f55-7f42a0a278e6_isrForCrosstalkSources_855443_17.stderr parsl.dataflow.dflow DEBUG: Task 3047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3047 has memoization hash 4883b72aa04fcb7e2d919ae316d5eeff parsl.dataflow.memoization INFO: Task 3047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3047 try 0 launched on executor multi with executor id 3048 parsl.dataflow.dflow INFO: Standard output for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f8af0d8d-d344-4654-afbd-d4ad47259a9e_isrForCrosstalkSources_855428_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f8af0d8d-d344-4654-afbd-d4ad47259a9e_isrForCrosstalkSources_855428_27.stderr parsl.dataflow.dflow DEBUG: Task 3048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3048 has memoization hash 8c31349704d747ed29e29340857a1b55 parsl.dataflow.memoization INFO: Task 3048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61589a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3048 try 0 launched on executor multi with executor id 3049 parsl.dataflow.dflow INFO: Standard output for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf472d30-9774-4c55-a8c4-024494836aa0_isrForCrosstalkSources_855402_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf472d30-9774-4c55-a8c4-024494836aa0_isrForCrosstalkSources_855402_19.stderr parsl.dataflow.dflow DEBUG: Task 3049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3049 has memoization hash 19732185f3d05d45754b936e7207c94b parsl.dataflow.memoization INFO: Task 3049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61587c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3049 try 0 launched on executor multi with executor id 3050 parsl.dataflow.dflow INFO: Standard output for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5202251c-026f-4cd7-8327-541bdc12e6b1_isrForCrosstalkSources_855434_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5202251c-026f-4cd7-8327-541bdc12e6b1_isrForCrosstalkSources_855434_31.stderr parsl.dataflow.dflow DEBUG: Task 3050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3050 has memoization hash d43e07ea208b3a553e1f4095b391d7b5 parsl.dataflow.memoization INFO: Task 3050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3050 try 0 launched on executor multi with executor id 3051 parsl.dataflow.dflow INFO: Standard output for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/4d8dcda9-088a-4bf9-aa3c-6648e07a6da7_isrForCrosstalkSources_855395_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/4d8dcda9-088a-4bf9-aa3c-6648e07a6da7_isrForCrosstalkSources_855395_55.stderr parsl.dataflow.dflow DEBUG: Task 3051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3051 has memoization hash b18feb79c3407d9699ad223948d81520 parsl.dataflow.memoization INFO: Task 3051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3051 try 0 launched on executor multi with executor id 3052 parsl.dataflow.dflow INFO: Standard output for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/20e16015-8539-462d-b148-acda03081c1d_isrForCrosstalkSources_855410_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/20e16015-8539-462d-b148-acda03081c1d_isrForCrosstalkSources_855410_55.stderr parsl.dataflow.dflow DEBUG: Task 3052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3052 has memoization hash dd6a707cac2f394d0a496a314c8aacc5 parsl.dataflow.memoization INFO: Task 3052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3052 try 0 launched on executor multi with executor id 3053 parsl.dataflow.dflow INFO: Standard output for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/757d2580-ddbf-424d-b5ae-58398da9bb07_isrForCrosstalkSources_855440_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/757d2580-ddbf-424d-b5ae-58398da9bb07_isrForCrosstalkSources_855440_36.stderr parsl.dataflow.dflow DEBUG: Task 3053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3053 has memoization hash b482056d6c386403c12e6c8881caec3c parsl.dataflow.memoization INFO: Task 3053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3053 try 0 launched on executor multi with executor id 3054 parsl.dataflow.dflow INFO: Standard output for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c562df56-d135-41bd-bff1-0c046226bf7a_isrForCrosstalkSources_855450_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c562df56-d135-41bd-bff1-0c046226bf7a_isrForCrosstalkSources_855450_47.stderr parsl.dataflow.dflow DEBUG: Task 3054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3054 has memoization hash f6143a2fcc8ddca9dfcb872b3d15a523 parsl.dataflow.memoization INFO: Task 3054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3054 try 0 launched on executor multi with executor id 3055 parsl.dataflow.dflow INFO: Standard output for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/1621226e-76d6-4021-b227-e953bf774c88_isrForCrosstalkSources_855418_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/1621226e-76d6-4021-b227-e953bf774c88_isrForCrosstalkSources_855418_43.stderr parsl.dataflow.dflow DEBUG: Task 3055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3055 has memoization hash ecfeb9ee6a9bd4e604fc76be21733efc parsl.dataflow.memoization INFO: Task 3055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:30 proc_lsst.multi:170 [INFO] [multi] job_id 16964164 proc_lsst.multi INFO: [multi] job_id 16964164 2024-03-09 14:51:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-09 14:51:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964164 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964164 parsl.executors.status_handling DEBUG: Launched block 138 on executor multi with job ID 16964164 parsl.executors.status_handling INFO: Allocated block ID 139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-139 2024-03-09 14:51:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 2024-03-09 14:51:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-09 14:51:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3055 try 0 launched on executor multi with executor id 3056 parsl.dataflow.dflow INFO: Standard output for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/44cf7bce-8780-4f5e-b40c-44659469529e_isrForCrosstalkSources_855451_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/44cf7bce-8780-4f5e-b40c-44659469529e_isrForCrosstalkSources_855451_32.stderr parsl.dataflow.dflow DEBUG: Task 3056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3056 has memoization hash ccd5e261abefcc579b67306073c3e713 parsl.dataflow.memoization INFO: Task 3056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 658372388d90bf0ad5d67fce650f6df3 with result from task 416 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3056 try 0 launched on executor multi with executor id 3057 parsl.dataflow.dflow INFO: Standard output for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/88aeb0b9-275b-490d-9ff0-f7d9cd8e0518_isrForCrosstalkSources_855443_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/88aeb0b9-275b-490d-9ff0-f7d9cd8e0518_isrForCrosstalkSources_855443_12.stderr parsl.dataflow.dflow DEBUG: Task 3057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3057 has memoization hash 1ff61c0c2a1f7de2baf57bf301dd4d64 parsl.dataflow.memoization INFO: Task 3057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61580e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4e6ac140-1dd7-47be-bc69-af348b0b4c26_isrForCrosstalkSources_855408_7.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4e6ac140-1dd7-47be-bc69-af348b0b4c26_isrForCrosstalkSources_855408_7.stderr parsl.dataflow.dflow INFO: Parsl task 3057 try 0 launched on executor multi with executor id 3058 parsl.dataflow.dflow INFO: Standard output for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/085d2b09-5d4f-4d03-ba6c-2ede309fe595_isrForCrosstalkSources_855381_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/085d2b09-5d4f-4d03-ba6c-2ede309fe595_isrForCrosstalkSources_855381_20.stderr parsl.dataflow.dflow DEBUG: Task 3058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3058 has memoization hash 4c80c1e759e8c78b7297ab47598195c9 parsl.dataflow.memoization INFO: Task 3058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3058 try 0 launched on executor multi with executor id 3059 parsl.dataflow.dflow INFO: Standard output for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4a6ff96c-f4e2-42d5-9917-58a94eb9b6f6_isrForCrosstalkSources_855451_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4a6ff96c-f4e2-42d5-9917-58a94eb9b6f6_isrForCrosstalkSources_855451_31.stderr parsl.dataflow.dflow DEBUG: Task 3059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3059 has memoization hash 316353f69c8160959bc958f4044d74f8 parsl.dataflow.memoization INFO: Task 3059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61591c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3059 try 0 launched on executor multi with executor id 3060 parsl.dataflow.dflow INFO: Standard output for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/31bcb7cb-a532-428e-a0ac-38140ee6b933_isrForCrosstalkSources_855413_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/31bcb7cb-a532-428e-a0ac-38140ee6b933_isrForCrosstalkSources_855413_60.stderr parsl.dataflow.dflow DEBUG: Task 3060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3060 has memoization hash 07acf34286f5bd5ef236af8a2f7872a8 parsl.dataflow.memoization INFO: Task 3060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3060 try 0 launched on executor multi with executor id 3061 parsl.dataflow.dflow INFO: Standard output for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ba79794e-922a-445d-b231-d5ecb0b92aa3_isrForCrosstalkSources_855439_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ba79794e-922a-445d-b231-d5ecb0b92aa3_isrForCrosstalkSources_855439_60.stderr parsl.dataflow.dflow DEBUG: Task 3061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3061 has memoization hash 2cbf27ef306c69786bfc3c799b3e9548 parsl.dataflow.memoization INFO: Task 3061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3061 try 0 launched on executor multi with executor id 3062 parsl.dataflow.dflow INFO: Standard output for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f7f152dd-d75c-48da-9aca-be59759c86ad_isrForCrosstalkSources_855418_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f7f152dd-d75c-48da-9aca-be59759c86ad_isrForCrosstalkSources_855418_7.stderr parsl.dataflow.dflow DEBUG: Task 3062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3062 has memoization hash af0c02a89b4a8d56a766af7d3541a4ce parsl.dataflow.memoization INFO: Task 3062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3062 try 0 launched on executor multi with executor id 3063 parsl.dataflow.dflow INFO: Standard output for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/25759866-53ba-431e-95b2-fcab8520b4c4_isrForCrosstalkSources_855396_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/25759866-53ba-431e-95b2-fcab8520b4c4_isrForCrosstalkSources_855396_29.stderr parsl.dataflow.dflow DEBUG: Task 3063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3063 has memoization hash 4f454f0c30117585e7c5ca6109faf1c1 parsl.dataflow.memoization INFO: Task 3063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3063 try 0 launched on executor multi with executor id 3064 parsl.dataflow.dflow INFO: Standard output for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e55978a3-318d-40da-8ff1-00ea563446b8_isrForCrosstalkSources_855386_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e55978a3-318d-40da-8ff1-00ea563446b8_isrForCrosstalkSources_855386_4.stderr parsl.dataflow.dflow DEBUG: Task 3064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3064 has memoization hash 7127f9304e368d798ec036895fa22aa3 parsl.dataflow.memoization INFO: Task 3064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3064 try 0 launched on executor multi with executor id 3065 parsl.dataflow.dflow INFO: Standard output for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/06a5168d-3530-4df8-8b80-2caf2619de0d_isrForCrosstalkSources_855391_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/06a5168d-3530-4df8-8b80-2caf2619de0d_isrForCrosstalkSources_855391_9.stderr parsl.dataflow.dflow DEBUG: Task 3065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3065 has memoization hash 721c9ab2a2f0c1d3f5e77b2d321dcc57 parsl.dataflow.memoization INFO: Task 3065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61593a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3065 try 0 launched on executor multi with executor id 3066 parsl.dataflow.dflow INFO: Standard output for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/87ad0db7-28d1-4fa1-aecb-19d94ae4d0a5_isrForCrosstalkSources_855392_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/87ad0db7-28d1-4fa1-aecb-19d94ae4d0a5_isrForCrosstalkSources_855392_31.stderr parsl.dataflow.dflow DEBUG: Task 3066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3066 has memoization hash 5f825062502028a64eec0aca6120e314 parsl.dataflow.memoization INFO: Task 3066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61594e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3066 try 0 launched on executor multi with executor id 3067 parsl.dataflow.dflow INFO: Standard output for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/adf30d3b-a151-4069-83cd-4f214929ca62_isrForCrosstalkSources_855381_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/adf30d3b-a151-4069-83cd-4f214929ca62_isrForCrosstalkSources_855381_32.stderr parsl.dataflow.dflow DEBUG: Task 3067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3067 has memoization hash 798eb9fdf0f1acdfa78b75325684faa7 parsl.dataflow.memoization INFO: Task 3067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3067 try 0 launched on executor multi with executor id 3068 parsl.dataflow.dflow INFO: Standard output for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/471a54e4-df06-4159-8a2f-b9ef665d4306_isrForCrosstalkSources_855437_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/471a54e4-df06-4159-8a2f-b9ef665d4306_isrForCrosstalkSources_855437_39.stderr parsl.dataflow.dflow DEBUG: Task 3068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3068 has memoization hash b61d80b3cc27513588f3447dff54b380 parsl.dataflow.memoization INFO: Task 3068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61596c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3068 try 0 launched on executor multi with executor id 3069 parsl.dataflow.dflow INFO: Standard output for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/84d17b0a-5198-4c5e-8b78-48d3f6946ad4_isrForCrosstalkSources_855387_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/84d17b0a-5198-4c5e-8b78-48d3f6946ad4_isrForCrosstalkSources_855387_31.stderr parsl.dataflow.dflow DEBUG: Task 3069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3069 has memoization hash 8c59658688345181fb58f281d8b505ce parsl.dataflow.memoization INFO: Task 3069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3069 try 0 launched on executor multi with executor id 3070 parsl.dataflow.dflow INFO: Standard output for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6b63290a-2bf4-446e-9897-37fbe00aaa5a_isrForCrosstalkSources_855383_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6b63290a-2bf4-446e-9897-37fbe00aaa5a_isrForCrosstalkSources_855383_23.stderr parsl.dataflow.dflow DEBUG: Task 3070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3070 has memoization hash 9b929df667f65b82a825dc4f92e0d6bb parsl.dataflow.memoization INFO: Task 3070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3070 try 0 launched on executor multi with executor id 3071 parsl.dataflow.dflow INFO: Standard output for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/48423b36-9c01-42e5-853c-09c9d2bc36b3_isrForCrosstalkSources_855446_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/48423b36-9c01-42e5-853c-09c9d2bc36b3_isrForCrosstalkSources_855446_8.stderr parsl.dataflow.dflow DEBUG: Task 3071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3071 has memoization hash a6f156186e130308dfe8285e76cfe7b5 parsl.dataflow.memoization INFO: Task 3071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3071 try 0 launched on executor multi with executor id 3072 parsl.dataflow.dflow INFO: Standard output for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ed412f6c-d6eb-4d6f-9961-7de0b7b0e23a_isrForCrosstalkSources_855392_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ed412f6c-d6eb-4d6f-9961-7de0b7b0e23a_isrForCrosstalkSources_855392_27.stderr parsl.dataflow.dflow DEBUG: Task 3072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3072 has memoization hash d9d72daa29c16a8e29cc5c2574543b6c parsl.dataflow.memoization INFO: Task 3072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3072 try 0 launched on executor multi with executor id 3073 parsl.dataflow.dflow INFO: Standard output for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ca322b92-a0df-4caa-9719-1ed2a6a6e86d_isrForCrosstalkSources_855459_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ca322b92-a0df-4caa-9719-1ed2a6a6e86d_isrForCrosstalkSources_855459_37.stderr parsl.dataflow.dflow DEBUG: Task 3073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3073 has memoization hash e1fdea2e7a497d77356e3893d559dd63 parsl.dataflow.memoization INFO: Task 3073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61599e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:31 proc_lsst.multi:170 [INFO] [multi] job_id 16964165 proc_lsst.multi INFO: [multi] job_id 16964165 2024-03-09 14:51:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-09 14:51:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964165 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964165 parsl.executors.status_handling DEBUG: Launched block 139 on executor multi with job ID 16964165 parsl.executors.status_handling INFO: Allocated block ID 140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-140 2024-03-09 14:51:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3073 try 0 launched on executor multi with executor id 3074 parsl.dataflow.dflow INFO: Standard output for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/30e8f903-1ab0-4a10-bc71-636420bbfd65_isrForCrosstalkSources_855411_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/30e8f903-1ab0-4a10-bc71-636420bbfd65_isrForCrosstalkSources_855411_7.stderr parsl.dataflow.dflow DEBUG: Task 3074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3074 has memoization hash 36eca2d99b24aa2b15a8da2bbd5dc523 parsl.dataflow.memoization INFO: Task 3074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61598a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3074 try 0 launched on executor multi with executor id 3075 parsl.dataflow.dflow INFO: Standard output for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f8cad71-3217-45c5-80b2-d16a60439fbd_isrForCrosstalkSources_855393_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f8cad71-3217-45c5-80b2-d16a60439fbd_isrForCrosstalkSources_855393_8.stderr parsl.dataflow.dflow DEBUG: Task 3075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3075 has memoization hash 894db393182c96af5510c4d4c945ee15 parsl.dataflow.memoization INFO: Task 3075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3075 try 0 launched on executor multi with executor id 3076 parsl.dataflow.dflow INFO: Standard output for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5c33899f-a189-4c88-b193-7c678aee69b1_isrForCrosstalkSources_855445_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5c33899f-a189-4c88-b193-7c678aee69b1_isrForCrosstalkSources_855445_56.stderr parsl.dataflow.dflow DEBUG: Task 3076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3076 has memoization hash 82e84746c866af702b41e3eaed1cebb1 parsl.dataflow.memoization INFO: Task 3076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3076 try 0 launched on executor multi with executor id 3077 parsl.dataflow.dflow INFO: Standard output for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ffe49e53-180a-4468-a53d-b76023572a89_isrForCrosstalkSources_855394_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ffe49e53-180a-4468-a53d-b76023572a89_isrForCrosstalkSources_855394_56.stderr parsl.dataflow.dflow DEBUG: Task 3077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3077 has memoization hash 9d62e8968d901d97a4011e9399bbce9c parsl.dataflow.memoization INFO: Task 3077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3077 try 0 launched on executor multi with executor id 3078 parsl.dataflow.dflow INFO: Standard output for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6c5da835-7bd4-4c98-b31a-b24e4fc46930_isrForCrosstalkSources_855412_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6c5da835-7bd4-4c98-b31a-b24e4fc46930_isrForCrosstalkSources_855412_48.stderr parsl.dataflow.dflow DEBUG: Task 3078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3078 has memoization hash aafc460dbfee19168a5537c7d9181b5c parsl.dataflow.memoization INFO: Task 3078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3078 try 0 launched on executor multi with executor id 3079 parsl.dataflow.dflow INFO: Standard output for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5fb5ab6e-c927-42e8-aa3f-9d941c2cb6b4_isrForCrosstalkSources_855403_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5fb5ab6e-c927-42e8-aa3f-9d941c2cb6b4_isrForCrosstalkSources_855403_23.stderr parsl.dataflow.dflow DEBUG: Task 3079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3079 has memoization hash df9cd2f47032b88bf41b35b2ab4ebdcc parsl.dataflow.memoization INFO: Task 3079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3079 try 0 launched on executor multi with executor id 3080 parsl.dataflow.dflow INFO: Standard output for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9e4f6810-a8ee-41d9-bcf0-5d2fdd213f2a_isrForCrosstalkSources_855423_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9e4f6810-a8ee-41d9-bcf0-5d2fdd213f2a_isrForCrosstalkSources_855423_32.stderr parsl.dataflow.dflow DEBUG: Task 3080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3080 has memoization hash 51e2487cc4889253f2e32fe8bb7691a4 parsl.dataflow.memoization INFO: Task 3080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3080 try 0 launched on executor multi with executor id 3081 parsl.dataflow.dflow INFO: Standard output for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/81c2782a-3b54-4a14-94ea-73235adf7d5f_isrForCrosstalkSources_855412_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/81c2782a-3b54-4a14-94ea-73235adf7d5f_isrForCrosstalkSources_855412_44.stderr parsl.dataflow.dflow DEBUG: Task 3081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3081 has memoization hash 69c724974e24ea43d0c72850a6f8505c parsl.dataflow.memoization INFO: Task 3081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3081 try 0 launched on executor multi with executor id 3082 parsl.dataflow.dflow INFO: Standard output for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c354db95-7e43-4db6-b10e-c746fb92e03f_isrForCrosstalkSources_855399_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c354db95-7e43-4db6-b10e-c746fb92e03f_isrForCrosstalkSources_855399_54.stderr parsl.dataflow.dflow DEBUG: Task 3082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3082 has memoization hash a5e8977e391198cd102fd512fe359666 parsl.dataflow.memoization INFO: Task 3082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3082 try 0 launched on executor multi with executor id 3083 parsl.dataflow.dflow INFO: Standard output for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0177d9ca-3c8c-4d9b-90d7-55c8acf0e14e_isrForCrosstalkSources_855411_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0177d9ca-3c8c-4d9b-90d7-55c8acf0e14e_isrForCrosstalkSources_855411_44.stderr parsl.dataflow.dflow DEBUG: Task 3083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3083 has memoization hash ce5d597799335eac34fe14a6c705a9e8 parsl.dataflow.memoization INFO: Task 3083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3083 try 0 launched on executor multi with executor id 3084 parsl.dataflow.dflow INFO: Standard output for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b7d8d590-f1ef-4696-a605-a3c05165194c_isrForCrosstalkSources_855389_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b7d8d590-f1ef-4696-a605-a3c05165194c_isrForCrosstalkSources_855389_50.stderr parsl.dataflow.dflow DEBUG: Task 3084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3084 has memoization hash 80f33f171471d93889b24affea5839d1 parsl.dataflow.memoization INFO: Task 3084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c6035ec823e2428e07d625c4efa0ee7 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dde47ee4-51bd-429e-9cdd-1d6eafe772e9_isrForCrosstalkSources_855458_32.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dde47ee4-51bd-429e-9cdd-1d6eafe772e9_isrForCrosstalkSources_855458_32.stderr parsl.dataflow.dflow INFO: Parsl task 3084 try 0 launched on executor multi with executor id 3085 parsl.dataflow.dflow INFO: Standard output for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8224a843-7255-43dc-b340-4d9ff7832f2a_isrForCrosstalkSources_855441_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8224a843-7255-43dc-b340-4d9ff7832f2a_isrForCrosstalkSources_855441_31.stderr parsl.dataflow.dflow DEBUG: Task 3085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3085 has memoization hash 25449e27ac5b46f4dfc2d329bbec33a7 parsl.dataflow.memoization INFO: Task 3085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3085 try 0 launched on executor multi with executor id 3086 parsl.dataflow.dflow INFO: Standard output for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fca48b90-8f7f-46df-a507-8110f93c16f6_isrForCrosstalkSources_855415_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fca48b90-8f7f-46df-a507-8110f93c16f6_isrForCrosstalkSources_855415_22.stderr parsl.dataflow.dflow DEBUG: Task 3086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3086 has memoization hash 7e507bcef407de3a88a150e9082874f4 parsl.dataflow.memoization INFO: Task 3086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3086 try 0 launched on executor multi with executor id 3087 parsl.dataflow.dflow INFO: Standard output for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/941643e5-5a88-408d-b8bc-0768e327c9a3_isrForCrosstalkSources_855404_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/941643e5-5a88-408d-b8bc-0768e327c9a3_isrForCrosstalkSources_855404_2.stderr parsl.dataflow.dflow DEBUG: Task 3087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3087 has memoization hash 951431e4a1480081b1e95245a47193ba parsl.dataflow.memoization INFO: Task 3087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3087 try 0 launched on executor multi with executor id 3088 parsl.dataflow.dflow INFO: Standard output for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6728434b-2f46-4b73-bb5b-0325cb56f27f_isrForCrosstalkSources_855420_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6728434b-2f46-4b73-bb5b-0325cb56f27f_isrForCrosstalkSources_855420_18.stderr parsl.dataflow.dflow DEBUG: Task 3088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3088 has memoization hash b5a34036dc0aca3dd2f8b622915df025 parsl.dataflow.memoization INFO: Task 3088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3088 try 0 launched on executor multi with executor id 3089 parsl.dataflow.dflow INFO: Standard output for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6dec1bc1-2edc-4447-9a0a-48d6c8452bde_isrForCrosstalkSources_855431_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6dec1bc1-2edc-4447-9a0a-48d6c8452bde_isrForCrosstalkSources_855431_47.stderr parsl.dataflow.dflow DEBUG: Task 3089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3089 has memoization hash 3f8f86122861c11d8e0a51279474e2db parsl.dataflow.memoization INFO: Task 3089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3089 try 0 launched on executor multi with executor id 3090 parsl.dataflow.dflow INFO: Standard output for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/93ff4fb1-251c-4e37-92fb-efa06e745c59_isrForCrosstalkSources_855435_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/93ff4fb1-251c-4e37-92fb-efa06e745c59_isrForCrosstalkSources_855435_25.stderr parsl.dataflow.dflow DEBUG: Task 3090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3090 has memoization hash 0a1ccb43a01662e1e8c40d8c6500a43a parsl.dataflow.memoization INFO: Task 3090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3090 try 0 launched on executor multi with executor id 3091 parsl.dataflow.dflow INFO: Standard output for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/47e26ca8-770a-4650-acc9-502b60959bb5_isrForCrosstalkSources_855394_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/47e26ca8-770a-4650-acc9-502b60959bb5_isrForCrosstalkSources_855394_54.stderr parsl.dataflow.dflow DEBUG: Task 3091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3091 has memoization hash aa1a670d0973a8fef8e72348e1f43dd5 parsl.dataflow.memoization INFO: Task 3091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3091 try 0 launched on executor multi with executor id 3092 parsl.dataflow.dflow INFO: Standard output for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0c4a138a-7943-4e98-abe0-9ae4e814d987_isrForCrosstalkSources_855388_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0c4a138a-7943-4e98-abe0-9ae4e814d987_isrForCrosstalkSources_855388_15.stderr parsl.dataflow.dflow DEBUG: Task 3092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3092 has memoization hash db3e9e60ced56ca3fbb605bd92528630 parsl.dataflow.memoization INFO: Task 3092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3092 try 0 launched on executor multi with executor id 3093 parsl.dataflow.dflow INFO: Standard output for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1653b9c5-0c45-4f66-b779-3c240442d317_isrForCrosstalkSources_855388_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1653b9c5-0c45-4f66-b779-3c240442d317_isrForCrosstalkSources_855388_34.stderr parsl.dataflow.dflow DEBUG: Task 3093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3093 has memoization hash ad274cd1edeff8d10a795886c379bccd parsl.dataflow.memoization INFO: Task 3093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3093 try 0 launched on executor multi with executor id 3094 parsl.dataflow.dflow INFO: Standard output for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7dea6153-4a94-470a-bb03-68a9696015e6_isrForCrosstalkSources_855414_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7dea6153-4a94-470a-bb03-68a9696015e6_isrForCrosstalkSources_855414_25.stderr parsl.dataflow.dflow DEBUG: Task 3094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3094 has memoization hash 6b4d97aec9a5a989c37d6f24d5e70a6d parsl.dataflow.memoization INFO: Task 3094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3094 try 0 launched on executor multi with executor id 3095 parsl.dataflow.dflow INFO: Standard output for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ee233356-fabe-4df0-8587-63fbfc8d5e9e_isrForCrosstalkSources_855405_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ee233356-fabe-4df0-8587-63fbfc8d5e9e_isrForCrosstalkSources_855405_6.stderr parsl.dataflow.dflow DEBUG: Task 3095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3095 has memoization hash 98d3635c70d5234da7abdbddca2ebce9 parsl.dataflow.memoization INFO: Task 3095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3095 try 0 launched on executor multi with executor id 3096 parsl.dataflow.dflow INFO: Standard output for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1089286f-1ac3-4cd9-a4fa-9f6fe455678f_isrForCrosstalkSources_855421_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1089286f-1ac3-4cd9-a4fa-9f6fe455678f_isrForCrosstalkSources_855421_42.stderr parsl.dataflow.dflow DEBUG: Task 3096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3096 has memoization hash 8f15bf538e3e49f7b857ce179f9c12ab parsl.dataflow.memoization INFO: Task 3096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3096 try 0 launched on executor multi with executor id 3097 parsl.dataflow.dflow INFO: Standard output for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8a98c94f-9fcb-4c23-b597-7ac875066d23_isrForCrosstalkSources_855425_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8a98c94f-9fcb-4c23-b597-7ac875066d23_isrForCrosstalkSources_855425_35.stderr parsl.dataflow.dflow DEBUG: Task 3097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3097 has memoization hash 120c9af6a90bbf939c119ce12e94e60e parsl.dataflow.memoization INFO: Task 3097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3097 try 0 launched on executor multi with executor id 3098 parsl.dataflow.dflow INFO: Standard output for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/aea7da20-708f-4cc1-8337-91a4ed06e834_isrForCrosstalkSources_855427_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/aea7da20-708f-4cc1-8337-91a4ed06e834_isrForCrosstalkSources_855427_45.stderr parsl.dataflow.dflow DEBUG: Task 3098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3098 has memoization hash f87b66d289f87120745e9e546280b330 parsl.dataflow.memoization INFO: Task 3098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3098 try 0 launched on executor multi with executor id 3099 parsl.dataflow.dflow INFO: Standard output for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e2a6f6a-3035-4551-879d-5c8eee9f23f2_isrForCrosstalkSources_855380_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e2a6f6a-3035-4551-879d-5c8eee9f23f2_isrForCrosstalkSources_855380_23.stderr parsl.dataflow.dflow DEBUG: Task 3099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3099 has memoization hash c75e701924bbf47e78a83ce8cb46f326 parsl.dataflow.memoization INFO: Task 3099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3099 try 0 launched on executor multi with executor id 3100 parsl.dataflow.dflow INFO: Standard output for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/2c946e58-0575-40f8-8b5e-b97086b046b8_isrForCrosstalkSources_855428_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/2c946e58-0575-40f8-8b5e-b97086b046b8_isrForCrosstalkSources_855428_17.stderr parsl.dataflow.dflow DEBUG: Task 3100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3100 has memoization hash a6145a5dfd13612d19bfe6f9a87e8ab0 parsl.dataflow.memoization INFO: Task 3100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3100 try 0 launched on executor multi with executor id 3101 parsl.dataflow.dflow INFO: Standard output for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3bd27bd0-49db-48b8-9cc1-db6a907988ca_isrForCrosstalkSources_855400_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3bd27bd0-49db-48b8-9cc1-db6a907988ca_isrForCrosstalkSources_855400_57.stderr parsl.dataflow.dflow DEBUG: Task 3101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3101 has memoization hash d053cb6de15ec4c8278fe4b6fc5e1e1b parsl.dataflow.memoization INFO: Task 3101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3101 try 0 launched on executor multi with executor id 3102 parsl.dataflow.dflow INFO: Standard output for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e03f4126-52f8-4778-af1f-646b828612a5_isrForCrosstalkSources_855404_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e03f4126-52f8-4778-af1f-646b828612a5_isrForCrosstalkSources_855404_29.stderr parsl.dataflow.dflow DEBUG: Task 3102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3102 has memoization hash a19620a10e28d5179b56bd7c7d48c624 parsl.dataflow.memoization INFO: Task 3102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:31 proc_lsst.multi:170 [INFO] [multi] job_id 16964166 proc_lsst.multi INFO: [multi] job_id 16964166 2024-03-09 14:51:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-09 14:51:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964166 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964166 parsl.executors.status_handling DEBUG: Launched block 140 on executor multi with job ID 16964166 parsl.executors.status_handling INFO: Allocated block ID 141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-141 2024-03-09 14:51:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-09 14:51:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3102 try 0 launched on executor multi with executor id 3103 parsl.dataflow.dflow INFO: Standard output for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2543b678-93d6-4e0d-b84e-f8bb8abdde2b_isrForCrosstalkSources_855421_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2543b678-93d6-4e0d-b84e-f8bb8abdde2b_isrForCrosstalkSources_855421_1.stderr parsl.dataflow.dflow DEBUG: Task 3103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3103 has memoization hash 8e9fa3918f644ac4899aa58b38269b98 parsl.dataflow.memoization INFO: Task 3103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c2c4748f9e223e196565bff40d50f2a with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/8f52564c-9dac-4aa4-bf6e-696967ddb4b1_isrForCrosstalkSources_855457_10.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/8f52564c-9dac-4aa4-bf6e-696967ddb4b1_isrForCrosstalkSources_855457_10.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3103 try 0 launched on executor multi with executor id 3104 parsl.dataflow.dflow INFO: Standard output for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afb05fae-b3ed-4295-966e-e706e543b9d2_isrForCrosstalkSources_855452_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afb05fae-b3ed-4295-966e-e706e543b9d2_isrForCrosstalkSources_855452_32.stderr parsl.dataflow.dflow DEBUG: Task 3104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3104 has memoization hash 60f2ed895eaa0e2e6d19a01a5bd8aa43 parsl.dataflow.memoization INFO: Task 3104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6159b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3104 try 0 launched on executor multi with executor id 3105 parsl.dataflow.dflow INFO: Standard output for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db45e412-231f-4950-9ad2-18ac46b203fa_isrForCrosstalkSources_855388_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db45e412-231f-4950-9ad2-18ac46b203fa_isrForCrosstalkSources_855388_56.stderr parsl.dataflow.dflow DEBUG: Task 3105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3105 has memoization hash 20eebde199d37c2fa33afe11cdc2560b parsl.dataflow.memoization INFO: Task 3105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3105 try 0 launched on executor multi with executor id 3106 parsl.dataflow.dflow INFO: Standard output for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/080ad1ce-3d12-4426-b000-c6de40212d88_isrForCrosstalkSources_855395_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/080ad1ce-3d12-4426-b000-c6de40212d88_isrForCrosstalkSources_855395_7.stderr parsl.dataflow.dflow DEBUG: Task 3106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3106 has memoization hash bac0f428644d3a2f41e564438d01c4b6 parsl.dataflow.memoization INFO: Task 3106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61582c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3106 try 0 launched on executor multi with executor id 3107 parsl.dataflow.dflow INFO: Standard output for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/97fb034e-fd53-4cca-b5cf-cb277c04e854_isrForCrosstalkSources_855397_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/97fb034e-fd53-4cca-b5cf-cb277c04e854_isrForCrosstalkSources_855397_58.stderr parsl.dataflow.dflow DEBUG: Task 3107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3107 has memoization hash b32de1ee90295a4a1bead09910e38c42 parsl.dataflow.memoization INFO: Task 3107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3107 try 0 launched on executor multi with executor id 3108 parsl.dataflow.dflow INFO: Standard output for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0648fbad-c73b-406b-b6fd-9414a588cb71_isrForCrosstalkSources_855412_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0648fbad-c73b-406b-b6fd-9414a588cb71_isrForCrosstalkSources_855412_9.stderr parsl.dataflow.dflow DEBUG: Task 3108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3108 has memoization hash 0b0f12d8f4ada74d2e6c69d12e7948f0 parsl.dataflow.memoization INFO: Task 3108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3108 try 0 launched on executor multi with executor id 3109 parsl.dataflow.dflow INFO: Standard output for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5ca04777-70a4-4275-825c-1bfd9a66bb14_isrForCrosstalkSources_855407_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5ca04777-70a4-4275-825c-1bfd9a66bb14_isrForCrosstalkSources_855407_12.stderr parsl.dataflow.dflow DEBUG: Task 3109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3109 has memoization hash 9b5535291377c8634925a6226078bdd4 parsl.dataflow.memoization INFO: Task 3109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3109 try 0 launched on executor multi with executor id 3110 parsl.dataflow.dflow INFO: Standard output for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/fef3471e-c8c6-4fe0-a4fb-253be46bcf94_isrForCrosstalkSources_855456_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/fef3471e-c8c6-4fe0-a4fb-253be46bcf94_isrForCrosstalkSources_855456_13.stderr parsl.dataflow.dflow DEBUG: Task 3110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3110 has memoization hash 0e194981bdd800fd635710ebdc27bc00 parsl.dataflow.memoization INFO: Task 3110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3110 try 0 launched on executor multi with executor id 3111 parsl.dataflow.dflow INFO: Standard output for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/16670da2-e09a-46f7-b274-dd9f711d171b_isrForCrosstalkSources_855398_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/16670da2-e09a-46f7-b274-dd9f711d171b_isrForCrosstalkSources_855398_9.stderr parsl.dataflow.dflow DEBUG: Task 3111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3111 has memoization hash 949084c8456dfa990a2bd36b46732ac5 parsl.dataflow.memoization INFO: Task 3111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3111 try 0 launched on executor multi with executor id 3112 parsl.dataflow.dflow INFO: Standard output for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/151441f9-1dbc-4861-bf9c-d03334088bce_isrForCrosstalkSources_855446_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/151441f9-1dbc-4861-bf9c-d03334088bce_isrForCrosstalkSources_855446_2.stderr parsl.dataflow.dflow DEBUG: Task 3112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3112 has memoization hash 7e83a7def455d3349861a8f8e0bb4f20 parsl.dataflow.memoization INFO: Task 3112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3112 try 0 launched on executor multi with executor id 3113 parsl.dataflow.dflow INFO: Standard output for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/cfc40796-2557-4782-9a95-65155da3929d_isrForCrosstalkSources_855450_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/cfc40796-2557-4782-9a95-65155da3929d_isrForCrosstalkSources_855450_19.stderr parsl.dataflow.dflow DEBUG: Task 3113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3113 has memoization hash ad0a32543c88f5ce72d79c3192af0db2 parsl.dataflow.memoization INFO: Task 3113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3113 try 0 launched on executor multi with executor id 3114 parsl.dataflow.dflow INFO: Standard output for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/19d7bf43-47ba-4194-868a-3e83f55ac243_isrForCrosstalkSources_855449_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/19d7bf43-47ba-4194-868a-3e83f55ac243_isrForCrosstalkSources_855449_14.stderr parsl.dataflow.dflow DEBUG: Task 3114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3114 has memoization hash d059fb5372d0646391b35a5ced4e0f4c parsl.dataflow.memoization INFO: Task 3114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3114 try 0 launched on executor multi with executor id 3115 parsl.dataflow.dflow INFO: Standard output for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a38bec8a-0cb8-4d55-be38-958991c26429_isrForCrosstalkSources_855403_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a38bec8a-0cb8-4d55-be38-958991c26429_isrForCrosstalkSources_855403_1.stderr parsl.dataflow.dflow DEBUG: Task 3115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3115 has memoization hash b7338b06a94c0544f3d7da273444fedb parsl.dataflow.memoization INFO: Task 3115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3115 try 0 launched on executor multi with executor id 3116 parsl.dataflow.dflow INFO: Standard output for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/fe561538-8534-4e04-ba4d-009612695ea1_isrForCrosstalkSources_855392_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/fe561538-8534-4e04-ba4d-009612695ea1_isrForCrosstalkSources_855392_6.stderr parsl.dataflow.dflow DEBUG: Task 3116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3116 has memoization hash 35d2c04ac068cf81d00cae9b9f7667af parsl.dataflow.memoization INFO: Task 3116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3116 try 0 launched on executor multi with executor id 3117 parsl.dataflow.dflow INFO: Standard output for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0ca423fa-18e0-4bcf-9b03-9b468161708b_isrForCrosstalkSources_855459_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0ca423fa-18e0-4bcf-9b03-9b468161708b_isrForCrosstalkSources_855459_13.stderr parsl.dataflow.dflow DEBUG: Task 3117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3117 has memoization hash 59d3e93974910e6d39ed2a4f7e648d29 parsl.dataflow.memoization INFO: Task 3117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3117 try 0 launched on executor multi with executor id 3118 parsl.dataflow.dflow INFO: Standard output for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d2a5541b-809c-4f2f-9ae0-9c918070555e_isrForCrosstalkSources_855442_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d2a5541b-809c-4f2f-9ae0-9c918070555e_isrForCrosstalkSources_855442_14.stderr parsl.dataflow.dflow DEBUG: Task 3118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3118 has memoization hash 9ebd3712a5d4dadfa697f1a2cf0088b1 parsl.dataflow.memoization INFO: Task 3118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3118 try 0 launched on executor multi with executor id 3119 parsl.dataflow.dflow INFO: Standard output for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/81e7cd0e-81ee-48b6-b6a2-e6af2cc7093c_isrForCrosstalkSources_855418_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/81e7cd0e-81ee-48b6-b6a2-e6af2cc7093c_isrForCrosstalkSources_855418_54.stderr parsl.dataflow.dflow DEBUG: Task 3119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3119 has memoization hash 70e8c4507dabe6b1e207e8087c0321ff parsl.dataflow.memoization INFO: Task 3119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3119 try 0 launched on executor multi with executor id 3120 parsl.dataflow.dflow INFO: Standard output for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/830555ed-471b-4190-8288-7469b58c4281_isrForCrosstalkSources_855419_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/830555ed-471b-4190-8288-7469b58c4281_isrForCrosstalkSources_855419_53.stderr parsl.dataflow.dflow DEBUG: Task 3120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3120 has memoization hash c8606877ae7d9540231f3d20281ac751 parsl.dataflow.memoization INFO: Task 3120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3120 try 0 launched on executor multi with executor id 3121 parsl.dataflow.dflow INFO: Standard output for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/778ea8ca-3f76-4c4c-b8d6-f13151176ca6_isrForCrosstalkSources_855418_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/778ea8ca-3f76-4c4c-b8d6-f13151176ca6_isrForCrosstalkSources_855418_12.stderr parsl.dataflow.dflow DEBUG: Task 3121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3121 has memoization hash 24a96c9911d3e8956ced19f91b5bab12 parsl.dataflow.memoization INFO: Task 3121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3121 try 0 launched on executor multi with executor id 3122 parsl.dataflow.dflow INFO: Standard output for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18f0e5fe-a34b-4102-a32e-5a6b71034718_isrForCrosstalkSources_855443_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18f0e5fe-a34b-4102-a32e-5a6b71034718_isrForCrosstalkSources_855443_30.stderr parsl.dataflow.dflow DEBUG: Task 3122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3122 has memoization hash 522b518f7a3294055c0cb442db119cf8 parsl.dataflow.memoization INFO: Task 3122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3122 try 0 launched on executor multi with executor id 3123 parsl.dataflow.dflow INFO: Standard output for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7e7606ea-df41-44bf-b54e-87ccd26c2926_isrForCrosstalkSources_855401_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7e7606ea-df41-44bf-b54e-87ccd26c2926_isrForCrosstalkSources_855401_27.stderr parsl.dataflow.dflow DEBUG: Task 3123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3123 has memoization hash 9d4cc64163c822399093f01ba4488821 parsl.dataflow.memoization INFO: Task 3123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3123 try 0 launched on executor multi with executor id 3124 parsl.dataflow.dflow INFO: Standard output for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7c90ac0f-cc74-408a-875a-7eff7f214ae1_isrForCrosstalkSources_855390_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7c90ac0f-cc74-408a-875a-7eff7f214ae1_isrForCrosstalkSources_855390_55.stderr parsl.dataflow.dflow DEBUG: Task 3124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3124 has memoization hash 8debda573090c56d5df1ff422b38102a parsl.dataflow.memoization INFO: Task 3124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3124 try 0 launched on executor multi with executor id 3125 parsl.dataflow.dflow INFO: Standard output for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a80334c6-4aba-4587-b0eb-7312d09322f8_isrForCrosstalkSources_855406_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a80334c6-4aba-4587-b0eb-7312d09322f8_isrForCrosstalkSources_855406_4.stderr parsl.dataflow.dflow DEBUG: Task 3125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3125 has memoization hash e7598cc8acb2b2ac04ddf2a9fe51fee0 parsl.dataflow.memoization INFO: Task 3125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3125 try 0 launched on executor multi with executor id 3126 parsl.dataflow.dflow INFO: Standard output for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/acca1293-970f-4be7-b386-1b7fc82ceffd_isrForCrosstalkSources_855459_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/acca1293-970f-4be7-b386-1b7fc82ceffd_isrForCrosstalkSources_855459_62.stderr parsl.dataflow.dflow DEBUG: Task 3126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3126 has memoization hash ba02b653a81837985badff31956fc3e0 parsl.dataflow.memoization INFO: Task 3126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3126 try 0 launched on executor multi with executor id 3127 parsl.dataflow.dflow INFO: Standard output for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8c400940-51f8-48be-98b8-5576111a894f_isrForCrosstalkSources_855456_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8c400940-51f8-48be-98b8-5576111a894f_isrForCrosstalkSources_855456_62.stderr parsl.dataflow.dflow DEBUG: Task 3127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies 2024-03-09 14:51:32 proc_lsst.multi:170 [INFO] [multi] job_id 16964167 parsl.dataflow.dflow INFO: Task 3127 submitted for App isrForCrosstalkSources, not waiting on any dependency proc_lsst.multi INFO: [multi] job_id 16964167 2024-03-09 14:51:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-09 14:51:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964167 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964167 parsl.dataflow.dflow DEBUG: Task 3127 set to pending state with AppFuture: parsl.executors.status_handling DEBUG: Launched block 141 on executor multi with job ID 16964167 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.executors.status_handling INFO: Allocated block ID 142 parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-142 2024-03-09 14:51:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Task 3127 has memoization hash 3da5b7be1e7b34732320447e067d2c2e parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization INFO: Task 3127 had no result in cache parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9bce13b2346dcaf09490079b1a7fff with result from task 335 parsl.dataflow.dflow INFO: Parsl task 3127 try 0 launched on executor multi with executor id 3128 parsl.dataflow.dflow INFO: Standard output for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8e15fbda-a6cb-4a0b-9b5d-4818721b7ee8_isrForCrosstalkSources_855414_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8e15fbda-a6cb-4a0b-9b5d-4818721b7ee8_isrForCrosstalkSources_855414_1.stderr parsl.dataflow.dflow DEBUG: Task 3128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3128 has memoization hash af966cd63b89aa6c91a0433440381f31 parsl.dataflow.memoization INFO: Task 3128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3128 try 0 launched on executor multi with executor id 3129 parsl.dataflow.dflow INFO: Standard output for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24be9d09-bb79-4de9-97d7-a1d5f6506014_isrForCrosstalkSources_855379_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24be9d09-bb79-4de9-97d7-a1d5f6506014_isrForCrosstalkSources_855379_15.stderr parsl.dataflow.dflow DEBUG: Task 3129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3129 has memoization hash ea9f77e4232844ffc094eb23115c167d parsl.dataflow.memoization INFO: Task 3129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/583c0134-f8ca-4b5e-9b1d-cd913ebbad53_isrForCrosstalkSources_855406_15.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/583c0134-f8ca-4b5e-9b1d-cd913ebbad53_isrForCrosstalkSources_855406_15.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0554d703423165cca0a66918da104ab3 with result from task 320 parsl.dataflow.dflow INFO: Parsl task 3129 try 0 launched on executor multi with executor id 3130 parsl.dataflow.dflow INFO: Standard output for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a66678d3-2bc9-4c19-9d94-ccd592876e75_isrForCrosstalkSources_855449_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a66678d3-2bc9-4c19-9d94-ccd592876e75_isrForCrosstalkSources_855449_42.stderr parsl.dataflow.dflow DEBUG: Task 3130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3130 has memoization hash 28d165db7dfb92797753d17bfe07541f parsl.dataflow.memoization INFO: Task 3130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/36608357-0891-482d-8532-d79e5a714698_isrForCrosstalkSources_855406_18.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/36608357-0891-482d-8532-d79e5a714698_isrForCrosstalkSources_855406_18.stderr parsl.dataflow.dflow INFO: Parsl task 3130 try 0 launched on executor multi with executor id 3131 parsl.dataflow.dflow INFO: Standard output for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a249783e-da8f-4c02-b5a6-bc8f7ee3ef31_isrForCrosstalkSources_855395_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a249783e-da8f-4c02-b5a6-bc8f7ee3ef31_isrForCrosstalkSources_855395_61.stderr parsl.dataflow.dflow DEBUG: Task 3131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3131 has memoization hash 7e09ccc93a526d368a8041d347ec4858 parsl.dataflow.memoization INFO: Task 3131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3131 try 0 launched on executor multi with executor id 3132 parsl.dataflow.dflow INFO: Standard output for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/7f429e02-bae3-47b9-81d7-3fa12bbf1184_isrForCrosstalkSources_855398_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/7f429e02-bae3-47b9-81d7-3fa12bbf1184_isrForCrosstalkSources_855398_25.stderr parsl.dataflow.dflow DEBUG: Task 3132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3132 has memoization hash 8b36122832d4e6fe28e988729e9627e1 parsl.dataflow.memoization INFO: Task 3132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3132 try 0 launched on executor multi with executor id 3133 parsl.dataflow.dflow INFO: Standard output for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4c60a38c-b757-4cb1-abe5-c1cd681ae9db_isrForCrosstalkSources_855426_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4c60a38c-b757-4cb1-abe5-c1cd681ae9db_isrForCrosstalkSources_855426_55.stderr parsl.dataflow.dflow DEBUG: Task 3133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3133 has memoization hash 01745799e1ba4f117e137aa63169a09f parsl.dataflow.memoization INFO: Task 3133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3133 try 0 launched on executor multi with executor id 3134 parsl.dataflow.dflow INFO: Standard output for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bc24003f-15dd-4bda-8ebc-d74dab03707c_isrForCrosstalkSources_855386_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bc24003f-15dd-4bda-8ebc-d74dab03707c_isrForCrosstalkSources_855386_28.stderr parsl.dataflow.dflow DEBUG: Task 3134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3134 has memoization hash 09a53edf9436f62002bcddfc54f19abe parsl.dataflow.memoization INFO: Task 3134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3134 try 0 launched on executor multi with executor id 3135 parsl.dataflow.dflow INFO: Standard output for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/faa91b9a-4c37-4a3a-84a8-0bc885237d6f_isrForCrosstalkSources_855456_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/faa91b9a-4c37-4a3a-84a8-0bc885237d6f_isrForCrosstalkSources_855456_47.stderr parsl.dataflow.dflow DEBUG: Task 3135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3135 has memoization hash bb6c269767549469c9d1bd0717f44443 parsl.dataflow.memoization INFO: Task 3135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3135 try 0 launched on executor multi with executor id 3136 parsl.dataflow.dflow INFO: Standard output for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/da2b94dc-ecee-4d84-b652-5d5831c5e327_isrForCrosstalkSources_855424_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/da2b94dc-ecee-4d84-b652-5d5831c5e327_isrForCrosstalkSources_855424_60.stderr parsl.dataflow.dflow DEBUG: Task 3136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3136 has memoization hash aad565b692a07f23cfc6841f94c5c455 parsl.dataflow.memoization INFO: Task 3136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3136 try 0 launched on executor multi with executor id 3137 parsl.dataflow.dflow INFO: Standard output for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5ecaad0c-6375-421e-9391-23678addbe86_isrForCrosstalkSources_855430_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5ecaad0c-6375-421e-9391-23678addbe86_isrForCrosstalkSources_855430_57.stderr parsl.dataflow.dflow DEBUG: Task 3137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3137 has memoization hash 4a721eed13529e2101a8b3dfbbfef79c parsl.dataflow.memoization INFO: Task 3137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3137 try 0 launched on executor multi with executor id 3138 parsl.dataflow.dflow INFO: Standard output for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ba184d63-eae0-41e2-90f1-42a10e8e405a_isrForCrosstalkSources_855429_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ba184d63-eae0-41e2-90f1-42a10e8e405a_isrForCrosstalkSources_855429_47.stderr parsl.dataflow.dflow DEBUG: Task 3138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3138 has memoization hash 8273fe20f6c4efccc87430eb5e29ef0a parsl.dataflow.memoization INFO: Task 3138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3138 try 0 launched on executor multi with executor id 3139 parsl.dataflow.dflow INFO: Standard output for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/309ada1d-cb8a-4cbe-8ff6-6ea00ef569a2_isrForCrosstalkSources_855442_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/309ada1d-cb8a-4cbe-8ff6-6ea00ef569a2_isrForCrosstalkSources_855442_20.stderr parsl.dataflow.dflow DEBUG: Task 3139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3139 has memoization hash a25da4789a10d3d3621c2ff6460fc1fc parsl.dataflow.memoization INFO: Task 3139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3139 try 0 launched on executor multi with executor id 3140 parsl.dataflow.dflow INFO: Standard output for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/9d5a393f-8cd1-4e34-b786-2a4239b75dde_isrForCrosstalkSources_855419_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/9d5a393f-8cd1-4e34-b786-2a4239b75dde_isrForCrosstalkSources_855419_3.stderr parsl.dataflow.dflow DEBUG: Task 3140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3140 has memoization hash 22c7cc708ba50580a328cc217303d72e parsl.dataflow.memoization INFO: Task 3140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3140 try 0 launched on executor multi with executor id 3141 parsl.dataflow.dflow INFO: Standard output for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/fd569ef2-153d-43ff-ae8c-c1cd76f62f72_isrForCrosstalkSources_855407_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/fd569ef2-153d-43ff-ae8c-c1cd76f62f72_isrForCrosstalkSources_855407_38.stderr parsl.dataflow.dflow DEBUG: Task 3141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3141 has memoization hash 0eb4c914742af45f08c6b1ef829361f8 parsl.dataflow.memoization INFO: Task 3141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3141 try 0 launched on executor multi with executor id 3142 parsl.dataflow.dflow INFO: Standard output for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/64fb6471-a692-414f-8edf-a47b21f64520_isrForCrosstalkSources_855435_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/64fb6471-a692-414f-8edf-a47b21f64520_isrForCrosstalkSources_855435_33.stderr parsl.dataflow.dflow DEBUG: Task 3142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3142 has memoization hash e63944d71ac8a68d083b2dc704a748d3 parsl.dataflow.memoization INFO: Task 3142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3142 try 0 launched on executor multi with executor id 3143 parsl.dataflow.dflow INFO: Standard output for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5da6fbfc-4554-4bac-b9e8-2ae4849cf266_isrForCrosstalkSources_855444_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5da6fbfc-4554-4bac-b9e8-2ae4849cf266_isrForCrosstalkSources_855444_41.stderr parsl.dataflow.dflow DEBUG: Task 3143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3143 has memoization hash 6ebe8294319044c7c204e9d332936c61 parsl.dataflow.memoization INFO: Task 3143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3143 try 0 launched on executor multi with executor id 3144 parsl.dataflow.dflow INFO: Standard output for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a84d9ac7-2bdd-43e7-b30a-afc21972307f_isrForCrosstalkSources_855421_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a84d9ac7-2bdd-43e7-b30a-afc21972307f_isrForCrosstalkSources_855421_15.stderr parsl.dataflow.dflow DEBUG: Task 3144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3144 has memoization hash 4ef6fe297c088d85779baa59eef57090 parsl.dataflow.memoization INFO: Task 3144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3144 try 0 launched on executor multi with executor id 3145 parsl.dataflow.dflow INFO: Standard output for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0fe2b54c-4a4a-4965-8f21-fad9d49704c2_isrForCrosstalkSources_855450_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0fe2b54c-4a4a-4965-8f21-fad9d49704c2_isrForCrosstalkSources_855450_54.stderr parsl.dataflow.dflow DEBUG: Task 3145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3145 has memoization hash 1d0e65fa279d6e139f73a6a3e2b3d22f parsl.dataflow.memoization INFO: Task 3145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3145 try 0 launched on executor multi with executor id 3146 parsl.dataflow.dflow INFO: Standard output for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/aaa52fb6-6f71-474d-bf70-75c34f2b9560_isrForCrosstalkSources_855441_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/aaa52fb6-6f71-474d-bf70-75c34f2b9560_isrForCrosstalkSources_855441_42.stderr parsl.dataflow.dflow DEBUG: Task 3146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3146 has memoization hash c9d79df8224b14042a4181636f830761 parsl.dataflow.memoization INFO: Task 3146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3146 try 0 launched on executor multi with executor id 3147 parsl.dataflow.dflow INFO: Standard output for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/2a4801a7-5262-49ab-b1d7-a38a14500290_isrForCrosstalkSources_855413_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/2a4801a7-5262-49ab-b1d7-a38a14500290_isrForCrosstalkSources_855413_57.stderr parsl.dataflow.dflow DEBUG: Task 3147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3147 has memoization hash d890b72675432f016eaa4abfff528e44 parsl.dataflow.memoization INFO: Task 3147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3147 try 0 launched on executor multi with executor id 3148 parsl.dataflow.dflow INFO: Standard output for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/580a8ac3-ade0-4a3b-829e-4a38a2e4fe07_isrForCrosstalkSources_855403_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/580a8ac3-ade0-4a3b-829e-4a38a2e4fe07_isrForCrosstalkSources_855403_13.stderr parsl.dataflow.dflow DEBUG: Task 3148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3148 has memoization hash 9e0122201afe6c4519ecd6aac2347806 parsl.dataflow.memoization INFO: Task 3148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3148 try 0 launched on executor multi with executor id 3149 parsl.dataflow.dflow INFO: Standard output for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8372b7e-0188-47f6-8a08-af8fe5369012_isrForCrosstalkSources_855450_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8372b7e-0188-47f6-8a08-af8fe5369012_isrForCrosstalkSources_855450_43.stderr parsl.dataflow.dflow DEBUG: Task 3149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3149 has memoization hash f5dafa97722c03990debd95c73a576cb parsl.dataflow.memoization INFO: Task 3149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3149 try 0 launched on executor multi with executor id 3150 parsl.dataflow.dflow INFO: Standard output for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/79b347b6-066b-4170-97ad-9006d697445d_isrForCrosstalkSources_855455_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/79b347b6-066b-4170-97ad-9006d697445d_isrForCrosstalkSources_855455_60.stderr parsl.dataflow.dflow DEBUG: Task 3150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3150 has memoization hash 7ea4fb47735fe25eced59a6056c81764 parsl.dataflow.memoization INFO: Task 3150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3150 try 0 launched on executor multi with executor id 3151 parsl.dataflow.dflow INFO: Standard output for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1a50e18a-f8e6-43ea-be0e-8d7e432730ab_isrForCrosstalkSources_855382_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1a50e18a-f8e6-43ea-be0e-8d7e432730ab_isrForCrosstalkSources_855382_38.stderr parsl.dataflow.dflow DEBUG: Task 3151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3151 has memoization hash 1e0485c231c73366cecc1a70c4c25f82 parsl.dataflow.memoization INFO: Task 3151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3151 try 0 launched on executor multi with executor id 3152 parsl.dataflow.dflow INFO: Standard output for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d18e6b20-398d-44ca-a0a6-2f314f852980_isrForCrosstalkSources_855385_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d18e6b20-398d-44ca-a0a6-2f314f852980_isrForCrosstalkSources_855385_46.stderr parsl.dataflow.dflow DEBUG: Task 3152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3152 has memoization hash aa826b04e48d847442b55870a9ee07f5 parsl.dataflow.memoization INFO: Task 3152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3152 try 0 launched on executor multi with executor id 3153 parsl.dataflow.dflow INFO: Standard output for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b6575efc-4324-4b1b-b3c9-9f1d98f5f705_isrForCrosstalkSources_855439_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b6575efc-4324-4b1b-b3c9-9f1d98f5f705_isrForCrosstalkSources_855439_11.stderr parsl.dataflow.dflow DEBUG: Task 3153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3153 has memoization hash def4eaf0e232762496253ed09df3e33b parsl.dataflow.memoization INFO: Task 3153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3153 try 0 launched on executor multi with executor id 3154 parsl.dataflow.dflow INFO: Standard output for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8908981e-9194-4a74-a90b-4e88cff77c5f_isrForCrosstalkSources_855445_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8908981e-9194-4a74-a90b-4e88cff77c5f_isrForCrosstalkSources_855445_60.stderr parsl.dataflow.dflow DEBUG: Task 3154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3154 has memoization hash a0a0c7926fa467b993b5c2f1c7e9e5f1 parsl.dataflow.memoization INFO: Task 3154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3154 try 0 launched on executor multi with executor id 3155 parsl.dataflow.dflow INFO: Standard output for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/853e3782-652f-4dd3-9fd2-7c2d7bcee168_isrForCrosstalkSources_855415_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/853e3782-652f-4dd3-9fd2-7c2d7bcee168_isrForCrosstalkSources_855415_3.stderr parsl.dataflow.dflow DEBUG: Task 3155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3155 has memoization hash 504e5e158f31094f0b5e9db73daa3249 parsl.dataflow.memoization INFO: Task 3155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3155 try 0 launched on executor multi with executor id 3156 parsl.dataflow.dflow INFO: Standard output for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f198a668-8569-441d-80d9-cc9b2edfaf00_isrForCrosstalkSources_855423_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f198a668-8569-441d-80d9-cc9b2edfaf00_isrForCrosstalkSources_855423_33.stderr parsl.dataflow.dflow DEBUG: Task 3156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3156 has memoization hash 74d8a3f79a77cf18d0669a491b857f74 parsl.dataflow.memoization INFO: Task 3156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3156 try 0 launched on executor multi with executor id 3157 parsl.dataflow.dflow INFO: Standard output for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e05006cf-1977-4752-975a-88e0cbc2155d_isrForCrosstalkSources_855382_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e05006cf-1977-4752-975a-88e0cbc2155d_isrForCrosstalkSources_855382_53.stderr parsl.dataflow.dflow DEBUG: Task 3157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3157 has memoization hash d196f0eb2ffb6b41c32343abc297d85e parsl.dataflow.memoization INFO: Task 3157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3157 try 0 launched on executor multi with executor id 3158 parsl.dataflow.dflow INFO: Standard output for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/445dcbf1-1a04-4ee0-88a4-bb09d6ee5916_isrForCrosstalkSources_855382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/445dcbf1-1a04-4ee0-88a4-bb09d6ee5916_isrForCrosstalkSources_855382_2.stderr parsl.dataflow.dflow DEBUG: Task 3158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3158 has memoization hash 3f7835e69b1e086b1070fb297702019e parsl.dataflow.memoization INFO: Task 3158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3158 try 0 launched on executor multi with executor id 3159 parsl.dataflow.dflow INFO: Standard output for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99b38168-9152-4bad-ba1f-35d2c6c6f297_isrForCrosstalkSources_855406_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99b38168-9152-4bad-ba1f-35d2c6c6f297_isrForCrosstalkSources_855406_19.stderr parsl.dataflow.dflow DEBUG: Task 3159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3159 has memoization hash f92f3af8422fa0832f360d13c0b77f8c parsl.dataflow.memoization INFO: Task 3159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3159 try 0 launched on executor multi with executor id 3160 parsl.dataflow.dflow INFO: Standard output for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/30f11702-0533-4d06-8d00-7216c6f8e7f4_isrForCrosstalkSources_855402_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/30f11702-0533-4d06-8d00-7216c6f8e7f4_isrForCrosstalkSources_855402_5.stderr parsl.dataflow.dflow DEBUG: Task 3160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3160 has memoization hash fdaee9229635a81837a35202ba66ac9d parsl.dataflow.memoization INFO: Task 3160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3160 try 0 launched on executor multi with executor id 3161 parsl.dataflow.dflow INFO: Standard output for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11b27807-dbdf-4afe-9544-cef4ec6ff83e_isrForCrosstalkSources_855426_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11b27807-dbdf-4afe-9544-cef4ec6ff83e_isrForCrosstalkSources_855426_34.stderr parsl.dataflow.dflow DEBUG: Task 3161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3161 has memoization hash fa10f3dc3c09f60da28da9a247a7d4b2 parsl.dataflow.memoization INFO: Task 3161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3161 try 0 launched on executor multi with executor id 3162 parsl.dataflow.dflow INFO: Standard output for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5a9557ac-0f3b-4db3-a104-2b4242c46285_isrForCrosstalkSources_855382_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5a9557ac-0f3b-4db3-a104-2b4242c46285_isrForCrosstalkSources_855382_60.stderr parsl.dataflow.dflow DEBUG: Task 3162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3162 has memoization hash cc642c1625bfb190ba2426e889c98357 parsl.dataflow.memoization INFO: Task 3162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3162 try 0 launched on executor multi with executor id 3163 parsl.dataflow.dflow INFO: Standard output for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/65e8dd92-84a0-4604-9114-a2a252649ff9_isrForCrosstalkSources_855417_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/65e8dd92-84a0-4604-9114-a2a252649ff9_isrForCrosstalkSources_855417_28.stderr parsl.dataflow.dflow DEBUG: Task 3163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3163 has memoization hash ca1386793ef46a2bbad42fd07e42d3ba parsl.dataflow.memoization INFO: Task 3163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3163 try 0 launched on executor multi with executor id 3164 parsl.dataflow.dflow INFO: Standard output for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/7d45e839-aebd-4405-9d8c-72a4149f2fee_isrForCrosstalkSources_855435_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/7d45e839-aebd-4405-9d8c-72a4149f2fee_isrForCrosstalkSources_855435_60.stderr parsl.dataflow.dflow DEBUG: Task 3164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3164 has memoization hash d9f21132d8a38ee789e7c575aa32b08d parsl.dataflow.memoization INFO: Task 3164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3164 try 0 launched on executor multi with executor id 3165 parsl.dataflow.dflow INFO: Standard output for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/836c955a-8f7b-4146-b905-6863a2bc45e5_isrForCrosstalkSources_855454_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/836c955a-8f7b-4146-b905-6863a2bc45e5_isrForCrosstalkSources_855454_21.stderr parsl.dataflow.dflow DEBUG: Task 3165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3165 has memoization hash f3da364826d66f27a9c5c37151355a4b parsl.dataflow.memoization INFO: Task 3165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3165 try 0 launched on executor multi with executor id 3166 parsl.dataflow.dflow INFO: Standard output for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/17861c5a-cdf5-4817-98f6-4e48c0168f80_isrForCrosstalkSources_855445_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/17861c5a-cdf5-4817-98f6-4e48c0168f80_isrForCrosstalkSources_855445_44.stderr parsl.dataflow.dflow DEBUG: Task 3166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3166 has memoization hash 87ae91884d3f88d959d155e8cb60ada3 parsl.dataflow.memoization INFO: Task 3166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3166 try 0 launched on executor multi with executor id 3167 parsl.dataflow.dflow INFO: Standard output for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1b655500-b1d0-4ddb-b037-f67e7f28e15f_isrForCrosstalkSources_855403_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1b655500-b1d0-4ddb-b037-f67e7f28e15f_isrForCrosstalkSources_855403_44.stderr parsl.dataflow.dflow DEBUG: Task 3167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3167 has memoization hash 17aaca43568e9454589040f8235464cb parsl.dataflow.memoization INFO: Task 3167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3167 try 0 launched on executor multi with executor id 3168 parsl.dataflow.dflow INFO: Standard output for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/f4ba1914-4649-4104-9a49-c64a4fe547ea_isrForCrosstalkSources_855442_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/f4ba1914-4649-4104-9a49-c64a4fe547ea_isrForCrosstalkSources_855442_57.stderr parsl.dataflow.dflow DEBUG: Task 3168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3168 has memoization hash 55416ab362e63ff50c46524043b4ca78 parsl.dataflow.memoization INFO: Task 3168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3168 try 0 launched on executor multi with executor id 3169 parsl.dataflow.dflow INFO: Standard output for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ea38f6d2-3356-4fae-bd1e-b3db26edd2a8_isrForCrosstalkSources_855441_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ea38f6d2-3356-4fae-bd1e-b3db26edd2a8_isrForCrosstalkSources_855441_21.stderr parsl.dataflow.dflow DEBUG: Task 3169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3169 has memoization hash fd1af5fc59bee3f3ae70a3806709dd78 parsl.dataflow.memoization INFO: Task 3169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3169 try 0 launched on executor multi with executor id 3170 parsl.dataflow.dflow INFO: Standard output for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1a538d35-3571-4fd8-a5d3-1d2e6021518f_isrForCrosstalkSources_855384_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1a538d35-3571-4fd8-a5d3-1d2e6021518f_isrForCrosstalkSources_855384_53.stderr parsl.dataflow.dflow DEBUG: Task 3170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3170 has memoization hash 8e3b907dce88f3c39ab279ea76823975 parsl.dataflow.memoization INFO: Task 3170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3170 try 0 launched on executor multi with executor id 3171 parsl.dataflow.dflow INFO: Standard output for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/777f044c-989a-420d-a866-9856ab75bfd4_isrForCrosstalkSources_855385_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/777f044c-989a-420d-a866-9856ab75bfd4_isrForCrosstalkSources_855385_43.stderr parsl.dataflow.dflow DEBUG: Task 3171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3171 has memoization hash 3ffe08a44d93a186f3892667ede7d138 parsl.dataflow.memoization INFO: Task 3171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3171 try 0 launched on executor multi with executor id 3172 parsl.dataflow.dflow INFO: Standard output for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1be526a5-f47d-4508-9504-3b89d1d2a81c_isrForCrosstalkSources_855421_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1be526a5-f47d-4508-9504-3b89d1d2a81c_isrForCrosstalkSources_855421_44.stderr parsl.dataflow.dflow DEBUG: Task 3172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3172 has memoization hash b59088f247794f0031b3c914433f5ae9 parsl.dataflow.memoization INFO: Task 3172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3172 try 0 launched on executor multi with executor id 3173 parsl.dataflow.dflow INFO: Standard output for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f746a478-aed0-4ec1-8252-dfc12870da28_isrForCrosstalkSources_855398_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f746a478-aed0-4ec1-8252-dfc12870da28_isrForCrosstalkSources_855398_57.stderr parsl.dataflow.dflow DEBUG: Task 3173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3173 has memoization hash db9a4235045eba93d6d8dab3c663eb51 parsl.dataflow.memoization INFO: Task 3173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3173 try 0 launched on executor multi with executor id 3174 parsl.dataflow.dflow INFO: Standard output for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1e1f2582-3b63-4b56-8635-66af830575b7_isrForCrosstalkSources_855380_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1e1f2582-3b63-4b56-8635-66af830575b7_isrForCrosstalkSources_855380_42.stderr parsl.dataflow.dflow DEBUG: Task 3174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3174 has memoization hash 8a502cf0244c97b3a3d2ae65e3c407b3 parsl.dataflow.memoization INFO: Task 3174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3174 try 0 launched on executor multi with executor id 3175 parsl.dataflow.dflow INFO: Standard output for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/54add0ca-0db0-49fb-b3d3-e346bef50085_isrForCrosstalkSources_855391_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/54add0ca-0db0-49fb-b3d3-e346bef50085_isrForCrosstalkSources_855391_60.stderr parsl.dataflow.dflow DEBUG: Task 3175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3175 has memoization hash aba43590b6d5ae79cbd4aeec7a0ffc5d parsl.dataflow.memoization INFO: Task 3175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3175 try 0 launched on executor multi with executor id 3176 parsl.dataflow.dflow INFO: Standard output for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/fba092a2-1304-4942-a29b-6b0800ec0625_isrForCrosstalkSources_855405_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/fba092a2-1304-4942-a29b-6b0800ec0625_isrForCrosstalkSources_855405_45.stderr parsl.dataflow.dflow DEBUG: Task 3176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3176 has memoization hash a57da48e7ebda9131e6b116c880d2b29 parsl.dataflow.memoization INFO: Task 3176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3176 try 0 launched on executor multi with executor id 3177 parsl.dataflow.dflow INFO: Standard output for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed53f06a-d00d-490a-9d85-f9a23f7c87e9_isrForCrosstalkSources_855455_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed53f06a-d00d-490a-9d85-f9a23f7c87e9_isrForCrosstalkSources_855455_62.stderr parsl.dataflow.dflow DEBUG: Task 3177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3177 has memoization hash 5ac723187b510a4c216962e44192e89d parsl.dataflow.memoization INFO: Task 3177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3177 try 0 launched on executor multi with executor id 3178 parsl.dataflow.dflow INFO: Standard output for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/108f9026-1909-4bbc-aa78-361b8e337298_isrForCrosstalkSources_855432_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/108f9026-1909-4bbc-aa78-361b8e337298_isrForCrosstalkSources_855432_58.stderr parsl.dataflow.dflow DEBUG: Task 3178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3178 has memoization hash 7cb8606087be03e37127a269ab1162ed parsl.dataflow.memoization INFO: Task 3178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3178 try 0 launched on executor multi with executor id 3179 parsl.dataflow.dflow INFO: Standard output for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d1a2981f-b157-4c07-b374-cc7bd304701e_isrForCrosstalkSources_855443_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d1a2981f-b157-4c07-b374-cc7bd304701e_isrForCrosstalkSources_855443_60.stderr parsl.dataflow.dflow DEBUG: Task 3179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3179 has memoization hash 5a3304701f7e16527905693eea88e271 parsl.dataflow.memoization INFO: Task 3179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3179 try 0 launched on executor multi with executor id 3180 parsl.dataflow.dflow INFO: Standard output for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6d54d8cb-f27a-4b87-8bda-5042e4ccf43a_isrForCrosstalkSources_855415_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6d54d8cb-f27a-4b87-8bda-5042e4ccf43a_isrForCrosstalkSources_855415_42.stderr parsl.dataflow.dflow DEBUG: Task 3180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3180 has memoization hash 4ac9839140289e0a15d37f01593bcf76 parsl.dataflow.memoization INFO: Task 3180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3180 try 0 launched on executor multi with executor id 3181 parsl.dataflow.dflow INFO: Standard output for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de53612c-9c40-40c6-9c2f-af72fdc74a27_isrForCrosstalkSources_855459_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de53612c-9c40-40c6-9c2f-af72fdc74a27_isrForCrosstalkSources_855459_3.stderr parsl.dataflow.dflow DEBUG: Task 3181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3181 has memoization hash 4493b6b99b909d6ecaf7ad88d662c746 parsl.dataflow.memoization INFO: Task 3181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3181 try 0 launched on executor multi with executor id 3182 parsl.dataflow.dflow INFO: Standard output for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/930cd220-b14a-4b44-85f5-0de7569e764c_isrForCrosstalkSources_855438_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/930cd220-b14a-4b44-85f5-0de7569e764c_isrForCrosstalkSources_855438_26.stderr parsl.dataflow.dflow DEBUG: Task 3182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3182 has memoization hash 207b26957c8c2b982f84e34ba656e474 parsl.dataflow.memoization INFO: Task 3182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3182 try 0 launched on executor multi with executor id 3183 parsl.dataflow.dflow INFO: Standard output for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3cc3d15a-3f84-41d8-a77b-f4847dd241c3_isrForCrosstalkSources_855424_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3cc3d15a-3f84-41d8-a77b-f4847dd241c3_isrForCrosstalkSources_855424_44.stderr parsl.dataflow.dflow DEBUG: Task 3183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3183 has memoization hash 9b78c1d471aad115b462e65f63208d63 parsl.dataflow.memoization INFO: Task 3183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:32 proc_lsst.multi:170 [INFO] [multi] job_id 16964168 proc_lsst.multi INFO: [multi] job_id 16964168 2024-03-09 14:51:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-09 14:51:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964168 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964168 parsl.executors.status_handling DEBUG: Launched block 142 on executor multi with job ID 16964168 parsl.executors.status_handling INFO: Allocated block ID 143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-143 2024-03-09 14:51:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-09 14:51:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3183 try 0 launched on executor multi with executor id 3184 parsl.dataflow.dflow INFO: Standard output for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ded8ac30-a29a-476e-9133-10af376514ec_isrForCrosstalkSources_855437_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ded8ac30-a29a-476e-9133-10af376514ec_isrForCrosstalkSources_855437_30.stderr parsl.dataflow.dflow DEBUG: Task 3184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3184 has memoization hash 368ab406d0f06617548bd52002b6fba3 parsl.dataflow.memoization INFO: Task 3184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3184 try 0 launched on executor multi with executor id 3185 parsl.dataflow.dflow INFO: Standard output for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d0961b78-a411-4800-b61e-cad83cf8106e_isrForCrosstalkSources_855389_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d0961b78-a411-4800-b61e-cad83cf8106e_isrForCrosstalkSources_855389_26.stderr parsl.dataflow.dflow DEBUG: Task 3185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3185 has memoization hash 82e56b78a87bc061593a8fec0050636a parsl.dataflow.memoization INFO: Task 3185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a6158180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3185 try 0 launched on executor multi with executor id 3186 parsl.dataflow.dflow INFO: Standard output for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d716a157-5d95-4453-85bb-b06dd7f82de4_isrForCrosstalkSources_855457_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d716a157-5d95-4453-85bb-b06dd7f82de4_isrForCrosstalkSources_855457_42.stderr parsl.dataflow.dflow DEBUG: Task 3186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3186 has memoization hash cfe33d7e004494cb9ee17f294068d86d parsl.dataflow.memoization INFO: Task 3186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a615a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3186 try 0 launched on executor multi with executor id 3187 parsl.dataflow.dflow INFO: Standard output for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/05e77e9c-b34b-4aba-a2ed-5dedace27adf_isrForCrosstalkSources_855459_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/05e77e9c-b34b-4aba-a2ed-5dedace27adf_isrForCrosstalkSources_855459_43.stderr parsl.dataflow.dflow DEBUG: Task 3187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3187 has memoization hash 51c4a008b0b91d3170ccee7c72927f96 parsl.dataflow.memoization INFO: Task 3187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3187 try 0 launched on executor multi with executor id 3188 parsl.dataflow.dflow INFO: Standard output for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ab2ee0b6-6fe6-453a-b83a-48c7824426b5_isrForCrosstalkSources_855411_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ab2ee0b6-6fe6-453a-b83a-48c7824426b5_isrForCrosstalkSources_855411_30.stderr parsl.dataflow.dflow DEBUG: Task 3188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3188 has memoization hash 94f895e32be9c0782b0510fb1f7f802c parsl.dataflow.memoization INFO: Task 3188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3188 try 0 launched on executor multi with executor id 3189 parsl.dataflow.dflow INFO: Standard output for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/63ea6ae9-ed73-4d08-b28e-49a8dd0a685a_isrForCrosstalkSources_855433_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/63ea6ae9-ed73-4d08-b28e-49a8dd0a685a_isrForCrosstalkSources_855433_44.stderr parsl.dataflow.dflow DEBUG: Task 3189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3189 has memoization hash e1759f88cbecf6a32fd16de16fcebc1c parsl.dataflow.memoization INFO: Task 3189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3189 try 0 launched on executor multi with executor id 3190 parsl.dataflow.dflow INFO: Standard output for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/46623f48-f95b-420b-80d3-cb470b12405e_isrForCrosstalkSources_855439_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/46623f48-f95b-420b-80d3-cb470b12405e_isrForCrosstalkSources_855439_45.stderr parsl.dataflow.dflow DEBUG: Task 3190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3190 has memoization hash a0ab080ded4a94cd5af1f77d51f35561 parsl.dataflow.memoization INFO: Task 3190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3190 try 0 launched on executor multi with executor id 3191 parsl.dataflow.dflow INFO: Standard output for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/988033b0-e16c-4e74-81a4-01a559c45d04_isrForCrosstalkSources_855440_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/988033b0-e16c-4e74-81a4-01a559c45d04_isrForCrosstalkSources_855440_51.stderr parsl.dataflow.dflow DEBUG: Task 3191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3191 has memoization hash 03a61fc8f2f032e194318ded02a92ad0 parsl.dataflow.memoization INFO: Task 3191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3191 try 0 launched on executor multi with executor id 3192 parsl.dataflow.dflow INFO: Standard output for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2fcdadca-4bbe-4417-a1d5-6b0e60a8b005_isrForCrosstalkSources_855442_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2fcdadca-4bbe-4417-a1d5-6b0e60a8b005_isrForCrosstalkSources_855442_35.stderr parsl.dataflow.dflow DEBUG: Task 3192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3192 has memoization hash 15f6da44a3f5faf4f9f336adcc238f1d parsl.dataflow.memoization INFO: Task 3192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3192 try 0 launched on executor multi with executor id 3193 parsl.dataflow.dflow INFO: Standard output for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/84e52086-02ea-4e49-8f6d-16bbdfaf5e68_isrForCrosstalkSources_855448_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/84e52086-02ea-4e49-8f6d-16bbdfaf5e68_isrForCrosstalkSources_855448_33.stderr parsl.dataflow.dflow DEBUG: Task 3193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3193 has memoization hash 01ae88617a948725cb5838519405fa8c parsl.dataflow.memoization INFO: Task 3193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3193 try 0 launched on executor multi with executor id 3194 parsl.dataflow.dflow INFO: Standard output for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9487032c-d180-4295-8afb-6656cedfbc0d_isrForCrosstalkSources_855456_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9487032c-d180-4295-8afb-6656cedfbc0d_isrForCrosstalkSources_855456_22.stderr parsl.dataflow.dflow DEBUG: Task 3194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3194 has memoization hash e41a3f17878f7375518668a024ef2b75 parsl.dataflow.memoization INFO: Task 3194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3194 try 0 launched on executor multi with executor id 3195 parsl.dataflow.dflow INFO: Standard output for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ea1dc659-90fa-4ba0-8980-2ef017ee73a4_isrForCrosstalkSources_855383_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ea1dc659-90fa-4ba0-8980-2ef017ee73a4_isrForCrosstalkSources_855383_37.stderr parsl.dataflow.dflow DEBUG: Task 3195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3195 has memoization hash 57614aca7a0cf1ad7e3200ec177a928c parsl.dataflow.memoization INFO: Task 3195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50ddd1ed3ddecebba3f42e054572f9a8 with result from task 164 parsl.dataflow.dflow INFO: Parsl task 3195 try 0 launched on executor multi with executor id 3196 parsl.dataflow.dflow INFO: Standard output for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1d32bb10-c2c2-40e8-91e3-0d4457064999_isrForCrosstalkSources_855437_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1d32bb10-c2c2-40e8-91e3-0d4457064999_isrForCrosstalkSources_855437_32.stderr parsl.dataflow.dflow DEBUG: Task 3196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3196 has memoization hash df6562bda0520a5383dd6a04fa5ba3e4 parsl.dataflow.memoization INFO: Task 3196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3196 try 0 launched on executor multi with executor id 3197 parsl.dataflow.dflow INFO: Standard output for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/738a3ac8-47c8-41e1-93ee-a51e6dc241c4_isrForCrosstalkSources_855394_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/738a3ac8-47c8-41e1-93ee-a51e6dc241c4_isrForCrosstalkSources_855394_11.stderr parsl.dataflow.dflow DEBUG: Task 3197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3197 has memoization hash b649140c364bb20e5e4d29c620d59603 parsl.dataflow.memoization INFO: Task 3197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a9305eb6-5ac7-42b4-9542-3a8689986e53_isrForCrosstalkSources_855397_17.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a9305eb6-5ac7-42b4-9542-3a8689986e53_isrForCrosstalkSources_855397_17.stderr parsl.dataflow.dflow INFO: Parsl task 3197 try 0 launched on executor multi with executor id 3198 parsl.dataflow.dflow INFO: Standard output for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/edd5a820-1c68-47be-b7d8-263e07ae92bf_isrForCrosstalkSources_855436_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/edd5a820-1c68-47be-b7d8-263e07ae92bf_isrForCrosstalkSources_855436_48.stderr parsl.dataflow.dflow DEBUG: Task 3198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3198 has memoization hash 056421ecf787f556f8ec34c254443b6d parsl.dataflow.memoization INFO: Task 3198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3198 try 0 launched on executor multi with executor id 3199 parsl.dataflow.dflow INFO: Standard output for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a854591f-3c71-402e-8cfc-a29992c8cf5a_isrForCrosstalkSources_855441_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a854591f-3c71-402e-8cfc-a29992c8cf5a_isrForCrosstalkSources_855441_60.stderr parsl.dataflow.dflow DEBUG: Task 3199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3199 has memoization hash 45e4219bf0e4f0f6949f8f27302d8d2c parsl.dataflow.memoization INFO: Task 3199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3199 try 0 launched on executor multi with executor id 3200 parsl.dataflow.dflow INFO: Standard output for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a8562c53-a77b-4732-8000-e02778069740_isrForCrosstalkSources_855408_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a8562c53-a77b-4732-8000-e02778069740_isrForCrosstalkSources_855408_19.stderr parsl.dataflow.dflow DEBUG: Task 3200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3200 has memoization hash 69371c4da87acfe7721a313fe1668696 parsl.dataflow.memoization INFO: Task 3200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3200 try 0 launched on executor multi with executor id 3201 parsl.dataflow.dflow INFO: Standard output for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a9bf15d2-1648-4960-9e63-d1e8298385d8_isrForCrosstalkSources_855408_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a9bf15d2-1648-4960-9e63-d1e8298385d8_isrForCrosstalkSources_855408_51.stderr parsl.dataflow.dflow DEBUG: Task 3201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3201 has memoization hash 61d3bb7e02231c12344d0174df7b77f2 parsl.dataflow.memoization INFO: Task 3201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3201 try 0 launched on executor multi with executor id 3202 parsl.dataflow.dflow INFO: Standard output for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3688b4-6e1f-4d11-bb41-c1a35f91e865_isrForCrosstalkSources_855445_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3688b4-6e1f-4d11-bb41-c1a35f91e865_isrForCrosstalkSources_855445_48.stderr parsl.dataflow.dflow DEBUG: Task 3202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3202 has memoization hash feea43eb493d8fb0bf4d36ee6f5ab0e0 parsl.dataflow.memoization INFO: Task 3202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3202 try 0 launched on executor multi with executor id 3203 parsl.dataflow.dflow INFO: Standard output for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e9b9c204-1019-46e2-b038-6616021399c5_isrForCrosstalkSources_855451_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e9b9c204-1019-46e2-b038-6616021399c5_isrForCrosstalkSources_855451_39.stderr parsl.dataflow.dflow DEBUG: Task 3203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3203 has memoization hash a23111972eb19b33a20032120aeef1fd parsl.dataflow.memoization INFO: Task 3203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3203 try 0 launched on executor multi with executor id 3204 parsl.dataflow.dflow INFO: Standard output for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/225d352b-6098-4f09-90e1-cd7f79a9d8cd_isrForCrosstalkSources_855400_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/225d352b-6098-4f09-90e1-cd7f79a9d8cd_isrForCrosstalkSources_855400_55.stderr parsl.dataflow.dflow DEBUG: Task 3204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3204 has memoization hash 408369da1d08c2420c572c499da4c436 parsl.dataflow.memoization INFO: Task 3204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3204 try 0 launched on executor multi with executor id 3205 parsl.dataflow.dflow INFO: Standard output for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/aacad426-95cb-4cc5-a93d-7aa6adbd6ad0_isrForCrosstalkSources_855382_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/aacad426-95cb-4cc5-a93d-7aa6adbd6ad0_isrForCrosstalkSources_855382_55.stderr parsl.dataflow.dflow DEBUG: Task 3205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3205 has memoization hash 36c8b71e255a9ffcd09679f63138723b parsl.dataflow.memoization INFO: Task 3205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3205 try 0 launched on executor multi with executor id 3206 parsl.dataflow.dflow INFO: Standard output for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/23baf001-7978-4ede-ad42-1410b03ada2f_isrForCrosstalkSources_855417_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/23baf001-7978-4ede-ad42-1410b03ada2f_isrForCrosstalkSources_855417_8.stderr parsl.dataflow.dflow DEBUG: Task 3206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3206 has memoization hash c4300f048e93d298bdd16344bab81400 parsl.dataflow.memoization INFO: Task 3206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3206 try 0 launched on executor multi with executor id 3207 parsl.dataflow.dflow INFO: Standard output for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6fc6a114-9ce6-4ce9-ad7e-c5f4cf67e9c7_isrForCrosstalkSources_855447_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6fc6a114-9ce6-4ce9-ad7e-c5f4cf67e9c7_isrForCrosstalkSources_855447_23.stderr parsl.dataflow.dflow DEBUG: Task 3207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3207 has memoization hash c6addb64a03cf265331b391690c0d6dd parsl.dataflow.memoization INFO: Task 3207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3207 try 0 launched on executor multi with executor id 3208 parsl.dataflow.dflow INFO: Standard output for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dee6f79c-fb92-469f-919f-2e8ca5b39bfd_isrForCrosstalkSources_855418_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dee6f79c-fb92-469f-919f-2e8ca5b39bfd_isrForCrosstalkSources_855418_22.stderr parsl.dataflow.dflow DEBUG: Task 3208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3208 has memoization hash ac6b3dd68b4f6500e805bf56777250fa parsl.dataflow.memoization INFO: Task 3208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3208 try 0 launched on executor multi with executor id 3209 parsl.dataflow.dflow INFO: Standard output for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5ca0df74-ad1b-465e-95ee-d654dc279e2b_isrForCrosstalkSources_855395_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5ca0df74-ad1b-465e-95ee-d654dc279e2b_isrForCrosstalkSources_855395_17.stderr parsl.dataflow.dflow DEBUG: Task 3209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3209 has memoization hash a1fead7ce76ebf556f6444b5c251ead8 parsl.dataflow.memoization INFO: Task 3209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3209 try 0 launched on executor multi with executor id 3210 parsl.dataflow.dflow INFO: Standard output for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c05924d7-effd-4459-a74d-ad1ca096ae3f_isrForCrosstalkSources_855418_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c05924d7-effd-4459-a74d-ad1ca096ae3f_isrForCrosstalkSources_855418_16.stderr parsl.dataflow.dflow DEBUG: Task 3210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3210 has memoization hash 8754b03d7a48b217140e30416b31c584 parsl.dataflow.memoization INFO: Task 3210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3210 try 0 launched on executor multi with executor id 3211 parsl.dataflow.dflow INFO: Standard output for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6cdc3705-e164-48e4-b8f6-dd789c590a05_isrForCrosstalkSources_855417_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6cdc3705-e164-48e4-b8f6-dd789c590a05_isrForCrosstalkSources_855417_1.stderr parsl.dataflow.dflow DEBUG: Task 3211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3211 has memoization hash c3404eec7790213bdebcceba95b752c2 parsl.dataflow.memoization INFO: Task 3211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3211 try 0 launched on executor multi with executor id 3212 parsl.dataflow.dflow INFO: Standard output for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e75126fc-c7fc-4052-b59a-57b8d7f44e98_isrForCrosstalkSources_855445_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e75126fc-c7fc-4052-b59a-57b8d7f44e98_isrForCrosstalkSources_855445_36.stderr parsl.dataflow.dflow DEBUG: Task 3212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3212 has memoization hash 38cee0b0800e6a2062944827883a5326 parsl.dataflow.memoization INFO: Task 3212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3212 try 0 launched on executor multi with executor id 3213 parsl.dataflow.dflow INFO: Standard output for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c1b84101-743f-4a65-ab9d-7c110ca7fe11_isrForCrosstalkSources_855421_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c1b84101-743f-4a65-ab9d-7c110ca7fe11_isrForCrosstalkSources_855421_11.stderr parsl.dataflow.dflow DEBUG: Task 3213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3213 has memoization hash 190cbeb1595c7525bf621cb1997fb8cb parsl.dataflow.memoization INFO: Task 3213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3213 try 0 launched on executor multi with executor id 3214 parsl.dataflow.dflow INFO: Standard output for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/12ada047-5aee-430d-b7c8-57ac303eb900_isrForCrosstalkSources_855409_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/12ada047-5aee-430d-b7c8-57ac303eb900_isrForCrosstalkSources_855409_47.stderr parsl.dataflow.dflow DEBUG: Task 3214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3214 has memoization hash 4ddf7ac90be41d6f4046e3a2ede224b2 parsl.dataflow.memoization INFO: Task 3214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3214 try 0 launched on executor multi with executor id 3215 parsl.dataflow.dflow INFO: Standard output for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/81806efd-31cb-47e1-8979-2644d4c6924a_isrForCrosstalkSources_855415_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/81806efd-31cb-47e1-8979-2644d4c6924a_isrForCrosstalkSources_855415_61.stderr parsl.dataflow.dflow DEBUG: Task 3215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3215 has memoization hash 35d6341c7e43f08253d59a43a8e4c6c6 parsl.dataflow.memoization INFO: Task 3215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3215 try 0 launched on executor multi with executor id 3216 parsl.dataflow.dflow INFO: Standard output for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e75f4c15-9c56-4af2-9c74-27c4648ff9ab_isrForCrosstalkSources_855423_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e75f4c15-9c56-4af2-9c74-27c4648ff9ab_isrForCrosstalkSources_855423_12.stderr parsl.dataflow.dflow DEBUG: Task 3216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3216 has memoization hash f5423d9c7bb1dec5287d9bf508330246 parsl.dataflow.memoization INFO: Task 3216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3216 try 0 launched on executor multi with executor id 3217 parsl.dataflow.dflow INFO: Standard output for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0bf3b81d-b0a1-467a-a3f7-cf677f301da1_isrForCrosstalkSources_855426_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0bf3b81d-b0a1-467a-a3f7-cf677f301da1_isrForCrosstalkSources_855426_32.stderr parsl.dataflow.dflow DEBUG: Task 3217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3217 has memoization hash cc4b2d95d6d0697ae5d59a36117f8983 parsl.dataflow.memoization INFO: Task 3217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3217 try 0 launched on executor multi with executor id 3218 parsl.dataflow.dflow INFO: Standard output for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2c228c88-57ea-4181-a5a9-0303b75bc56b_isrForCrosstalkSources_855392_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2c228c88-57ea-4181-a5a9-0303b75bc56b_isrForCrosstalkSources_855392_47.stderr parsl.dataflow.dflow DEBUG: Task 3218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3218 has memoization hash 6a0c7ae63b05175bd6df04c76d7ad48b parsl.dataflow.memoization INFO: Task 3218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3218 try 0 launched on executor multi with executor id 3219 parsl.dataflow.dflow INFO: Standard output for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a31cc8-b4d5-4f60-99d4-35cfb35a5697_isrForCrosstalkSources_855391_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a31cc8-b4d5-4f60-99d4-35cfb35a5697_isrForCrosstalkSources_855391_58.stderr parsl.dataflow.dflow DEBUG: Task 3219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3219 has memoization hash e044e95019ab39a3ac845039da7ba93e parsl.dataflow.memoization INFO: Task 3219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3219 try 0 launched on executor multi with executor id 3220 parsl.dataflow.dflow INFO: Standard output for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/b2a6fa28-2ea7-4397-b2b9-74be16c7b4e3_isrForCrosstalkSources_855418_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/b2a6fa28-2ea7-4397-b2b9-74be16c7b4e3_isrForCrosstalkSources_855418_52.stderr parsl.dataflow.dflow DEBUG: Task 3220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3220 has memoization hash dbdb3365b6d99ff65fbac108ef0646f8 parsl.dataflow.memoization INFO: Task 3220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3220 try 0 launched on executor multi with executor id 3221 parsl.dataflow.dflow INFO: Standard output for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a9dcd3d-0d81-427c-877a-a551b3563df2_isrForCrosstalkSources_855409_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a9dcd3d-0d81-427c-877a-a551b3563df2_isrForCrosstalkSources_855409_10.stderr parsl.dataflow.dflow DEBUG: Task 3221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3221 has memoization hash 8c213045efc5684dbe39ddaa3314533d parsl.dataflow.memoization INFO: Task 3221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3221 try 0 launched on executor multi with executor id 3222 parsl.dataflow.dflow INFO: Standard output for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bc32e6f3-66a7-49de-bcbe-ec0532ef82b5_isrForCrosstalkSources_855405_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bc32e6f3-66a7-49de-bcbe-ec0532ef82b5_isrForCrosstalkSources_855405_41.stderr parsl.dataflow.dflow DEBUG: Task 3222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3222 has memoization hash 8c7880ff6937a65890d67a935f203ef6 parsl.dataflow.memoization INFO: Task 3222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:33 proc_lsst.multi:170 [INFO] [multi] job_id 16964169 proc_lsst.multi INFO: [multi] job_id 16964169 2024-03-09 14:51:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-09 14:51:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964169 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964169 parsl.executors.status_handling DEBUG: Launched block 143 on executor multi with job ID 16964169 parsl.executors.status_handling INFO: Allocated block ID 144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-144 2024-03-09 14:51:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 2024-03-09 14:51:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-09 14:51:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3222 try 0 launched on executor multi with executor id 3223 parsl.dataflow.dflow INFO: Standard output for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fa784283-9a02-45ac-a008-562c75d4a439_isrForCrosstalkSources_855418_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fa784283-9a02-45ac-a008-562c75d4a439_isrForCrosstalkSources_855418_1.stderr parsl.dataflow.dflow DEBUG: Task 3223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3223 has memoization hash 13083f73a916915eed36bb5f2bd2dae1 parsl.dataflow.memoization INFO: Task 3223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3223 try 0 launched on executor multi with executor id 3224 parsl.dataflow.dflow INFO: Standard output for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f85253cd-caea-47ba-9d52-96d6f6ff13cc_isrForCrosstalkSources_855454_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f85253cd-caea-47ba-9d52-96d6f6ff13cc_isrForCrosstalkSources_855454_57.stderr parsl.dataflow.dflow DEBUG: Task 3224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3224 has memoization hash 5a9e516118077d17e21dd7af82a289b7 parsl.dataflow.memoization INFO: Task 3224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3224 try 0 launched on executor multi with executor id 3225 parsl.dataflow.dflow INFO: Standard output for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b199ca80-65f3-4dfa-986d-f4fb4f9061ea_isrForCrosstalkSources_855424_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b199ca80-65f3-4dfa-986d-f4fb4f9061ea_isrForCrosstalkSources_855424_53.stderr parsl.dataflow.dflow DEBUG: Task 3225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3225 has memoization hash 08022ff6b6c1b4dcfaa3acfad57e707f parsl.dataflow.memoization INFO: Task 3225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3225 try 0 launched on executor multi with executor id 3226 parsl.dataflow.dflow INFO: Standard output for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a41bf571-cd49-4591-8719-14286d9aa538_isrForCrosstalkSources_855390_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a41bf571-cd49-4591-8719-14286d9aa538_isrForCrosstalkSources_855390_33.stderr parsl.dataflow.dflow DEBUG: Task 3226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3226 has memoization hash aea7c1b8902f3ddd72af8a3f746a97e0 parsl.dataflow.memoization INFO: Task 3226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3226 try 0 launched on executor multi with executor id 3227 parsl.dataflow.dflow INFO: Standard output for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d95f3e71-5664-453b-820c-7e5b4543a7e6_isrForCrosstalkSources_855386_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d95f3e71-5664-453b-820c-7e5b4543a7e6_isrForCrosstalkSources_855386_55.stderr parsl.dataflow.dflow DEBUG: Task 3227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3227 has memoization hash adc14b354130b267513f82ef39f1e532 parsl.dataflow.memoization INFO: Task 3227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3227 try 0 launched on executor multi with executor id 3228 parsl.dataflow.dflow INFO: Standard output for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/48ebe5a6-102a-40a9-b4a8-75f5bd224caa_isrForCrosstalkSources_855397_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/48ebe5a6-102a-40a9-b4a8-75f5bd224caa_isrForCrosstalkSources_855397_8.stderr parsl.dataflow.dflow DEBUG: Task 3228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3228 has memoization hash 89a5d57e3f4c0d836cadc1db2f44fb7b parsl.dataflow.memoization INFO: Task 3228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3228 try 0 launched on executor multi with executor id 3229 parsl.dataflow.dflow INFO: Standard output for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/82342c8c-1628-4f59-a68b-6e71b77bd72b_isrForCrosstalkSources_855435_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/82342c8c-1628-4f59-a68b-6e71b77bd72b_isrForCrosstalkSources_855435_37.stderr parsl.dataflow.dflow DEBUG: Task 3229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3229 has memoization hash aaf6387dc0f986611e5bc01f32299e5e parsl.dataflow.memoization INFO: Task 3229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3229 try 0 launched on executor multi with executor id 3230 parsl.dataflow.dflow INFO: Standard output for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f085d25f-fc94-48c0-b06a-7f785bd70bd9_isrForCrosstalkSources_855458_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f085d25f-fc94-48c0-b06a-7f785bd70bd9_isrForCrosstalkSources_855458_47.stderr parsl.dataflow.dflow DEBUG: Task 3230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3230 has memoization hash 57a9aaef4f63f841da133ca40d83be46 parsl.dataflow.memoization INFO: Task 3230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d3c344bce0e7159cebc48e2ae145ea7 with result from task 510 parsl.dataflow.dflow INFO: Parsl task 3230 try 0 launched on executor multi with executor id 3231 parsl.dataflow.dflow INFO: Standard output for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/d770272a-b7cc-4b92-b647-41d377fef552_isrForCrosstalkSources_855413_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/d770272a-b7cc-4b92-b647-41d377fef552_isrForCrosstalkSources_855413_43.stderr parsl.dataflow.dflow DEBUG: Task 3231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3231 has memoization hash e4fd932c39c88f285dc818fd3275de11 parsl.dataflow.memoization INFO: Task 3231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b2b39faf-9c4b-4506-87db-050df079640e_isrForCrosstalkSources_855380_13.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b2b39faf-9c4b-4506-87db-050df079640e_isrForCrosstalkSources_855380_13.stderr parsl.dataflow.dflow INFO: Parsl task 3231 try 0 launched on executor multi with executor id 3232 parsl.dataflow.dflow INFO: Standard output for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/71f63767-67ca-4b66-b433-23d2039876d8_isrForCrosstalkSources_855379_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/71f63767-67ca-4b66-b433-23d2039876d8_isrForCrosstalkSources_855379_18.stderr parsl.dataflow.dflow DEBUG: Task 3232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3232 has memoization hash 6b4d6be4bb60054278d854ff2e4093fa parsl.dataflow.memoization INFO: Task 3232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3232 try 0 launched on executor multi with executor id 3233 parsl.dataflow.dflow INFO: Standard output for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f67af9af-f81e-4531-9e68-62a4bfdaa865_isrForCrosstalkSources_855384_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f67af9af-f81e-4531-9e68-62a4bfdaa865_isrForCrosstalkSources_855384_10.stderr parsl.dataflow.dflow DEBUG: Task 3233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3233 has memoization hash 259bd14771ef4403d73dfef1eaa939e9 parsl.dataflow.memoization INFO: Task 3233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3233 try 0 launched on executor multi with executor id 3234 parsl.dataflow.dflow INFO: Standard output for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/722d0f14-1333-4133-b0bd-bc3ebce8487d_isrForCrosstalkSources_855388_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/722d0f14-1333-4133-b0bd-bc3ebce8487d_isrForCrosstalkSources_855388_47.stderr parsl.dataflow.dflow DEBUG: Task 3234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3234 has memoization hash e9fdf3851f4c9bb0cd88ff0fa7a7fda2 parsl.dataflow.memoization INFO: Task 3234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3234 try 0 launched on executor multi with executor id 3235 parsl.dataflow.dflow INFO: Standard output for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a4d96484-14cf-41e8-9409-0c5a418bc2cc_isrForCrosstalkSources_855421_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a4d96484-14cf-41e8-9409-0c5a418bc2cc_isrForCrosstalkSources_855421_29.stderr parsl.dataflow.dflow DEBUG: Task 3235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3235 has memoization hash 5a171aa306502bf9e866a75f56419b64 parsl.dataflow.memoization INFO: Task 3235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3235 try 0 launched on executor multi with executor id 3236 parsl.dataflow.dflow INFO: Standard output for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2f298ecd-5578-4216-9a85-75e0c299c17f_isrForCrosstalkSources_855457_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2f298ecd-5578-4216-9a85-75e0c299c17f_isrForCrosstalkSources_855457_58.stderr parsl.dataflow.dflow DEBUG: Task 3236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3236 has memoization hash b1cf5779dafa0514ef29b9322171a9be parsl.dataflow.memoization INFO: Task 3236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3236 try 0 launched on executor multi with executor id 3237 parsl.dataflow.dflow INFO: Standard output for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/182de1bc-f421-40c9-a3c0-602eb6bc4bf0_isrForCrosstalkSources_855444_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/182de1bc-f421-40c9-a3c0-602eb6bc4bf0_isrForCrosstalkSources_855444_48.stderr parsl.dataflow.dflow DEBUG: Task 3237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3237 has memoization hash f1dc8f4ef89710777be63421da6d0291 parsl.dataflow.memoization INFO: Task 3237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3237 try 0 launched on executor multi with executor id 3238 parsl.dataflow.dflow INFO: Standard output for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/05f2a6e2-c175-4e89-a8bd-4153c69fddfb_isrForCrosstalkSources_855435_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/05f2a6e2-c175-4e89-a8bd-4153c69fddfb_isrForCrosstalkSources_855435_24.stderr parsl.dataflow.dflow DEBUG: Task 3238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3238 has memoization hash a7e2ba6ab998ab6ca728a0221993c0e4 parsl.dataflow.memoization INFO: Task 3238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3238 try 0 launched on executor multi with executor id 3239 parsl.dataflow.dflow INFO: Standard output for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/03a54ed6-ddff-413f-a546-4ef396d2485c_isrForCrosstalkSources_855390_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/03a54ed6-ddff-413f-a546-4ef396d2485c_isrForCrosstalkSources_855390_41.stderr parsl.dataflow.dflow DEBUG: Task 3239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3239 has memoization hash 7fb59274a4d85fbc13b96991e9b62752 parsl.dataflow.memoization INFO: Task 3239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3239 try 0 launched on executor multi with executor id 3240 parsl.dataflow.dflow INFO: Standard output for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/dd61b315-e3a1-415a-852d-5f24d6094767_isrForCrosstalkSources_855449_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/dd61b315-e3a1-415a-852d-5f24d6094767_isrForCrosstalkSources_855449_45.stderr parsl.dataflow.dflow DEBUG: Task 3240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3240 has memoization hash e50f5ae985b03230e33ed90dc320990b parsl.dataflow.memoization INFO: Task 3240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3240 try 0 launched on executor multi with executor id 3241 parsl.dataflow.dflow INFO: Standard output for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d86ef9a2-be1d-4639-a97d-0034c8b6207e_isrForCrosstalkSources_855429_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d86ef9a2-be1d-4639-a97d-0034c8b6207e_isrForCrosstalkSources_855429_32.stderr parsl.dataflow.dflow DEBUG: Task 3241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3241 has memoization hash 2105285125aad0332ede01d9e980d699 parsl.dataflow.memoization INFO: Task 3241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3241 try 0 launched on executor multi with executor id 3242 parsl.dataflow.dflow INFO: Standard output for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e804c7ee-adbb-4106-a515-86f9ff88ca12_isrForCrosstalkSources_855382_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e804c7ee-adbb-4106-a515-86f9ff88ca12_isrForCrosstalkSources_855382_57.stderr parsl.dataflow.dflow DEBUG: Task 3242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3242 has memoization hash c25147841631ecf8c231ed418464af47 parsl.dataflow.memoization INFO: Task 3242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3242 try 0 launched on executor multi with executor id 3243 parsl.dataflow.dflow INFO: Standard output for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/60826d6a-b235-4622-939d-93f808a52157_isrForCrosstalkSources_855388_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/60826d6a-b235-4622-939d-93f808a52157_isrForCrosstalkSources_855388_25.stderr parsl.dataflow.dflow DEBUG: Task 3243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3243 has memoization hash 0b0f9e731f88fb657dd8d388c174ebdf parsl.dataflow.memoization INFO: Task 3243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3243 try 0 launched on executor multi with executor id 3244 parsl.dataflow.dflow INFO: Standard output for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7d6bc5f7-7ea2-44ba-a386-fc4cf159ac95_isrForCrosstalkSources_855380_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7d6bc5f7-7ea2-44ba-a386-fc4cf159ac95_isrForCrosstalkSources_855380_24.stderr parsl.dataflow.dflow DEBUG: Task 3244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3244 has memoization hash 41e9fc3097d958c27913aa48fb2a7f73 parsl.dataflow.memoization INFO: Task 3244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3244 try 0 launched on executor multi with executor id 3245 parsl.dataflow.dflow INFO: Standard output for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/531fa6b5-6af4-4d1b-9b0c-75863d646999_isrForCrosstalkSources_855429_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/531fa6b5-6af4-4d1b-9b0c-75863d646999_isrForCrosstalkSources_855429_21.stderr parsl.dataflow.dflow DEBUG: Task 3245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3245 has memoization hash 5a5dccc61439221fc3ad4716d8b39525 parsl.dataflow.memoization INFO: Task 3245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3245 try 0 launched on executor multi with executor id 3246 parsl.dataflow.dflow INFO: Standard output for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8dc2818f-4db5-4aa4-874e-2e5066797d5a_isrForCrosstalkSources_855453_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8dc2818f-4db5-4aa4-874e-2e5066797d5a_isrForCrosstalkSources_855453_34.stderr parsl.dataflow.dflow DEBUG: Task 3246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3246 has memoization hash ed27b71f815e98155f990881f544506e parsl.dataflow.memoization INFO: Task 3246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3246 try 0 launched on executor multi with executor id 3247 parsl.dataflow.dflow INFO: Standard output for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66f053dd-54fd-45af-9fab-58bb78f1a522_isrForCrosstalkSources_855390_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66f053dd-54fd-45af-9fab-58bb78f1a522_isrForCrosstalkSources_855390_61.stderr parsl.dataflow.dflow DEBUG: Task 3247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3247 has memoization hash c3594bebfd26e08c3089c5411779d6e8 parsl.dataflow.memoization INFO: Task 3247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3247 try 0 launched on executor multi with executor id 3248 parsl.dataflow.dflow INFO: Standard output for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d09f6af0-e994-446f-8c06-4969ed03af12_isrForCrosstalkSources_855417_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d09f6af0-e994-446f-8c06-4969ed03af12_isrForCrosstalkSources_855417_21.stderr parsl.dataflow.dflow DEBUG: Task 3248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3248 has memoization hash bd7994351275e11afeab30ae4c500eb0 parsl.dataflow.memoization INFO: Task 3248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3248 try 0 launched on executor multi with executor id 3249 parsl.dataflow.dflow INFO: Standard output for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/21b88370-a577-4dda-ba3a-a323d21263e9_isrForCrosstalkSources_855445_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/21b88370-a577-4dda-ba3a-a323d21263e9_isrForCrosstalkSources_855445_53.stderr parsl.dataflow.dflow DEBUG: Task 3249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3249 has memoization hash 8a7584782297d98a6fc7427b1a1a99f7 parsl.dataflow.memoization INFO: Task 3249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3249 try 0 launched on executor multi with executor id 3250 parsl.dataflow.dflow INFO: Standard output for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3d81103c-db4f-4eda-aabf-555fc04bdbe7_isrForCrosstalkSources_855390_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3d81103c-db4f-4eda-aabf-555fc04bdbe7_isrForCrosstalkSources_855390_57.stderr parsl.dataflow.dflow DEBUG: Task 3250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3250 has memoization hash ca7f102bcedfe8c924dce5f9155023d6 parsl.dataflow.memoization INFO: Task 3250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3250 try 0 launched on executor multi with executor id 3251 parsl.dataflow.dflow INFO: Standard output for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/183f17c2-4bfc-4d26-9001-7205f4206992_isrForCrosstalkSources_855390_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/183f17c2-4bfc-4d26-9001-7205f4206992_isrForCrosstalkSources_855390_39.stderr parsl.dataflow.dflow DEBUG: Task 3251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3251 has memoization hash 03c3c545f8ea2ca245f6df662829b137 parsl.dataflow.memoization INFO: Task 3251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3251 try 0 launched on executor multi with executor id 3252 parsl.dataflow.dflow INFO: Standard output for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cd9e5c1d-5b40-4c77-bc4b-150feb546ee4_isrForCrosstalkSources_855409_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cd9e5c1d-5b40-4c77-bc4b-150feb546ee4_isrForCrosstalkSources_855409_32.stderr parsl.dataflow.dflow DEBUG: Task 3252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3252 has memoization hash 166f4f7f39c908fa22ed5fe1f792b3d8 parsl.dataflow.memoization INFO: Task 3252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3252 try 0 launched on executor multi with executor id 3253 parsl.dataflow.dflow INFO: Standard output for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/090d2c07-7edd-45eb-902e-bed49b82a37a_isrForCrosstalkSources_855432_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/090d2c07-7edd-45eb-902e-bed49b82a37a_isrForCrosstalkSources_855432_9.stderr parsl.dataflow.dflow DEBUG: Task 3253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3253 has memoization hash 53b42505f21b447de559c180474740f2 parsl.dataflow.memoization INFO: Task 3253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3253 try 0 launched on executor multi with executor id 3254 parsl.dataflow.dflow INFO: Standard output for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0aaa98c7-d054-4565-b89f-e2d6a371a7ac_isrForCrosstalkSources_855417_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0aaa98c7-d054-4565-b89f-e2d6a371a7ac_isrForCrosstalkSources_855417_35.stderr parsl.dataflow.dflow DEBUG: Task 3254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3254 has memoization hash 99756d5ff861b8d1452182dee08a8512 parsl.dataflow.memoization INFO: Task 3254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3254 try 0 launched on executor multi with executor id 3255 parsl.dataflow.dflow INFO: Standard output for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/203f96ea-7ef8-465e-89f8-3388727b1fe2_isrForCrosstalkSources_855437_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/203f96ea-7ef8-465e-89f8-3388727b1fe2_isrForCrosstalkSources_855437_59.stderr parsl.dataflow.dflow DEBUG: Task 3255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3255 has memoization hash 9964029dc45be9be020725329c09b7f8 parsl.dataflow.memoization INFO: Task 3255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3255 try 0 launched on executor multi with executor id 3256 parsl.dataflow.dflow INFO: Standard output for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/14350e7b-5db1-43b1-9cf4-89fc90f0354e_isrForCrosstalkSources_855421_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/14350e7b-5db1-43b1-9cf4-89fc90f0354e_isrForCrosstalkSources_855421_12.stderr parsl.dataflow.dflow DEBUG: Task 3256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3256 has memoization hash 94b169295e47af1f13ac85a49f668a3f parsl.dataflow.memoization INFO: Task 3256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3256 try 0 launched on executor multi with executor id 3257 parsl.dataflow.dflow INFO: Standard output for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/91fba636-dcac-4a09-bd97-068bd1350272_isrForCrosstalkSources_855391_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/91fba636-dcac-4a09-bd97-068bd1350272_isrForCrosstalkSources_855391_51.stderr parsl.dataflow.dflow DEBUG: Task 3257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3257 has memoization hash 7dd08704b63cdfdbb2adcd6f58eb72bb parsl.dataflow.memoization INFO: Task 3257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3257 try 0 launched on executor multi with executor id 3258 parsl.dataflow.dflow INFO: Standard output for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2d10b200-29db-4e8b-abd3-fdbf00f15e7f_isrForCrosstalkSources_855432_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2d10b200-29db-4e8b-abd3-fdbf00f15e7f_isrForCrosstalkSources_855432_59.stderr parsl.dataflow.dflow DEBUG: Task 3258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3258 has memoization hash e548a5bfaae82081eb5a49088cc5154e parsl.dataflow.memoization INFO: Task 3258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3258 try 0 launched on executor multi with executor id 3259 parsl.dataflow.dflow INFO: Standard output for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/453bfdbf-375e-4186-9163-c1744be546a5_isrForCrosstalkSources_855420_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/453bfdbf-375e-4186-9163-c1744be546a5_isrForCrosstalkSources_855420_14.stderr parsl.dataflow.dflow DEBUG: Task 3259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3259 has memoization hash da502217ab43eadf0931f56429edaf00 parsl.dataflow.memoization INFO: Task 3259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3259 try 0 launched on executor multi with executor id 3260 parsl.dataflow.dflow INFO: Standard output for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed2be071-0051-4713-99dc-ae6f7e449a13_isrForCrosstalkSources_855455_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed2be071-0051-4713-99dc-ae6f7e449a13_isrForCrosstalkSources_855455_45.stderr parsl.dataflow.dflow DEBUG: Task 3260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3260 has memoization hash 6450a931ee5991448e438c1b5209e92f parsl.dataflow.memoization INFO: Task 3260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3260 try 0 launched on executor multi with executor id 3261 parsl.dataflow.dflow INFO: Standard output for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/77ca0aa2-61bc-4402-b96b-16071fd832b8_isrForCrosstalkSources_855399_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/77ca0aa2-61bc-4402-b96b-16071fd832b8_isrForCrosstalkSources_855399_62.stderr parsl.dataflow.dflow DEBUG: Task 3261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3261 has memoization hash cd9e4b0317e0f0c0ce5730fe8be7cc7c parsl.dataflow.memoization INFO: Task 3261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3261 try 0 launched on executor multi with executor id 3262 parsl.dataflow.dflow INFO: Standard output for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/dcdc45c5-bde7-421e-8180-96c2ff6c41cc_isrForCrosstalkSources_855410_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/dcdc45c5-bde7-421e-8180-96c2ff6c41cc_isrForCrosstalkSources_855410_31.stderr parsl.dataflow.dflow DEBUG: Task 3262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3262 has memoization hash e81b834a31db9a7bbc82a643367ca610 parsl.dataflow.memoization INFO: Task 3262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3262 try 0 launched on executor multi with executor id 3263 parsl.dataflow.dflow INFO: Standard output for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/4f54f249-ef20-4716-91e4-ac0cd64d1a3d_isrForCrosstalkSources_855431_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/4f54f249-ef20-4716-91e4-ac0cd64d1a3d_isrForCrosstalkSources_855431_28.stderr parsl.dataflow.dflow DEBUG: Task 3263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3263 has memoization hash d2dbeac2b93e186b2812d9afe8706280 parsl.dataflow.memoization INFO: Task 3263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3263 try 0 launched on executor multi with executor id 3264 parsl.dataflow.dflow INFO: Standard output for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cdbc92ab-6cef-4b46-8e53-b384abdb789f_isrForCrosstalkSources_855412_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cdbc92ab-6cef-4b46-8e53-b384abdb789f_isrForCrosstalkSources_855412_35.stderr parsl.dataflow.dflow DEBUG: Task 3264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3264 has memoization hash e9f7b98f150f979b6dfa1314bdf6d324 parsl.dataflow.memoization INFO: Task 3264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3264 try 0 launched on executor multi with executor id 3265 parsl.dataflow.dflow INFO: Standard output for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b9a228c2-4de1-4e93-b8ba-877293c45e3d_isrForCrosstalkSources_855454_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b9a228c2-4de1-4e93-b8ba-877293c45e3d_isrForCrosstalkSources_855454_19.stderr parsl.dataflow.dflow DEBUG: Task 3265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3265 has memoization hash cead9dc45af259c0d994eb52b33d83f5 parsl.dataflow.memoization INFO: Task 3265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3265 try 0 launched on executor multi with executor id 3266 parsl.dataflow.dflow INFO: Standard output for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e7ff9757-7191-4a82-ba64-d028a2847ebf_isrForCrosstalkSources_855409_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e7ff9757-7191-4a82-ba64-d028a2847ebf_isrForCrosstalkSources_855409_1.stderr parsl.dataflow.dflow DEBUG: Task 3266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3266 has memoization hash 4ad185b077b769f9776e73acc206d2c9 parsl.dataflow.memoization INFO: Task 3266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3266 try 0 launched on executor multi with executor id 3267 parsl.dataflow.dflow INFO: Standard output for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/577d4a80-272c-4522-86a4-25bdb6bda9e4_isrForCrosstalkSources_855400_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/577d4a80-272c-4522-86a4-25bdb6bda9e4_isrForCrosstalkSources_855400_34.stderr parsl.dataflow.dflow DEBUG: Task 3267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3267 has memoization hash cfe1b001d999af508d9fb0ee0b1ee70b parsl.dataflow.memoization INFO: Task 3267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3267 try 0 launched on executor multi with executor id 3268 parsl.dataflow.dflow INFO: Standard output for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6e0853a6-2a6b-47b0-9b1e-c0f8e6da957a_isrForCrosstalkSources_855411_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6e0853a6-2a6b-47b0-9b1e-c0f8e6da957a_isrForCrosstalkSources_855411_2.stderr parsl.dataflow.dflow DEBUG: Task 3268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3268 has memoization hash 91179648853a43d1eef7c0f5283a194a parsl.dataflow.memoization INFO: Task 3268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3268 try 0 launched on executor multi with executor id 3269 parsl.dataflow.dflow INFO: Standard output for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/edec96b1-9fae-4fd5-8ec9-48bea8875438_isrForCrosstalkSources_855399_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/edec96b1-9fae-4fd5-8ec9-48bea8875438_isrForCrosstalkSources_855399_23.stderr parsl.dataflow.dflow DEBUG: Task 3269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3269 has memoization hash fdcfcde40311c67457feeb46fb73a6a6 parsl.dataflow.memoization INFO: Task 3269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3269 try 0 launched on executor multi with executor id 3270 parsl.dataflow.dflow INFO: Standard output for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2b946e7b-9fc9-408b-9237-3da1e6094eb9_isrForCrosstalkSources_855434_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2b946e7b-9fc9-408b-9237-3da1e6094eb9_isrForCrosstalkSources_855434_24.stderr parsl.dataflow.dflow DEBUG: Task 3270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3270 has memoization hash 3b4650df60176552418b8da2c4dad47a parsl.dataflow.memoization INFO: Task 3270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3270 try 0 launched on executor multi with executor id 3271 parsl.dataflow.dflow INFO: Standard output for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a48cd0f6-7729-4428-8e8d-06ad827c31b3_isrForCrosstalkSources_855452_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a48cd0f6-7729-4428-8e8d-06ad827c31b3_isrForCrosstalkSources_855452_42.stderr parsl.dataflow.dflow DEBUG: Task 3271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3271 has memoization hash 461630d395f899e5673a1b3a9a59e403 parsl.dataflow.memoization INFO: Task 3271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3271 try 0 launched on executor multi with executor id 3272 parsl.dataflow.dflow INFO: Standard output for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/35a3c222-9ed2-480f-840f-98d7ce9c9297_isrForCrosstalkSources_855407_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/35a3c222-9ed2-480f-840f-98d7ce9c9297_isrForCrosstalkSources_855407_50.stderr parsl.dataflow.dflow DEBUG: Task 3272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3272 has memoization hash 397e6278ef217732f429cc6e87512c61 parsl.dataflow.memoization INFO: Task 3272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3272 try 0 launched on executor multi with executor id 3273 parsl.dataflow.dflow INFO: Standard output for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/160ca8d3-9622-42e6-b54d-5ce55b7bbbe8_isrForCrosstalkSources_855438_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/160ca8d3-9622-42e6-b54d-5ce55b7bbbe8_isrForCrosstalkSources_855438_57.stderr parsl.dataflow.dflow DEBUG: Task 3273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3273 has memoization hash d31388402bac38babf80364a8af6e436 parsl.dataflow.memoization INFO: Task 3273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3273 try 0 launched on executor multi with executor id 3274 parsl.dataflow.dflow INFO: Standard output for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/7a09041c-52db-4ec9-9d80-5aba81b5dcba_isrForCrosstalkSources_855420_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/7a09041c-52db-4ec9-9d80-5aba81b5dcba_isrForCrosstalkSources_855420_28.stderr parsl.dataflow.dflow DEBUG: Task 3274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3274 has memoization hash ec299d1e48c93ca10e4eec0ff80a3e79 parsl.dataflow.memoization INFO: Task 3274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3274 try 0 launched on executor multi with executor id 3275 parsl.dataflow.dflow INFO: Standard output for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d3e936d6-6163-4849-ab60-30ab27b390a4_isrForCrosstalkSources_855438_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d3e936d6-6163-4849-ab60-30ab27b390a4_isrForCrosstalkSources_855438_11.stderr parsl.dataflow.dflow DEBUG: Task 3275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3275 has memoization hash e2a9ee8f588fd453591022c0eb41e1e8 parsl.dataflow.memoization INFO: Task 3275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3275 try 0 launched on executor multi with executor id 3276 parsl.dataflow.dflow INFO: Standard output for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/2ad3dedf-3ef1-4307-9b11-b3aed27ebaf3_isrForCrosstalkSources_855446_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/2ad3dedf-3ef1-4307-9b11-b3aed27ebaf3_isrForCrosstalkSources_855446_52.stderr parsl.dataflow.dflow DEBUG: Task 3276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3276 has memoization hash 29a89ce91564b54ca304feb561a9a87a parsl.dataflow.memoization INFO: Task 3276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3276 try 0 launched on executor multi with executor id 3277 parsl.dataflow.dflow INFO: Standard output for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e3c96271-9e98-4811-a02d-90f80ccc1287_isrForCrosstalkSources_855435_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e3c96271-9e98-4811-a02d-90f80ccc1287_isrForCrosstalkSources_855435_8.stderr parsl.dataflow.dflow DEBUG: Task 3277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3277 has memoization hash a81f2f016a4faf23acf1210911a52e87 parsl.dataflow.memoization INFO: Task 3277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3277 try 0 launched on executor multi with executor id 3278 parsl.dataflow.dflow INFO: Standard output for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7dbe451c-9e32-4245-8c96-f3124b354111_isrForCrosstalkSources_855382_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7dbe451c-9e32-4245-8c96-f3124b354111_isrForCrosstalkSources_855382_23.stderr parsl.dataflow.dflow DEBUG: Task 3278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3278 has memoization hash 51a8cdc4d186d843a71332c0c600e1da parsl.dataflow.memoization INFO: Task 3278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3278 try 0 launched on executor multi with executor id 3279 parsl.dataflow.dflow INFO: Standard output for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea44618e-751c-4ece-9905-f5c10a54dcca_isrForCrosstalkSources_855382_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea44618e-751c-4ece-9905-f5c10a54dcca_isrForCrosstalkSources_855382_36.stderr parsl.dataflow.dflow DEBUG: Task 3279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3279 has memoization hash 39955790b5519373e6fc4995050ed8be parsl.dataflow.memoization INFO: Task 3279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3279 try 0 launched on executor multi with executor id 3280 parsl.dataflow.dflow INFO: Standard output for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/e683f363-f3e5-4d34-967c-36d087b1e99d_isrForCrosstalkSources_855442_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/e683f363-f3e5-4d34-967c-36d087b1e99d_isrForCrosstalkSources_855442_33.stderr parsl.dataflow.dflow DEBUG: Task 3280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3280 has memoization hash b154a898f1a2feb2e8c72db802bde7dd parsl.dataflow.memoization INFO: Task 3280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3280 try 0 launched on executor multi with executor id 3281 parsl.dataflow.dflow INFO: Standard output for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/36db1cbb-f095-4ac2-9283-2df3aa427d5a_isrForCrosstalkSources_855459_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/36db1cbb-f095-4ac2-9283-2df3aa427d5a_isrForCrosstalkSources_855459_40.stderr parsl.dataflow.dflow DEBUG: Task 3281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3281 has memoization hash b6d2c38ff0cea8e0c9a85952a19896b7 parsl.dataflow.memoization INFO: Task 3281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3281 try 0 launched on executor multi with executor id 3282 parsl.dataflow.dflow INFO: Standard output for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e0507c7-991f-4e4f-9e21-08e7c30ce0e3_isrForCrosstalkSources_855438_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e0507c7-991f-4e4f-9e21-08e7c30ce0e3_isrForCrosstalkSources_855438_9.stderr parsl.dataflow.dflow DEBUG: Task 3282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3282 has memoization hash 78af73007fc783bd443ec75cca4d2564 parsl.dataflow.memoization INFO: Task 3282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3282 try 0 launched on executor multi with executor id 3283 parsl.dataflow.dflow INFO: Standard output for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0a4d14ed-1502-4ed6-88f1-073f70ea5ccd_isrForCrosstalkSources_855405_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0a4d14ed-1502-4ed6-88f1-073f70ea5ccd_isrForCrosstalkSources_855405_17.stderr parsl.dataflow.dflow DEBUG: Task 3283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3283 has memoization hash 3a03b45590d13178b7a3ec9078f560dc parsl.dataflow.memoization INFO: Task 3283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3283 try 0 launched on executor multi with executor id 3284 parsl.dataflow.dflow INFO: Standard output for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eb9a7d0-6658-481b-b398-74da10b51297_isrForCrosstalkSources_855434_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eb9a7d0-6658-481b-b398-74da10b51297_isrForCrosstalkSources_855434_59.stderr parsl.dataflow.dflow DEBUG: Task 3284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3284 has memoization hash 44e88399ff603a81c830cea4440182d2 parsl.dataflow.memoization INFO: Task 3284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3284 try 0 launched on executor multi with executor id 3285 parsl.dataflow.dflow INFO: Standard output for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/abdd798f-a489-44c2-82cc-8a81041c2d4b_isrForCrosstalkSources_855396_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/abdd798f-a489-44c2-82cc-8a81041c2d4b_isrForCrosstalkSources_855396_55.stderr parsl.dataflow.dflow DEBUG: Task 3285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3285 has memoization hash 82b5b4122d86225c07a1ddd5537caa13 parsl.dataflow.memoization INFO: Task 3285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3285 try 0 launched on executor multi with executor id 3286 parsl.dataflow.dflow INFO: Standard output for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/24760688-6112-446b-af2b-eacbf0f0b025_isrForCrosstalkSources_855420_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/24760688-6112-446b-af2b-eacbf0f0b025_isrForCrosstalkSources_855420_48.stderr parsl.dataflow.dflow DEBUG: Task 3286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3286 has memoization hash 1107c29669bcc1e5be171bd88beaf797 parsl.dataflow.memoization INFO: Task 3286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3286 try 0 launched on executor multi with executor id 3287 parsl.dataflow.dflow INFO: Standard output for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ffbd624b-bc17-4dbf-b53d-885172eb654f_isrForCrosstalkSources_855398_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ffbd624b-bc17-4dbf-b53d-885172eb654f_isrForCrosstalkSources_855398_26.stderr parsl.dataflow.dflow DEBUG: Task 3287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3287 has memoization hash 33d81bbb14227f3f6e724cfc14b43a0a parsl.dataflow.memoization INFO: Task 3287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3287 try 0 launched on executor multi with executor id 3288 parsl.dataflow.dflow INFO: Standard output for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dad2ffd7-0bab-4de6-9679-cfd51bdee422_isrForCrosstalkSources_855418_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dad2ffd7-0bab-4de6-9679-cfd51bdee422_isrForCrosstalkSources_855418_26.stderr parsl.dataflow.dflow DEBUG: Task 3288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3288 has memoization hash daa2aab53344ed7c17551cd3cef299b3 parsl.dataflow.memoization INFO: Task 3288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3288 try 0 launched on executor multi with executor id 3289 parsl.dataflow.dflow INFO: Standard output for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/177ec516-102d-41f5-8cb5-a41a18c243dd_isrForCrosstalkSources_855455_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/177ec516-102d-41f5-8cb5-a41a18c243dd_isrForCrosstalkSources_855455_27.stderr parsl.dataflow.dflow DEBUG: Task 3289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3289 has memoization hash 3156662feb49ebd30e3ed7f70f9da440 parsl.dataflow.memoization INFO: Task 3289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3289 try 0 launched on executor multi with executor id 3290 parsl.dataflow.dflow INFO: Standard output for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/927f96ba-d510-476c-a656-d02ea7969808_isrForCrosstalkSources_855404_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/927f96ba-d510-476c-a656-d02ea7969808_isrForCrosstalkSources_855404_12.stderr parsl.dataflow.dflow DEBUG: Task 3290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3290 has memoization hash 5cf89ab3852867b3eca88ff404c298b0 parsl.dataflow.memoization INFO: Task 3290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3290 try 0 launched on executor multi with executor id 3291 parsl.dataflow.dflow INFO: Standard output for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/54932d80-3a26-4ff9-b768-7019f65053b2_isrForCrosstalkSources_855393_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/54932d80-3a26-4ff9-b768-7019f65053b2_isrForCrosstalkSources_855393_1.stderr parsl.dataflow.dflow DEBUG: Task 3291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3291 has memoization hash aa9bee3d00949492dc9fce3555f0df8a parsl.dataflow.memoization INFO: Task 3291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3291 try 0 launched on executor multi with executor id 3292 parsl.dataflow.dflow INFO: Standard output for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/8a6aaab0-90b2-4c84-982e-6944c2f8ef50_isrForCrosstalkSources_855405_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/8a6aaab0-90b2-4c84-982e-6944c2f8ef50_isrForCrosstalkSources_855405_33.stderr parsl.dataflow.dflow DEBUG: Task 3292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3292 has memoization hash a47eee0c531ae171332e5dc7b168fd05 parsl.dataflow.memoization INFO: Task 3292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3292 try 0 launched on executor multi with executor id 3293 parsl.dataflow.dflow INFO: Standard output for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/86e09342-1ec0-4e83-82c3-e45202c05f5e_isrForCrosstalkSources_855430_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/86e09342-1ec0-4e83-82c3-e45202c05f5e_isrForCrosstalkSources_855430_12.stderr parsl.dataflow.dflow DEBUG: Task 3293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3293 has memoization hash a04f6b7cd58de16f25e122ccf06fc580 parsl.dataflow.memoization INFO: Task 3293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3293 try 0 launched on executor multi with executor id 3294 parsl.dataflow.dflow INFO: Standard output for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d2bf9c0b-db94-4ffe-a4cc-b56f4502d375_isrForCrosstalkSources_855446_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d2bf9c0b-db94-4ffe-a4cc-b56f4502d375_isrForCrosstalkSources_855446_21.stderr parsl.dataflow.dflow DEBUG: Task 3294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3294 has memoization hash 14498b1ae78b1008d019b01d679343c4 parsl.dataflow.memoization INFO: Task 3294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3294 try 0 launched on executor multi with executor id 3295 parsl.dataflow.dflow INFO: Standard output for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4ebaab23-e92b-411c-8e71-9f7cb67e4aa7_isrForCrosstalkSources_855451_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4ebaab23-e92b-411c-8e71-9f7cb67e4aa7_isrForCrosstalkSources_855451_18.stderr parsl.dataflow.dflow DEBUG: Task 3295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3295 has memoization hash 32e27a9f4eaf8b236c8620f9a748704f parsl.dataflow.memoization INFO: Task 3295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3295 try 0 launched on executor multi with executor id 3296 parsl.dataflow.dflow INFO: Standard output for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46171d7c-d804-4393-bc58-87d9baf436f0_isrForCrosstalkSources_855419_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46171d7c-d804-4393-bc58-87d9baf436f0_isrForCrosstalkSources_855419_31.stderr parsl.dataflow.dflow DEBUG: Task 3296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3296 has memoization hash 7483eb6e0659ed55346b5cd37d9308a6 parsl.dataflow.memoization INFO: Task 3296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3296 try 0 launched on executor multi with executor id 3297 parsl.dataflow.dflow INFO: Standard output for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec9896af-2420-4be4-8b05-f07330c9f444_isrForCrosstalkSources_855391_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec9896af-2420-4be4-8b05-f07330c9f444_isrForCrosstalkSources_855391_22.stderr parsl.dataflow.dflow DEBUG: Task 3297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3297 has memoization hash 5a521b03cd1c04ddb29ff481abc24890 parsl.dataflow.memoization INFO: Task 3297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3297 try 0 launched on executor multi with executor id 3298 parsl.dataflow.dflow INFO: Standard output for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6565babd-aef0-4e70-8e1a-e28032c8600a_isrForCrosstalkSources_855380_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6565babd-aef0-4e70-8e1a-e28032c8600a_isrForCrosstalkSources_855380_20.stderr parsl.dataflow.dflow DEBUG: Task 3298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3298 has memoization hash 823732da5636908422190aaa30df20f3 parsl.dataflow.memoization INFO: Task 3298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3298 try 0 launched on executor multi with executor id 3299 parsl.dataflow.dflow INFO: Standard output for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/45f54443-35bd-4317-93f5-a59cf0e73ae4_isrForCrosstalkSources_855453_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/45f54443-35bd-4317-93f5-a59cf0e73ae4_isrForCrosstalkSources_855453_9.stderr parsl.dataflow.dflow DEBUG: Task 3299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3299 has memoization hash 67a1fbd105aff003c94ef10c33fbc9bf parsl.dataflow.memoization INFO: Task 3299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad263e856b19b28590e187b69acb65d8 with result from task 517 parsl.dataflow.dflow INFO: Parsl task 3299 try 0 launched on executor multi with executor id 3300 parsl.dataflow.dflow INFO: Standard output for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/7c9c86c1-e793-4ee5-8658-7bf9d96a5c68_isrForCrosstalkSources_855395_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/7c9c86c1-e793-4ee5-8658-7bf9d96a5c68_isrForCrosstalkSources_855395_13.stderr parsl.dataflow.dflow DEBUG: Task 3300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3300 has memoization hash fe4ab725b791b0769cc32d636409b4d0 parsl.dataflow.memoization INFO: Task 3300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3300 try 0 launched on executor multi with executor id 3301 parsl.dataflow.dflow INFO: Standard output for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f338996d-c0c1-4ee1-beef-2c0b243c5c6e_isrForCrosstalkSources_855404_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f338996d-c0c1-4ee1-beef-2c0b243c5c6e_isrForCrosstalkSources_855404_56.stderr parsl.dataflow.dflow DEBUG: Task 3301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3301 has memoization hash 7ef91110dc5eeef6372dc7cd38e5b4e2 parsl.dataflow.memoization INFO: Task 3301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3301 try 0 launched on executor multi with executor id 3302 parsl.dataflow.dflow INFO: Standard output for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/df80d7bc-c85d-4306-b3fa-f8f6ec0a3b4e_isrForCrosstalkSources_855427_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/df80d7bc-c85d-4306-b3fa-f8f6ec0a3b4e_isrForCrosstalkSources_855427_35.stderr parsl.dataflow.dflow DEBUG: Task 3302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3302 has memoization hash f721a729ebf3d64ef6d8cfb486f70f91 parsl.dataflow.memoization INFO: Task 3302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/af839781-1958-40fa-a6f0-6b2cf63504d3_isrForCrosstalkSources_855425_8.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/af839781-1958-40fa-a6f0-6b2cf63504d3_isrForCrosstalkSources_855425_8.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2614b24a8fc188f2198da36c05112c6 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ddc34dc5-914c-4fea-bda2-37e5218d0882_isrForCrosstalkSources_855379_35.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ddc34dc5-914c-4fea-bda2-37e5218d0882_isrForCrosstalkSources_855379_35.stderr parsl.dataflow.dflow INFO: Parsl task 3302 try 0 launched on executor multi with executor id 3303 parsl.dataflow.dflow INFO: Standard output for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3491356d-45b2-4bb0-9d8f-08d1ac06762e_isrForCrosstalkSources_855420_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3491356d-45b2-4bb0-9d8f-08d1ac06762e_isrForCrosstalkSources_855420_50.stderr parsl.dataflow.dflow DEBUG: Task 3303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3303 has memoization hash e747b9a254858efee844e4ec585207ad parsl.dataflow.memoization INFO: Task 3303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3303 try 0 launched on executor multi with executor id 3304 parsl.dataflow.dflow INFO: Standard output for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0940b16c-c90f-44c5-b5fc-b420fd90ad91_isrForCrosstalkSources_855407_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0940b16c-c90f-44c5-b5fc-b420fd90ad91_isrForCrosstalkSources_855407_28.stderr parsl.dataflow.dflow DEBUG: Task 3304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3304 has memoization hash 65a4b7cb3981e0538bb6c4ad18f2be0c parsl.dataflow.memoization INFO: Task 3304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3304 try 0 launched on executor multi with executor id 3305 parsl.dataflow.dflow INFO: Standard output for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/205f4a96-6b34-4924-868f-df8b74a70a9c_isrForCrosstalkSources_855399_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/205f4a96-6b34-4924-868f-df8b74a70a9c_isrForCrosstalkSources_855399_42.stderr parsl.dataflow.dflow DEBUG: Task 3305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3305 has memoization hash 0d1f28cf93dd0d041c99b552f2cf84a1 parsl.dataflow.memoization INFO: Task 3305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3305 try 0 launched on executor multi with executor id 3306 parsl.dataflow.dflow INFO: Standard output for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2f4618eb-7bdf-4c76-af6a-2ef0c568f00a_isrForCrosstalkSources_855398_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2f4618eb-7bdf-4c76-af6a-2ef0c568f00a_isrForCrosstalkSources_855398_42.stderr parsl.dataflow.dflow DEBUG: Task 3306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3306 has memoization hash b62c0a5f1b13549d408922954011a662 parsl.dataflow.memoization INFO: Task 3306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3306 try 0 launched on executor multi with executor id 3307 parsl.dataflow.dflow INFO: Standard output for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/111f4768-f00f-4d31-9a53-bef9c5faea42_isrForCrosstalkSources_855414_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/111f4768-f00f-4d31-9a53-bef9c5faea42_isrForCrosstalkSources_855414_8.stderr parsl.dataflow.dflow DEBUG: Task 3307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3307 has memoization hash 3c8be923fef3d460b5b843c1764a6267 parsl.dataflow.memoization INFO: Task 3307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f337705f49747a66e802767692acbdb with result from task 185 parsl.dataflow.dflow INFO: Parsl task 3307 try 0 launched on executor multi with executor id 3308 parsl.dataflow.dflow INFO: Standard output for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c9a6ccb3-5f20-4010-af03-f3d2abb4531d_isrForCrosstalkSources_855421_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c9a6ccb3-5f20-4010-af03-f3d2abb4531d_isrForCrosstalkSources_855421_26.stderr parsl.dataflow.dflow DEBUG: Task 3308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3308 has memoization hash 89bf9b43dd741e18f46e470c020c3dd5 parsl.dataflow.memoization INFO: Task 3308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3308 try 0 launched on executor multi with executor id 3309 parsl.dataflow.dflow INFO: Standard output for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/365eb78c-ff63-41c8-9c68-e94e091f4583_isrForCrosstalkSources_855402_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/365eb78c-ff63-41c8-9c68-e94e091f4583_isrForCrosstalkSources_855402_7.stderr parsl.dataflow.dflow DEBUG: Task 3309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3309 has memoization hash 043e3698fa7541a222fe409d3d4c3999 parsl.dataflow.memoization INFO: Task 3309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3309 try 0 launched on executor multi with executor id 3310 parsl.dataflow.dflow INFO: Standard output for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/89e618a8-b74d-4413-9b46-73c5884eb316_isrForCrosstalkSources_855441_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/89e618a8-b74d-4413-9b46-73c5884eb316_isrForCrosstalkSources_855441_38.stderr parsl.dataflow.dflow DEBUG: Task 3310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3310 has memoization hash 1aa1becbbf22094bd860e79a382d390d parsl.dataflow.memoization INFO: Task 3310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e4531a74-8ca4-423d-893e-7da425fef2ad_isrForCrosstalkSources_855416_39.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e4531a74-8ca4-423d-893e-7da425fef2ad_isrForCrosstalkSources_855416_39.stderr parsl.dataflow.dflow INFO: Parsl task 3310 try 0 launched on executor multi with executor id 3311 parsl.dataflow.dflow INFO: Standard output for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/112f10eb-0231-475f-94b9-f970cd9d98da_isrForCrosstalkSources_855459_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/112f10eb-0231-475f-94b9-f970cd9d98da_isrForCrosstalkSources_855459_16.stderr parsl.dataflow.dflow DEBUG: Task 3311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3311 has memoization hash 15126935e924ea7c6093e2cf94f5bc98 parsl.dataflow.memoization INFO: Task 3311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3311 try 0 launched on executor multi with executor id 3312 parsl.dataflow.dflow INFO: Standard output for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fe6c3c45-f3b1-4a7b-8e1d-0239a85b7fd5_isrForCrosstalkSources_855385_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fe6c3c45-f3b1-4a7b-8e1d-0239a85b7fd5_isrForCrosstalkSources_855385_29.stderr parsl.dataflow.dflow DEBUG: Task 3312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3312 has memoization hash 346d4658336f0369465ebcc918a06ae0 parsl.dataflow.memoization INFO: Task 3312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3312 try 0 launched on executor multi with executor id 3313 parsl.dataflow.dflow INFO: Standard output for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/65117874-6b8e-49ca-9379-a6afcda90585_isrForCrosstalkSources_855457_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/65117874-6b8e-49ca-9379-a6afcda90585_isrForCrosstalkSources_855457_39.stderr parsl.dataflow.dflow DEBUG: Task 3313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3313 has memoization hash 1160b7ba22f24bbfa38b606ea5db0f99 parsl.dataflow.memoization INFO: Task 3313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3313 try 0 launched on executor multi with executor id 3314 parsl.dataflow.dflow INFO: Standard output for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/77f0c514-8f89-450f-bc1e-0cd285c92b58_isrForCrosstalkSources_855438_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/77f0c514-8f89-450f-bc1e-0cd285c92b58_isrForCrosstalkSources_855438_17.stderr parsl.dataflow.dflow DEBUG: Task 3314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3314 has memoization hash e4cdca9757a1d2c7b34375f3f6040023 parsl.dataflow.memoization INFO: Task 3314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3314 try 0 launched on executor multi with executor id 3315 parsl.dataflow.dflow INFO: Standard output for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b43d4b5c-8b36-4ba9-9d97-17ff7d607681_isrForCrosstalkSources_855448_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b43d4b5c-8b36-4ba9-9d97-17ff7d607681_isrForCrosstalkSources_855448_25.stderr parsl.dataflow.dflow DEBUG: Task 3315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3315 has memoization hash 6cb64ecce9e51644067c9f65eb318d4d parsl.dataflow.memoization INFO: Task 3315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2a86b67b9fa6eca592415e5634313d3 with result from task 501 parsl.dataflow.dflow INFO: Parsl task 3315 try 0 launched on executor multi with executor id 3316 parsl.dataflow.dflow INFO: Standard output for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcc0b76-2fc5-4a79-9c44-5571bbaa0027_isrForCrosstalkSources_855409_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcc0b76-2fc5-4a79-9c44-5571bbaa0027_isrForCrosstalkSources_855409_23.stderr parsl.dataflow.dflow DEBUG: Task 3316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3316 has memoization hash 4ee45eb9fe84596ea728e72c55479df4 parsl.dataflow.memoization INFO: Task 3316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3316 try 0 launched on executor multi with executor id 3317 parsl.dataflow.dflow INFO: Standard output for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/24d7cb09-956f-4c08-9491-7d728c52f3b6_isrForCrosstalkSources_855426_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/24d7cb09-956f-4c08-9491-7d728c52f3b6_isrForCrosstalkSources_855426_44.stderr parsl.dataflow.dflow DEBUG: Task 3317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3317 has memoization hash 81fa6e490e80de90dae4902a7aae2223 parsl.dataflow.memoization INFO: Task 3317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3317 try 0 launched on executor multi with executor id 3318 parsl.dataflow.dflow INFO: Standard output for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/bff0bb6d-7a32-44b3-a280-1d7cbf212e36_isrForCrosstalkSources_855404_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/bff0bb6d-7a32-44b3-a280-1d7cbf212e36_isrForCrosstalkSources_855404_46.stderr parsl.dataflow.dflow DEBUG: Task 3318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3318 has memoization hash 9e1d5b8e53584cbd8bd9bccd75e02f03 parsl.dataflow.memoization INFO: Task 3318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/48870a82-7911-4958-8459-cd30fa45c15d_isrForCrosstalkSources_855443_22.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/48870a82-7911-4958-8459-cd30fa45c15d_isrForCrosstalkSources_855443_22.stderr parsl.dataflow.dflow INFO: Parsl task 3318 try 0 launched on executor multi with executor id 3319 parsl.dataflow.dflow INFO: Standard output for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/16e40b65-7be4-4539-a079-cb5101eaaa34_isrForCrosstalkSources_855401_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/16e40b65-7be4-4539-a079-cb5101eaaa34_isrForCrosstalkSources_855401_22.stderr parsl.dataflow.dflow DEBUG: Task 3319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3319 has memoization hash fda0d79a961a169ce2012b0d602470ea parsl.dataflow.memoization INFO: Task 3319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3319 try 0 launched on executor multi with executor id 3320 parsl.dataflow.dflow INFO: Standard output for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/37c56ddf-5ed2-4a71-b4a0-89c67ea2006c_isrForCrosstalkSources_855393_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/37c56ddf-5ed2-4a71-b4a0-89c67ea2006c_isrForCrosstalkSources_855393_50.stderr parsl.dataflow.dflow DEBUG: Task 3320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3320 has memoization hash 598d6148dc5bba084f06156698eb592c parsl.dataflow.memoization INFO: Task 3320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3320 try 0 launched on executor multi with executor id 3321 parsl.dataflow.dflow INFO: Standard output for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/cbb83a5a-3046-4cd8-ace9-eefcd9016d10_isrForCrosstalkSources_855396_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/cbb83a5a-3046-4cd8-ace9-eefcd9016d10_isrForCrosstalkSources_855396_62.stderr parsl.dataflow.dflow DEBUG: Task 3321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3321 has memoization hash 910cca861ea88a4292b16c764db7fade parsl.dataflow.memoization INFO: Task 3321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3321 try 0 launched on executor multi with executor id 3322 parsl.dataflow.dflow INFO: Standard output for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f262d202-cb53-4bd9-8df5-9c475ab1f501_isrForCrosstalkSources_855453_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f262d202-cb53-4bd9-8df5-9c475ab1f501_isrForCrosstalkSources_855453_6.stderr parsl.dataflow.dflow DEBUG: Task 3322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3322 has memoization hash 038d69a0315ae60a3515da256ec87b0e parsl.dataflow.memoization INFO: Task 3322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ad3be52f5a1e116c47008703807fcf with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c36a562e-cd5c-41fc-ae18-f585233d7409_isrForCrosstalkSources_855426_21.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c36a562e-cd5c-41fc-ae18-f585233d7409_isrForCrosstalkSources_855426_21.stderr parsl.dataflow.dflow INFO: Parsl task 3322 try 0 launched on executor multi with executor id 3323 parsl.dataflow.dflow INFO: Standard output for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e428c86d-57a7-41ce-859d-20110be2c55e_isrForCrosstalkSources_855411_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e428c86d-57a7-41ce-859d-20110be2c55e_isrForCrosstalkSources_855411_28.stderr parsl.dataflow.dflow DEBUG: Task 3323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3323 has memoization hash 4454e568a6c2d9fdb1f9ebadbc6104e8 parsl.dataflow.memoization INFO: Task 3323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3323 try 0 launched on executor multi with executor id 3324 parsl.dataflow.dflow INFO: Standard output for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ea92af3f-4923-4fb1-879f-98b52c785648_isrForCrosstalkSources_855459_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ea92af3f-4923-4fb1-879f-98b52c785648_isrForCrosstalkSources_855459_60.stderr parsl.dataflow.dflow DEBUG: Task 3324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3324 has memoization hash c588fda30ac590d265d8c10f31c375f0 parsl.dataflow.memoization INFO: Task 3324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3324 try 0 launched on executor multi with executor id 3325 parsl.dataflow.dflow INFO: Standard output for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/19a9d313-ade1-4d42-aa60-306b254669f6_isrForCrosstalkSources_855443_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/19a9d313-ade1-4d42-aa60-306b254669f6_isrForCrosstalkSources_855443_41.stderr parsl.dataflow.dflow DEBUG: Task 3325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3325 has memoization hash 43a9623cf626f3829c6ad5922ce1871a parsl.dataflow.memoization INFO: Task 3325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3325 try 0 launched on executor multi with executor id 3326 parsl.dataflow.dflow INFO: Standard output for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/46fdd612-820d-4183-a519-730c0ec547e6_isrForCrosstalkSources_855385_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/46fdd612-820d-4183-a519-730c0ec547e6_isrForCrosstalkSources_855385_41.stderr parsl.dataflow.dflow DEBUG: Task 3326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3326 has memoization hash fb673b19bf9b34d3abeed1d81a88e3de parsl.dataflow.memoization INFO: Task 3326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3326 try 0 launched on executor multi with executor id 3327 parsl.dataflow.dflow INFO: Standard output for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6d4b2595-12b7-4b5e-85bb-5b9ec084790f_isrForCrosstalkSources_855421_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6d4b2595-12b7-4b5e-85bb-5b9ec084790f_isrForCrosstalkSources_855421_6.stderr parsl.dataflow.dflow DEBUG: Task 3327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3327 has memoization hash 0438d112ebab515ef55a731cd29d849f parsl.dataflow.memoization INFO: Task 3327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3327 try 0 launched on executor multi with executor id 3328 parsl.dataflow.dflow INFO: Standard output for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/29a520b1-18b7-44d5-889c-630966fead6b_isrForCrosstalkSources_855457_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/29a520b1-18b7-44d5-889c-630966fead6b_isrForCrosstalkSources_855457_25.stderr parsl.dataflow.dflow DEBUG: Task 3328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3328 has memoization hash 292d649cdb7256c3a58a526f9e19a79f parsl.dataflow.memoization INFO: Task 3328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3328 try 0 launched on executor multi with executor id 3329 parsl.dataflow.dflow INFO: Standard output for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/cf2baab9-89ff-40b0-91bd-c3ab17dcce72_isrForCrosstalkSources_855438_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/cf2baab9-89ff-40b0-91bd-c3ab17dcce72_isrForCrosstalkSources_855438_35.stderr parsl.dataflow.dflow DEBUG: Task 3329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3329 has memoization hash fe6b13b0ed146c74ae4240627cb99623 parsl.dataflow.memoization INFO: Task 3329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3329 try 0 launched on executor multi with executor id 3330 parsl.dataflow.dflow INFO: Standard output for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f94bbcb-eaf2-439d-990c-0151cf42b738_isrForCrosstalkSources_855415_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f94bbcb-eaf2-439d-990c-0151cf42b738_isrForCrosstalkSources_855415_41.stderr parsl.dataflow.dflow DEBUG: Task 3330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3330 has memoization hash 551927b34fb40cc5c09ac1baa64d250e parsl.dataflow.memoization INFO: Task 3330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3330 try 0 launched on executor multi with executor id 3331 parsl.dataflow.dflow INFO: Standard output for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3c426b99-c7d7-4c82-b1aa-ad387efcf019_isrForCrosstalkSources_855405_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3c426b99-c7d7-4c82-b1aa-ad387efcf019_isrForCrosstalkSources_855405_55.stderr parsl.dataflow.dflow DEBUG: Task 3331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3331 has memoization hash 6052bd26c98173b7d3f19d52e59eb1c9 parsl.dataflow.memoization INFO: Task 3331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3331 try 0 launched on executor multi with executor id 3332 parsl.dataflow.dflow INFO: Standard output for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c338c74e-2b20-48d8-9759-4a910be38d4b_isrForCrosstalkSources_855442_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c338c74e-2b20-48d8-9759-4a910be38d4b_isrForCrosstalkSources_855442_18.stderr parsl.dataflow.dflow DEBUG: Task 3332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3332 has memoization hash fe9f60551a07e998d27dfbfa53fd4769 parsl.dataflow.memoization INFO: Task 3332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3332 try 0 launched on executor multi with executor id 3333 parsl.dataflow.dflow INFO: Standard output for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/106ca6c3-f4f4-46e6-afe2-4c7565a39d11_isrForCrosstalkSources_855414_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/106ca6c3-f4f4-46e6-afe2-4c7565a39d11_isrForCrosstalkSources_855414_20.stderr parsl.dataflow.dflow DEBUG: Task 3333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3333 has memoization hash 03d6398a94448818f09c503c29161605 parsl.dataflow.memoization INFO: Task 3333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3333 try 0 launched on executor multi with executor id 3334 parsl.dataflow.dflow INFO: Standard output for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1d8473fd-5d56-48b2-ad58-85e120e58ec1_isrForCrosstalkSources_855446_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1d8473fd-5d56-48b2-ad58-85e120e58ec1_isrForCrosstalkSources_855446_12.stderr parsl.dataflow.dflow DEBUG: Task 3334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3334 has memoization hash da4765bbac46ea9d44d652304e371d6d parsl.dataflow.memoization INFO: Task 3334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3334 try 0 launched on executor multi with executor id 3335 parsl.dataflow.dflow INFO: Standard output for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d7c6a38a-14cc-4882-a742-76d60faf9017_isrForCrosstalkSources_855438_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d7c6a38a-14cc-4882-a742-76d60faf9017_isrForCrosstalkSources_855438_4.stderr parsl.dataflow.dflow DEBUG: Task 3335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3335 has memoization hash 6167aa54134bf75f69c26941dc306650 parsl.dataflow.memoization INFO: Task 3335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3335 try 0 launched on executor multi with executor id 3336 parsl.dataflow.dflow INFO: Standard output for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c023d42e-59b1-41d9-8ade-1298a7d9456d_isrForCrosstalkSources_855456_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c023d42e-59b1-41d9-8ade-1298a7d9456d_isrForCrosstalkSources_855456_25.stderr parsl.dataflow.dflow DEBUG: Task 3336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3336 has memoization hash c88c1f502060bbce9acfeb3fcae6fd84 parsl.dataflow.memoization INFO: Task 3336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3336 try 0 launched on executor multi with executor id 3337 parsl.dataflow.dflow INFO: Standard output for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b9ff11c-c6bf-4b54-bef2-a0778e2379bc_isrForCrosstalkSources_855400_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b9ff11c-c6bf-4b54-bef2-a0778e2379bc_isrForCrosstalkSources_855400_15.stderr parsl.dataflow.dflow DEBUG: Task 3337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3337 has memoization hash ac7f38688a2739ad50204664f338d622 parsl.dataflow.memoization INFO: Task 3337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3337 try 0 launched on executor multi with executor id 3338 parsl.dataflow.dflow INFO: Standard output for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7e7039a7-6744-4675-b91f-65409ce8a0a2_isrForCrosstalkSources_855393_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7e7039a7-6744-4675-b91f-65409ce8a0a2_isrForCrosstalkSources_855393_10.stderr parsl.dataflow.dflow DEBUG: Task 3338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3338 has memoization hash 1f634e31eccc4ece2ee86a3b6ceab490 parsl.dataflow.memoization INFO: Task 3338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3338 try 0 launched on executor multi with executor id 3339 parsl.dataflow.dflow INFO: Standard output for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6ca0c531-4eee-4836-a916-e2f15f7d92a2_isrForCrosstalkSources_855447_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6ca0c531-4eee-4836-a916-e2f15f7d92a2_isrForCrosstalkSources_855447_53.stderr parsl.dataflow.dflow DEBUG: Task 3339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3339 has memoization hash b154154ca28cedfac2ac6ec811989865 parsl.dataflow.memoization INFO: Task 3339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3339 try 0 launched on executor multi with executor id 3340 parsl.dataflow.dflow INFO: Standard output for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/67435a15-511c-4ff0-b467-3dfea52e54e6_isrForCrosstalkSources_855440_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/67435a15-511c-4ff0-b467-3dfea52e54e6_isrForCrosstalkSources_855440_38.stderr parsl.dataflow.dflow DEBUG: Task 3340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3340 has memoization hash df4a1d4184b20f6b95993b219640abe9 parsl.dataflow.memoization INFO: Task 3340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a605c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3340 try 0 launched on executor multi with executor id 3341 parsl.dataflow.dflow INFO: Standard output for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d3c25951-4fa8-47f6-a0db-fb27a811949d_isrForCrosstalkSources_855439_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d3c25951-4fa8-47f6-a0db-fb27a811949d_isrForCrosstalkSources_855439_22.stderr parsl.dataflow.dflow DEBUG: Task 3341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3341 has memoization hash 3647c15a5afd51cec07d9131a9bfeb6f parsl.dataflow.memoization INFO: Task 3341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3341 try 0 launched on executor multi with executor id 3342 parsl.dataflow.dflow INFO: Standard output for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/5700b2d9-e122-428c-98f5-558ba04ec00d_isrForCrosstalkSources_855423_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/5700b2d9-e122-428c-98f5-558ba04ec00d_isrForCrosstalkSources_855423_48.stderr parsl.dataflow.dflow DEBUG: Task 3342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3342 has memoization hash d1a02f71c5322e229fcfaf29553f55ed parsl.dataflow.memoization INFO: Task 3342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3342 try 0 launched on executor multi with executor id 3343 parsl.dataflow.dflow INFO: Standard output for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0fa7e19a-79a3-4b3a-83c2-0687b45a7abf_isrForCrosstalkSources_855457_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0fa7e19a-79a3-4b3a-83c2-0687b45a7abf_isrForCrosstalkSources_855457_14.stderr parsl.dataflow.dflow DEBUG: Task 3343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3343 has memoization hash fc7e19ebdccff9cabf61200a69f70387 parsl.dataflow.memoization INFO: Task 3343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3343 try 0 launched on executor multi with executor id 3344 parsl.dataflow.dflow INFO: Standard output for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a54b702d-c78a-4474-a73b-197dce6bb0d4_isrForCrosstalkSources_855448_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a54b702d-c78a-4474-a73b-197dce6bb0d4_isrForCrosstalkSources_855448_57.stderr parsl.dataflow.dflow DEBUG: Task 3344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3344 has memoization hash 85607e86f48d4d6d46dc54b0d5a6af7a parsl.dataflow.memoization INFO: Task 3344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3344 try 0 launched on executor multi with executor id 3345 parsl.dataflow.dflow INFO: Standard output for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/bfdaf816-be8f-4f54-830b-fac0e1c859e2_isrForCrosstalkSources_855445_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/bfdaf816-be8f-4f54-830b-fac0e1c859e2_isrForCrosstalkSources_855445_58.stderr parsl.dataflow.dflow DEBUG: Task 3345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3345 has memoization hash abac1aa8cc88776d8cb9dd74762b5455 parsl.dataflow.memoization INFO: Task 3345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3345 try 0 launched on executor multi with executor id 3346 parsl.dataflow.dflow INFO: Standard output for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d3e643c1-4532-48c7-a9c7-137ad03b8e45_isrForCrosstalkSources_855397_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d3e643c1-4532-48c7-a9c7-137ad03b8e45_isrForCrosstalkSources_855397_24.stderr parsl.dataflow.dflow DEBUG: Task 3346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3346 has memoization hash 5fc73e438e59f2fe80f0fd9072636e46 parsl.dataflow.memoization INFO: Task 3346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3346 try 0 launched on executor multi with executor id 3347 parsl.dataflow.dflow INFO: Standard output for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b5378692-ec56-47de-936f-4721b1f11595_isrForCrosstalkSources_855458_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b5378692-ec56-47de-936f-4721b1f11595_isrForCrosstalkSources_855458_49.stderr parsl.dataflow.dflow DEBUG: Task 3347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3347 has memoization hash 936d7af57c2e567ad8b8cdff270721fb parsl.dataflow.memoization INFO: Task 3347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3347 try 0 launched on executor multi with executor id 3348 parsl.dataflow.dflow INFO: Standard output for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/badd45d1-4e6a-412f-bd81-ce4838bdcb0e_isrForCrosstalkSources_855453_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/badd45d1-4e6a-412f-bd81-ce4838bdcb0e_isrForCrosstalkSources_855453_47.stderr parsl.dataflow.dflow DEBUG: Task 3348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3348 has memoization hash 556207e9396dd4cf605092b6e27aff1b parsl.dataflow.memoization INFO: Task 3348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3348 try 0 launched on executor multi with executor id 3349 parsl.dataflow.dflow INFO: Standard output for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/181468a7-9bb0-49ef-a35f-8319c917053a_isrForCrosstalkSources_855431_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/181468a7-9bb0-49ef-a35f-8319c917053a_isrForCrosstalkSources_855431_22.stderr parsl.dataflow.dflow DEBUG: Task 3349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3349 has memoization hash 055b0070861bf2b3ae08e94b9629c5e7 parsl.dataflow.memoization INFO: Task 3349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3349 try 0 launched on executor multi with executor id 3350 parsl.dataflow.dflow INFO: Standard output for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/20814ec4-7b25-40ac-8912-bf446f434f55_isrForCrosstalkSources_855387_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/20814ec4-7b25-40ac-8912-bf446f434f55_isrForCrosstalkSources_855387_28.stderr parsl.dataflow.dflow DEBUG: Task 3350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3350 has memoization hash 115275db087fdfc37b09ccea55586339 parsl.dataflow.memoization INFO: Task 3350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3350 try 0 launched on executor multi with executor id 3351 parsl.dataflow.dflow INFO: Standard output for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/19a61281-f927-4d1c-a52b-c48fd421ebe2_isrForCrosstalkSources_855454_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/19a61281-f927-4d1c-a52b-c48fd421ebe2_isrForCrosstalkSources_855454_23.stderr parsl.dataflow.dflow DEBUG: Task 3351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3351 has memoization hash 8df01bbd456ce86ecbee0ea8cf0e6db3 parsl.dataflow.memoization INFO: Task 3351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3351 try 0 launched on executor multi with executor id 3352 parsl.dataflow.dflow INFO: Standard output for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e3fbfeea-08a1-4366-9d3c-88e819d73968_isrForCrosstalkSources_855391_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e3fbfeea-08a1-4366-9d3c-88e819d73968_isrForCrosstalkSources_855391_40.stderr parsl.dataflow.dflow DEBUG: Task 3352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3352 has memoization hash 330dcb591e6ad756bbfaa4237c32c353 parsl.dataflow.memoization INFO: Task 3352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3352 try 0 launched on executor multi with executor id 3353 parsl.dataflow.dflow INFO: Standard output for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/70094a6c-074f-404d-afd8-be75026cd251_isrForCrosstalkSources_855453_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/70094a6c-074f-404d-afd8-be75026cd251_isrForCrosstalkSources_855453_17.stderr parsl.dataflow.dflow DEBUG: Task 3353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3353 has memoization hash 4d1a47b541174bd77429f573fc8dfdca parsl.dataflow.memoization INFO: Task 3353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3353 try 0 launched on executor multi with executor id 3354 parsl.dataflow.dflow INFO: Standard output for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/049984de-e4be-437c-b9ac-4efe10e0b8f7_isrForCrosstalkSources_855421_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/049984de-e4be-437c-b9ac-4efe10e0b8f7_isrForCrosstalkSources_855421_5.stderr parsl.dataflow.dflow DEBUG: Task 3354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3354 has memoization hash 631008413c118d7321b2aa37165c4d44 parsl.dataflow.memoization INFO: Task 3354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3354 try 0 launched on executor multi with executor id 3355 parsl.dataflow.dflow INFO: Standard output for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/f26ef99f-5411-4801-9458-7392eaa1149e_isrForCrosstalkSources_855415_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/f26ef99f-5411-4801-9458-7392eaa1149e_isrForCrosstalkSources_855415_56.stderr parsl.dataflow.dflow DEBUG: Task 3355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3355 has memoization hash 11535d976e9cb8f6a04de4a57079cfe6 parsl.dataflow.memoization INFO: Task 3355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3355 try 0 launched on executor multi with executor id 3356 parsl.dataflow.dflow INFO: Standard output for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/15c6b4f1-956b-4787-b031-26d7161018a9_isrForCrosstalkSources_855391_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/15c6b4f1-956b-4787-b031-26d7161018a9_isrForCrosstalkSources_855391_37.stderr parsl.dataflow.dflow DEBUG: Task 3356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3356 has memoization hash aa9eb54f79ee5e90b956fa953d5991f9 parsl.dataflow.memoization INFO: Task 3356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3356 try 0 launched on executor multi with executor id 3357 parsl.dataflow.dflow INFO: Standard output for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/87cbe565-d88e-4664-a78a-54dbca48a1b1_isrForCrosstalkSources_855388_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/87cbe565-d88e-4664-a78a-54dbca48a1b1_isrForCrosstalkSources_855388_38.stderr parsl.dataflow.dflow DEBUG: Task 3357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3357 has memoization hash c1aed864508a9b3d8d0a9997fca68b2b parsl.dataflow.memoization INFO: Task 3357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3357 try 0 launched on executor multi with executor id 3358 parsl.dataflow.dflow INFO: Standard output for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/24093dfa-1f4d-477d-a510-7ea42bb9190d_isrForCrosstalkSources_855455_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/24093dfa-1f4d-477d-a510-7ea42bb9190d_isrForCrosstalkSources_855455_34.stderr parsl.dataflow.dflow DEBUG: Task 3358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3358 has memoization hash a545bb4a65f3bd5546237f364804673e parsl.dataflow.memoization INFO: Task 3358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3358 try 0 launched on executor multi with executor id 3359 parsl.dataflow.dflow INFO: Standard output for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1702a573-18f8-4c82-b87d-6134152070ee_isrForCrosstalkSources_855454_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1702a573-18f8-4c82-b87d-6134152070ee_isrForCrosstalkSources_855454_15.stderr parsl.dataflow.dflow DEBUG: Task 3359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3359 has memoization hash e50a0f8a243f648381b333a0dcf5026f parsl.dataflow.memoization INFO: Task 3359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3359 try 0 launched on executor multi with executor id 3360 parsl.dataflow.dflow INFO: Standard output for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a4d4bc13-1207-4fbb-898f-2718e2e1d708_isrForCrosstalkSources_855456_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a4d4bc13-1207-4fbb-898f-2718e2e1d708_isrForCrosstalkSources_855456_50.stderr parsl.dataflow.dflow DEBUG: Task 3360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3360 has memoization hash 0377e7e1ee8245b2548ff3c70aee926b parsl.dataflow.memoization INFO: Task 3360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3360 try 0 launched on executor multi with executor id 3361 parsl.dataflow.dflow INFO: Standard output for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ac10cf85-57e9-4f91-ba40-3f0527b45860_isrForCrosstalkSources_855404_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ac10cf85-57e9-4f91-ba40-3f0527b45860_isrForCrosstalkSources_855404_6.stderr parsl.dataflow.dflow DEBUG: Task 3361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3361 has memoization hash 6a0e845a8908d08c29b65769a12fbd0b parsl.dataflow.memoization INFO: Task 3361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3361 try 0 launched on executor multi with executor id 3362 parsl.dataflow.dflow INFO: Standard output for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4070d7eb-1362-4cf7-a9c9-d2185a396d53_isrForCrosstalkSources_855392_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4070d7eb-1362-4cf7-a9c9-d2185a396d53_isrForCrosstalkSources_855392_52.stderr parsl.dataflow.dflow DEBUG: Task 3362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3362 has memoization hash e7c909cabe0faecc482469c32f3aa7c3 parsl.dataflow.memoization INFO: Task 3362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3362 try 0 launched on executor multi with executor id 3363 parsl.dataflow.dflow INFO: Standard output for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7ac05089-1c75-4531-8ba5-7af6a3212b8b_isrForCrosstalkSources_855391_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7ac05089-1c75-4531-8ba5-7af6a3212b8b_isrForCrosstalkSources_855391_8.stderr parsl.dataflow.dflow DEBUG: Task 3363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3363 has memoization hash 13dbfd5a7c296e37ef1124acd7d9e45b parsl.dataflow.memoization INFO: Task 3363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3363 try 0 launched on executor multi with executor id 3364 parsl.dataflow.dflow INFO: Standard output for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80e88815-87b0-4a0a-a9ec-9406785584da_isrForCrosstalkSources_855426_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80e88815-87b0-4a0a-a9ec-9406785584da_isrForCrosstalkSources_855426_40.stderr parsl.dataflow.dflow DEBUG: Task 3364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3364 has memoization hash 1145aa37034b6d392f7fe8bdd53bd190 parsl.dataflow.memoization INFO: Task 3364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3364 try 0 launched on executor multi with executor id 3365 parsl.dataflow.dflow INFO: Standard output for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/61f50193-ee37-4641-80c4-3d732595a32f_isrForCrosstalkSources_855419_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/61f50193-ee37-4641-80c4-3d732595a32f_isrForCrosstalkSources_855419_61.stderr parsl.dataflow.dflow DEBUG: Task 3365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3365 has memoization hash 35f952c4a7d03480e1f28b56cd9c3eb4 parsl.dataflow.memoization INFO: Task 3365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3365 try 0 launched on executor multi with executor id 3366 parsl.dataflow.dflow INFO: Standard output for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/c6d741d9-414b-4f5c-b33b-0d7aace65a3e_isrForCrosstalkSources_855396_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/c6d741d9-414b-4f5c-b33b-0d7aace65a3e_isrForCrosstalkSources_855396_32.stderr parsl.dataflow.dflow DEBUG: Task 3366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3366 has memoization hash 55f7e2de965ba70ad7ea4e8ded84184b parsl.dataflow.memoization INFO: Task 3366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dfded93159157b9225aca2781b19031 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/785e1362-3196-4404-b220-da5cd889643b_isrForCrosstalkSources_855454_22.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/785e1362-3196-4404-b220-da5cd889643b_isrForCrosstalkSources_855454_22.stderr parsl.dataflow.dflow INFO: Parsl task 3366 try 0 launched on executor multi with executor id 3367 parsl.dataflow.dflow INFO: Standard output for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/28b08aae-ed3c-4b0e-b078-44b872797cde_isrForCrosstalkSources_855418_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/28b08aae-ed3c-4b0e-b078-44b872797cde_isrForCrosstalkSources_855418_49.stderr parsl.dataflow.dflow DEBUG: Task 3367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3367 has memoization hash 56aa915d2bd60ccf0867b0624604c7ee parsl.dataflow.memoization INFO: Task 3367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3367 try 0 launched on executor multi with executor id 3368 parsl.dataflow.dflow INFO: Standard output for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/be2beb1e-96b1-4d84-bacc-a3903841e90e_isrForCrosstalkSources_855385_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/be2beb1e-96b1-4d84-bacc-a3903841e90e_isrForCrosstalkSources_855385_27.stderr parsl.dataflow.dflow DEBUG: Task 3368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3368 has memoization hash ad488ca53868a708ddab0b135baa5278 parsl.dataflow.memoization INFO: Task 3368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3368 try 0 launched on executor multi with executor id 3369 parsl.dataflow.dflow INFO: Standard output for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/27f95e20-8453-4972-847f-0837d2bd09d1_isrForCrosstalkSources_855439_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/27f95e20-8453-4972-847f-0837d2bd09d1_isrForCrosstalkSources_855439_5.stderr parsl.dataflow.dflow DEBUG: Task 3369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3369 has memoization hash 6b5a15abb7569e0512ba05ad034b4964 parsl.dataflow.memoization INFO: Task 3369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3369 try 0 launched on executor multi with executor id 3370 parsl.dataflow.dflow INFO: Standard output for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aa3af8ba-2f35-43e4-b933-aa585a4c3621_isrForCrosstalkSources_855449_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aa3af8ba-2f35-43e4-b933-aa585a4c3621_isrForCrosstalkSources_855449_59.stderr parsl.dataflow.dflow DEBUG: Task 3370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3370 has memoization hash 161ed729d2aebd220ca0bda435f0439d parsl.dataflow.memoization INFO: Task 3370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3370 try 0 launched on executor multi with executor id 3371 parsl.dataflow.dflow INFO: Standard output for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9277b11c-f790-456b-8478-c4bc9a5729f3_isrForCrosstalkSources_855396_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9277b11c-f790-456b-8478-c4bc9a5729f3_isrForCrosstalkSources_855396_14.stderr parsl.dataflow.dflow DEBUG: Task 3371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3371 has memoization hash c810dc501424d27e40b2af65e9d74333 parsl.dataflow.memoization INFO: Task 3371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3371 try 0 launched on executor multi with executor id 3372 parsl.dataflow.dflow INFO: Standard output for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/31dace39-4684-443c-99c5-5b869aa5b392_isrForCrosstalkSources_855391_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/31dace39-4684-443c-99c5-5b869aa5b392_isrForCrosstalkSources_855391_11.stderr parsl.dataflow.dflow DEBUG: Task 3372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3372 has memoization hash dfc7e77799827a12b903d3c7a8fe56c6 parsl.dataflow.memoization INFO: Task 3372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3372 try 0 launched on executor multi with executor id 3373 parsl.dataflow.dflow INFO: Standard output for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f5541ea-5ede-4116-af1d-9c330e20ccc7_isrForCrosstalkSources_855387_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f5541ea-5ede-4116-af1d-9c330e20ccc7_isrForCrosstalkSources_855387_16.stderr parsl.dataflow.dflow DEBUG: Task 3373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3373 has memoization hash d950695bbcb8def6a36f769882fbd848 parsl.dataflow.memoization INFO: Task 3373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3373 try 0 launched on executor multi with executor id 3374 parsl.dataflow.dflow INFO: Standard output for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/72d91b75-1529-4760-98de-116815f11a66_isrForCrosstalkSources_855408_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/72d91b75-1529-4760-98de-116815f11a66_isrForCrosstalkSources_855408_47.stderr parsl.dataflow.dflow DEBUG: Task 3374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3374 has memoization hash 0541556ecf661ae5a6104a062153726a parsl.dataflow.memoization INFO: Task 3374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3374 try 0 launched on executor multi with executor id 3375 parsl.dataflow.dflow INFO: Standard output for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2c132c63-330e-4301-afed-6d3ef4653d78_isrForCrosstalkSources_855382_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2c132c63-330e-4301-afed-6d3ef4653d78_isrForCrosstalkSources_855382_30.stderr parsl.dataflow.dflow DEBUG: Task 3375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3375 has memoization hash dad7c228c244987804d31aa717ebabe3 parsl.dataflow.memoization INFO: Task 3375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3375 try 0 launched on executor multi with executor id 3376 parsl.dataflow.dflow INFO: Standard output for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/29698ad7-a62e-47d0-abe5-7ae6c92d1708_isrForCrosstalkSources_855456_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/29698ad7-a62e-47d0-abe5-7ae6c92d1708_isrForCrosstalkSources_855456_9.stderr parsl.dataflow.dflow DEBUG: Task 3376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3376 has memoization hash bf3b5abb528b22b2d829ff413971acea parsl.dataflow.memoization INFO: Task 3376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3376 try 0 launched on executor multi with executor id 3377 parsl.dataflow.dflow INFO: Standard output for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9062b442-854e-4d95-b469-19b7ca1468d9_isrForCrosstalkSources_855418_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9062b442-854e-4d95-b469-19b7ca1468d9_isrForCrosstalkSources_855418_15.stderr parsl.dataflow.dflow DEBUG: Task 3377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3377 has memoization hash 6f47293dce13a8733d2ee2f42c6a4cd7 parsl.dataflow.memoization INFO: Task 3377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3377 try 0 launched on executor multi with executor id 3378 parsl.dataflow.dflow INFO: Standard output for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4dbddeb2-d65b-4527-98e6-d6228f90ac64_isrForCrosstalkSources_855389_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4dbddeb2-d65b-4527-98e6-d6228f90ac64_isrForCrosstalkSources_855389_5.stderr parsl.dataflow.dflow DEBUG: Task 3378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3378 has memoization hash 4c9811655269a9fa9fd1b4f7fc92e69c parsl.dataflow.memoization INFO: Task 3378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3378 try 0 launched on executor multi with executor id 3379 parsl.dataflow.dflow INFO: Standard output for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/4f69030a-4e82-4cf2-b87a-2e1e01ba6e0c_isrForCrosstalkSources_855441_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/4f69030a-4e82-4cf2-b87a-2e1e01ba6e0c_isrForCrosstalkSources_855441_20.stderr parsl.dataflow.dflow DEBUG: Task 3379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3379 has memoization hash b0e3b3af61e5e28bab3d7c22556017a0 parsl.dataflow.memoization INFO: Task 3379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3379 try 0 launched on executor multi with executor id 3380 parsl.dataflow.dflow INFO: Standard output for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4ebd1c0d-f607-4879-898d-4aea448111e0_isrForCrosstalkSources_855435_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4ebd1c0d-f607-4879-898d-4aea448111e0_isrForCrosstalkSources_855435_55.stderr parsl.dataflow.dflow DEBUG: Task 3380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3380 has memoization hash d51011386f3f17c73742f971d4001010 parsl.dataflow.memoization INFO: Task 3380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3380 try 0 launched on executor multi with executor id 3381 parsl.dataflow.dflow INFO: Standard output for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/8b1647d2-9a27-41d5-b3af-b27dd8e946f1_isrForCrosstalkSources_855415_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/8b1647d2-9a27-41d5-b3af-b27dd8e946f1_isrForCrosstalkSources_855415_48.stderr parsl.dataflow.dflow DEBUG: Task 3381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3381 has memoization hash 19862d6e8e55d6d1536f976ef0c1a812 parsl.dataflow.memoization INFO: Task 3381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3381 try 0 launched on executor multi with executor id 3382 parsl.dataflow.dflow INFO: Standard output for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/617889d0-910d-4899-a81e-0e8e5f2310de_isrForCrosstalkSources_855442_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/617889d0-910d-4899-a81e-0e8e5f2310de_isrForCrosstalkSources_855442_6.stderr parsl.dataflow.dflow DEBUG: Task 3382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3382 has memoization hash 24457b5358b43ca3a39e060f9433e80f parsl.dataflow.memoization INFO: Task 3382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3382 try 0 launched on executor multi with executor id 3383 parsl.dataflow.dflow INFO: Standard output for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3e1d5f5-cb20-4b40-980e-c61a56917b8e_isrForCrosstalkSources_855414_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3e1d5f5-cb20-4b40-980e-c61a56917b8e_isrForCrosstalkSources_855414_2.stderr parsl.dataflow.dflow DEBUG: Task 3383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3383 has memoization hash 37307451f58886031901d55e3f8aaacc parsl.dataflow.memoization INFO: Task 3383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3383 try 0 launched on executor multi with executor id 3384 parsl.dataflow.dflow INFO: Standard output for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/4b3eda03-6f67-4424-a91d-3255c78c3494_isrForCrosstalkSources_855427_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/4b3eda03-6f67-4424-a91d-3255c78c3494_isrForCrosstalkSources_855427_13.stderr parsl.dataflow.dflow DEBUG: Task 3384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3384 has memoization hash 506c7cf1498a76c1bfd6dd298d826f39 parsl.dataflow.memoization INFO: Task 3384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3384 try 0 launched on executor multi with executor id 3385 parsl.dataflow.dflow INFO: Standard output for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/54c0b434-5401-49e8-8945-216d5043067c_isrForCrosstalkSources_855427_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/54c0b434-5401-49e8-8945-216d5043067c_isrForCrosstalkSources_855427_9.stderr parsl.dataflow.dflow DEBUG: Task 3385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3385 has memoization hash 5ce5193c4126cf218a8783f601090c3d parsl.dataflow.memoization INFO: Task 3385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3385 try 0 launched on executor multi with executor id 3386 parsl.dataflow.dflow INFO: Standard output for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23bd345c-6299-4cea-998a-f3726dd9ee87_isrForCrosstalkSources_855396_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23bd345c-6299-4cea-998a-f3726dd9ee87_isrForCrosstalkSources_855396_35.stderr parsl.dataflow.dflow DEBUG: Task 3386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3386 has memoization hash 4ad019fa327205bf31bed7350626bfd5 parsl.dataflow.memoization INFO: Task 3386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3386 try 0 launched on executor multi with executor id 3387 parsl.dataflow.dflow INFO: Standard output for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/789a794b-d699-4e6b-b710-a2954a6ee26e_isrForCrosstalkSources_855399_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/789a794b-d699-4e6b-b710-a2954a6ee26e_isrForCrosstalkSources_855399_38.stderr parsl.dataflow.dflow DEBUG: Task 3387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3387 has memoization hash 2b3764023a10efc609b6a382e2a96869 parsl.dataflow.memoization INFO: Task 3387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3387 try 0 launched on executor multi with executor id 3388 parsl.dataflow.dflow INFO: Standard output for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/7c123490-d2b9-4c0b-b5f6-e3699becf56e_isrForCrosstalkSources_855407_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/7c123490-d2b9-4c0b-b5f6-e3699becf56e_isrForCrosstalkSources_855407_29.stderr parsl.dataflow.dflow DEBUG: Task 3388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3388 has memoization hash 24ca263493ba6c9ec3bb4a6eea01dd5e parsl.dataflow.memoization INFO: Task 3388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3388 try 0 launched on executor multi with executor id 3389 parsl.dataflow.dflow INFO: Standard output for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/8fd63271-097e-40b9-8ec3-bac9321bd005_isrForCrosstalkSources_855383_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/8fd63271-097e-40b9-8ec3-bac9321bd005_isrForCrosstalkSources_855383_60.stderr parsl.dataflow.dflow DEBUG: Task 3389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3389 has memoization hash 6069938b714966539b65884ed69712fd parsl.dataflow.memoization INFO: Task 3389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3389 try 0 launched on executor multi with executor id 3390 parsl.dataflow.dflow INFO: Standard output for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/168baace-19a4-4344-b925-af9ee984bbd8_isrForCrosstalkSources_855407_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/168baace-19a4-4344-b925-af9ee984bbd8_isrForCrosstalkSources_855407_3.stderr parsl.dataflow.dflow DEBUG: Task 3390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3390 has memoization hash 066afb20fe0bfe41e2ac78dcd7510413 parsl.dataflow.memoization INFO: Task 3390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3390 try 0 launched on executor multi with executor id 3391 parsl.dataflow.dflow INFO: Standard output for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b9e8ff41-4703-4740-8356-2a69b46b1bd5_isrForCrosstalkSources_855402_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b9e8ff41-4703-4740-8356-2a69b46b1bd5_isrForCrosstalkSources_855402_40.stderr parsl.dataflow.dflow DEBUG: Task 3391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3391 has memoization hash c612265e0e8c2d1abe2bde773cc28f5f parsl.dataflow.memoization INFO: Task 3391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3391 try 0 launched on executor multi with executor id 3392 parsl.dataflow.dflow INFO: Standard output for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7eb0d558-fc64-4987-ba09-f731e13f89b0_isrForCrosstalkSources_855386_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7eb0d558-fc64-4987-ba09-f731e13f89b0_isrForCrosstalkSources_855386_44.stderr parsl.dataflow.dflow DEBUG: Task 3392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3392 has memoization hash 9165593dca94f7b2cc1ee6500f506197 parsl.dataflow.memoization INFO: Task 3392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3392 try 0 launched on executor multi with executor id 3393 parsl.dataflow.dflow INFO: Standard output for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a0dc2f0e-24ec-4990-8a3a-b88320f20964_isrForCrosstalkSources_855385_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a0dc2f0e-24ec-4990-8a3a-b88320f20964_isrForCrosstalkSources_855385_30.stderr parsl.dataflow.dflow DEBUG: Task 3393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3393 has memoization hash ca2b4824b3887da67e5e2a1275227371 parsl.dataflow.memoization INFO: Task 3393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3393 try 0 launched on executor multi with executor id 3394 parsl.dataflow.dflow INFO: Standard output for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bef7bc5e-93d3-4d33-9776-643a60d86c46_isrForCrosstalkSources_855381_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bef7bc5e-93d3-4d33-9776-643a60d86c46_isrForCrosstalkSources_855381_31.stderr parsl.dataflow.dflow DEBUG: Task 3394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3394 has memoization hash 3af2442d4d469bc06db6873c6408e5d8 parsl.dataflow.memoization INFO: Task 3394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3394 try 0 launched on executor multi with executor id 3395 parsl.dataflow.dflow INFO: Standard output for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fe0c2afe-735c-4468-8cb8-fb896061b297_isrForCrosstalkSources_855430_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fe0c2afe-735c-4468-8cb8-fb896061b297_isrForCrosstalkSources_855430_46.stderr parsl.dataflow.dflow DEBUG: Task 3395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3395 has memoization hash 927f167eeebc1632a0a734365237600e parsl.dataflow.memoization INFO: Task 3395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3395 try 0 launched on executor multi with executor id 3396 parsl.dataflow.dflow INFO: Standard output for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/562f57e2-5766-4f4a-9e41-f5e012465bb1_isrForCrosstalkSources_855401_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/562f57e2-5766-4f4a-9e41-f5e012465bb1_isrForCrosstalkSources_855401_6.stderr parsl.dataflow.dflow DEBUG: Task 3396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3396 has memoization hash 2b4df547fc47e0c4750dfafe52c62687 parsl.dataflow.memoization INFO: Task 3396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3396 try 0 launched on executor multi with executor id 3397 parsl.dataflow.dflow INFO: Standard output for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/523ef4a9-7b9c-42e6-af9b-8aeee1f4bed0_isrForCrosstalkSources_855450_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/523ef4a9-7b9c-42e6-af9b-8aeee1f4bed0_isrForCrosstalkSources_855450_53.stderr parsl.dataflow.dflow DEBUG: Task 3397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3397 has memoization hash 9dc812d81f10490f8e17cf899b31c12a parsl.dataflow.memoization INFO: Task 3397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3397 try 0 launched on executor multi with executor id 3398 parsl.dataflow.dflow INFO: Standard output for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5b77f20-46e3-4d6d-a39b-640c70576a84_isrForCrosstalkSources_855456_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5b77f20-46e3-4d6d-a39b-640c70576a84_isrForCrosstalkSources_855456_5.stderr parsl.dataflow.dflow DEBUG: Task 3398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3398 has memoization hash 2bda2473b83656e7a22d5aecf4739ba4 parsl.dataflow.memoization INFO: Task 3398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3398 try 0 launched on executor multi with executor id 3399 parsl.dataflow.dflow INFO: Standard output for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6d12d94-8fd4-444e-be24-fc715c82337f_isrForCrosstalkSources_855412_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6d12d94-8fd4-444e-be24-fc715c82337f_isrForCrosstalkSources_855412_17.stderr parsl.dataflow.dflow DEBUG: Task 3399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3399 has memoization hash f943a13467e37a9fe44f470f9867b6f2 parsl.dataflow.memoization INFO: Task 3399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3399 try 0 launched on executor multi with executor id 3400 parsl.dataflow.dflow INFO: Standard output for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0dbcd168-361b-4861-8ec8-00e8e3319314_isrForCrosstalkSources_855445_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0dbcd168-361b-4861-8ec8-00e8e3319314_isrForCrosstalkSources_855445_25.stderr parsl.dataflow.dflow DEBUG: Task 3400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3400 has memoization hash d7e3b992b7b16fae61dfc7829ad11ee9 parsl.dataflow.memoization INFO: Task 3400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 561d3266cfac23c38a5a68e5b746db64 with result from task 174 parsl.dataflow.dflow INFO: Parsl task 3400 try 0 launched on executor multi with executor id 3401 parsl.dataflow.dflow INFO: Standard output for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/aa55b2aa-806b-431c-94f7-cb7b37273d1a_isrForCrosstalkSources_855390_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/aa55b2aa-806b-431c-94f7-cb7b37273d1a_isrForCrosstalkSources_855390_22.stderr parsl.dataflow.dflow DEBUG: Task 3401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3401 has memoization hash a6cf237806ce6f3e16e74477b9d66197 parsl.dataflow.memoization INFO: Task 3401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5d035bec-230b-4312-9e98-7081ef6b6bc3_isrForCrosstalkSources_855441_23.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5d035bec-230b-4312-9e98-7081ef6b6bc3_isrForCrosstalkSources_855441_23.stderr parsl.dataflow.dflow INFO: Parsl task 3401 try 0 launched on executor multi with executor id 3402 parsl.dataflow.dflow INFO: Standard output for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f4386468-35f5-4f59-99e4-f4249136db28_isrForCrosstalkSources_855393_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f4386468-35f5-4f59-99e4-f4249136db28_isrForCrosstalkSources_855393_46.stderr parsl.dataflow.dflow DEBUG: Task 3402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3402 has memoization hash 6c2b846eaa10161c55f3e0bfb88343c6 parsl.dataflow.memoization INFO: Task 3402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3402 try 0 launched on executor multi with executor id 3403 parsl.dataflow.dflow INFO: Standard output for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3c737218-d2b3-468a-b682-950d691b21d4_isrForCrosstalkSources_855404_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3c737218-d2b3-468a-b682-950d691b21d4_isrForCrosstalkSources_855404_39.stderr parsl.dataflow.dflow DEBUG: Task 3403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3403 has memoization hash c53d90ec6b7f50c4940e0a1d6ab405fd parsl.dataflow.memoization INFO: Task 3403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3403 try 0 launched on executor multi with executor id 3404 parsl.dataflow.dflow INFO: Standard output for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/10dac8f3-51f7-4d55-874a-abeaefa8eba8_isrForCrosstalkSources_855391_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/10dac8f3-51f7-4d55-874a-abeaefa8eba8_isrForCrosstalkSources_855391_5.stderr parsl.dataflow.dflow DEBUG: Task 3404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3404 has memoization hash 248644aaf35cd09b32a1d2b08af58a09 parsl.dataflow.memoization INFO: Task 3404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3404 try 0 launched on executor multi with executor id 3405 parsl.dataflow.dflow INFO: Standard output for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c67453f4-5275-443b-9e5a-b67b6094b44b_isrForCrosstalkSources_855389_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c67453f4-5275-443b-9e5a-b67b6094b44b_isrForCrosstalkSources_855389_46.stderr parsl.dataflow.dflow DEBUG: Task 3405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3405 has memoization hash 03b4bca069ceb22d643d0f2111d64c91 parsl.dataflow.memoization INFO: Task 3405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3405 try 0 launched on executor multi with executor id 3406 parsl.dataflow.dflow INFO: Standard output for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/39688a18-723d-4234-a925-9552850744ab_isrForCrosstalkSources_855436_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/39688a18-723d-4234-a925-9552850744ab_isrForCrosstalkSources_855436_52.stderr parsl.dataflow.dflow DEBUG: Task 3406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3406 has memoization hash c9fb32357d3f69f6d75298f67d81a4c1 parsl.dataflow.memoization INFO: Task 3406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3406 try 0 launched on executor multi with executor id 3407 parsl.dataflow.dflow INFO: Standard output for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/abe884fc-730f-4efe-b1c7-fc4e383f5e43_isrForCrosstalkSources_855407_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/abe884fc-730f-4efe-b1c7-fc4e383f5e43_isrForCrosstalkSources_855407_35.stderr parsl.dataflow.dflow DEBUG: Task 3407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3407 has memoization hash 4e7ef7f4ee12cf83dc22ec49b55288d4 parsl.dataflow.memoization INFO: Task 3407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3407 try 0 launched on executor multi with executor id 3408 parsl.dataflow.dflow INFO: Standard output for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/69966d3a-1bb9-4fb8-b22d-d0c73e63d892_isrForCrosstalkSources_855410_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/69966d3a-1bb9-4fb8-b22d-d0c73e63d892_isrForCrosstalkSources_855410_26.stderr parsl.dataflow.dflow DEBUG: Task 3408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3408 has memoization hash 974ce5981df1a63d6832e1ae9cfeee60 parsl.dataflow.memoization INFO: Task 3408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3408 try 0 launched on executor multi with executor id 3409 parsl.dataflow.dflow INFO: Standard output for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/367da439-8cc9-46eb-9289-7a8e47221f11_isrForCrosstalkSources_855423_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/367da439-8cc9-46eb-9289-7a8e47221f11_isrForCrosstalkSources_855423_59.stderr parsl.dataflow.dflow DEBUG: Task 3409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3409 has memoization hash 367628b9d71f343d58c7938a1b7a9f6c parsl.dataflow.memoization INFO: Task 3409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3409 try 0 launched on executor multi with executor id 3410 parsl.dataflow.dflow INFO: Standard output for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ae3a5866-67e9-46c1-bc7f-620176f12045_isrForCrosstalkSources_855405_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ae3a5866-67e9-46c1-bc7f-620176f12045_isrForCrosstalkSources_855405_12.stderr parsl.dataflow.dflow DEBUG: Task 3410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3410 has memoization hash f222fcdac88959fb44022c2969c56b57 parsl.dataflow.memoization INFO: Task 3410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3410 try 0 launched on executor multi with executor id 3411 parsl.dataflow.dflow INFO: Standard output for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/181886ad-03ce-4f54-9e5f-c7370d34dbd7_isrForCrosstalkSources_855405_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/181886ad-03ce-4f54-9e5f-c7370d34dbd7_isrForCrosstalkSources_855405_20.stderr parsl.dataflow.dflow DEBUG: Task 3411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3411 has memoization hash 55918bace708ec72d4db905a8493b45b parsl.dataflow.memoization INFO: Task 3411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3411 try 0 launched on executor multi with executor id 3412 parsl.dataflow.dflow INFO: Standard output for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac2b4776-0947-44f1-89e2-220f376b2eca_isrForCrosstalkSources_855394_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac2b4776-0947-44f1-89e2-220f376b2eca_isrForCrosstalkSources_855394_30.stderr parsl.dataflow.dflow DEBUG: Task 3412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3412 has memoization hash 314ce39031984b7e9165233999ac1e86 parsl.dataflow.memoization INFO: Task 3412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3412 try 0 launched on executor multi with executor id 3413 parsl.dataflow.dflow INFO: Standard output for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9cd63c91-ef38-4ced-97f7-c544c7b96ef3_isrForCrosstalkSources_855446_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9cd63c91-ef38-4ced-97f7-c544c7b96ef3_isrForCrosstalkSources_855446_49.stderr parsl.dataflow.dflow DEBUG: Task 3413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3413 has memoization hash 45c8a2cec3f6557941103180d76343b2 parsl.dataflow.memoization INFO: Task 3413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3413 try 0 launched on executor multi with executor id 3414 parsl.dataflow.dflow INFO: Standard output for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0e0a8bc0-dcc0-4a84-9cb1-33604e528266_isrForCrosstalkSources_855419_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0e0a8bc0-dcc0-4a84-9cb1-33604e528266_isrForCrosstalkSources_855419_58.stderr parsl.dataflow.dflow DEBUG: Task 3414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3414 has memoization hash 9430fd2c2448b8c337d5f547ccfe32c1 parsl.dataflow.memoization INFO: Task 3414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3414 try 0 launched on executor multi with executor id 3415 parsl.dataflow.dflow INFO: Standard output for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/b2e8baf0-5372-4e63-a0c4-e5dca8aee349_isrForCrosstalkSources_855401_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/b2e8baf0-5372-4e63-a0c4-e5dca8aee349_isrForCrosstalkSources_855401_48.stderr parsl.dataflow.dflow DEBUG: Task 3415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3415 has memoization hash 0446782af5a38da56615f7ccab8c809c parsl.dataflow.memoization INFO: Task 3415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3415 try 0 launched on executor multi with executor id 3416 parsl.dataflow.dflow INFO: Standard output for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e6e5bd5d-38c9-402b-89b4-605d8d08d500_isrForCrosstalkSources_855424_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e6e5bd5d-38c9-402b-89b4-605d8d08d500_isrForCrosstalkSources_855424_49.stderr parsl.dataflow.dflow DEBUG: Task 3416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3416 has memoization hash 7ea7dc7712ff82456d47bc357519f150 parsl.dataflow.memoization INFO: Task 3416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3416 try 0 launched on executor multi with executor id 3417 parsl.dataflow.dflow INFO: Standard output for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/dcf9f9ac-f74a-414b-aa25-3d56ff424c0e_isrForCrosstalkSources_855414_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/dcf9f9ac-f74a-414b-aa25-3d56ff424c0e_isrForCrosstalkSources_855414_41.stderr parsl.dataflow.dflow DEBUG: Task 3417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3417 has memoization hash 526537aee93bf17347a090db3e86a889 parsl.dataflow.memoization INFO: Task 3417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3417 try 0 launched on executor multi with executor id 3418 parsl.dataflow.dflow INFO: Standard output for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/87ac3516-649e-4402-921c-29cab0773676_isrForCrosstalkSources_855419_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/87ac3516-649e-4402-921c-29cab0773676_isrForCrosstalkSources_855419_7.stderr parsl.dataflow.dflow DEBUG: Task 3418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3418 has memoization hash bfb0563dca82bc450b96fd3a4da07ebf parsl.dataflow.memoization INFO: Task 3418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3418 try 0 launched on executor multi with executor id 3419 parsl.dataflow.dflow INFO: Standard output for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/52a2b556-151e-472f-9a8e-a7f6de621d20_isrForCrosstalkSources_855402_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/52a2b556-151e-472f-9a8e-a7f6de621d20_isrForCrosstalkSources_855402_28.stderr parsl.dataflow.dflow DEBUG: Task 3419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3419 has memoization hash 3e9e1b45ff75b7e07f0615a506f3976e parsl.dataflow.memoization INFO: Task 3419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3419 try 0 launched on executor multi with executor id 3420 parsl.dataflow.dflow INFO: Standard output for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2c246f36-383e-4555-949b-e3ce2c96abfc_isrForCrosstalkSources_855441_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2c246f36-383e-4555-949b-e3ce2c96abfc_isrForCrosstalkSources_855441_57.stderr parsl.dataflow.dflow DEBUG: Task 3420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3420 has memoization hash d01d7876c6c811df0fd287e89b0a4930 parsl.dataflow.memoization INFO: Task 3420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3420 try 0 launched on executor multi with executor id 3421 parsl.dataflow.dflow INFO: Standard output for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f18b569c-8ae7-4fa3-8f4e-1e7f6b455960_isrForCrosstalkSources_855451_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f18b569c-8ae7-4fa3-8f4e-1e7f6b455960_isrForCrosstalkSources_855451_52.stderr parsl.dataflow.dflow DEBUG: Task 3421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3421 has memoization hash dfa8b6c6a7d9ba8a971471e8b2bee495 parsl.dataflow.memoization INFO: Task 3421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3421 try 0 launched on executor multi with executor id 3422 parsl.dataflow.dflow INFO: Standard output for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/22420c90-cd35-4d88-b178-2d0d7fcc6a76_isrForCrosstalkSources_855391_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/22420c90-cd35-4d88-b178-2d0d7fcc6a76_isrForCrosstalkSources_855391_47.stderr parsl.dataflow.dflow DEBUG: Task 3422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3422 has memoization hash 9a6835eff538c5939fd207a85e56bc76 parsl.dataflow.memoization INFO: Task 3422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3422 try 0 launched on executor multi with executor id 3423 parsl.dataflow.dflow INFO: Standard output for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/64fa625a-9205-40fc-83d9-4f1c9068ced2_isrForCrosstalkSources_855409_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/64fa625a-9205-40fc-83d9-4f1c9068ced2_isrForCrosstalkSources_855409_58.stderr parsl.dataflow.dflow DEBUG: Task 3423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3423 has memoization hash 06abe4f24497e13954dbfc045f16ced5 parsl.dataflow.memoization INFO: Task 3423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3423 try 0 launched on executor multi with executor id 3424 parsl.dataflow.dflow INFO: Standard output for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/71f23d17-183b-4a79-add2-3eee295e02f8_isrForCrosstalkSources_855455_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/71f23d17-183b-4a79-add2-3eee295e02f8_isrForCrosstalkSources_855455_23.stderr parsl.dataflow.dflow DEBUG: Task 3424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3424 has memoization hash 38d1362a4669da004af1a102001b471a parsl.dataflow.memoization INFO: Task 3424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3424 try 0 launched on executor multi with executor id 3425 parsl.dataflow.dflow INFO: Standard output for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e50a56aa-3d4d-430d-8cc2-aa836de1be3d_isrForCrosstalkSources_855430_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e50a56aa-3d4d-430d-8cc2-aa836de1be3d_isrForCrosstalkSources_855430_37.stderr parsl.dataflow.dflow DEBUG: Task 3425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3425 has memoization hash 11325940b38f2a0f419e52b3f065e5ea parsl.dataflow.memoization INFO: Task 3425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3425 try 0 launched on executor multi with executor id 3426 parsl.dataflow.dflow INFO: Standard output for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c97ac78d-099f-4672-b7e5-c0d0c5ac231b_isrForCrosstalkSources_855410_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c97ac78d-099f-4672-b7e5-c0d0c5ac231b_isrForCrosstalkSources_855410_16.stderr parsl.dataflow.dflow DEBUG: Task 3426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3426 has memoization hash 2aa943888de40c5e2451eb9be5188cf8 parsl.dataflow.memoization INFO: Task 3426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3426 try 0 launched on executor multi with executor id 3427 parsl.dataflow.dflow INFO: Standard output for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/daea178a-440c-41f6-a8fb-fa174512c70b_isrForCrosstalkSources_855429_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/daea178a-440c-41f6-a8fb-fa174512c70b_isrForCrosstalkSources_855429_11.stderr parsl.dataflow.dflow DEBUG: Task 3427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3427 has memoization hash 6fd3e361d642925cfd602c7b018ef8b1 parsl.dataflow.memoization INFO: Task 3427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3427 try 0 launched on executor multi with executor id 3428 parsl.dataflow.dflow INFO: Standard output for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/52b4a8d5-4874-4169-b32a-f25652b34b9e_isrForCrosstalkSources_855411_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/52b4a8d5-4874-4169-b32a-f25652b34b9e_isrForCrosstalkSources_855411_62.stderr parsl.dataflow.dflow DEBUG: Task 3428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3428 has memoization hash 777ca244d48d2bc47f75c7ad37997a92 parsl.dataflow.memoization INFO: Task 3428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3428 try 0 launched on executor multi with executor id 3429 parsl.dataflow.dflow INFO: Standard output for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f0a4a63a-e794-4fb3-9105-b4219a67592b_isrForCrosstalkSources_855424_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f0a4a63a-e794-4fb3-9105-b4219a67592b_isrForCrosstalkSources_855424_48.stderr parsl.dataflow.dflow DEBUG: Task 3429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3429 has memoization hash e5523f4da28b60cfdbb3ee66d3ffd1a8 parsl.dataflow.memoization INFO: Task 3429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3429 try 0 launched on executor multi with executor id 3430 parsl.dataflow.dflow INFO: Standard output for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7cddf91d-1fe6-4c69-8f5d-0ae668d1739f_isrForCrosstalkSources_855419_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7cddf91d-1fe6-4c69-8f5d-0ae668d1739f_isrForCrosstalkSources_855419_10.stderr parsl.dataflow.dflow DEBUG: Task 3430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3430 has memoization hash d7f57a405a9813dc9a51c874381e1c12 parsl.dataflow.memoization INFO: Task 3430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3430 try 0 launched on executor multi with executor id 3431 parsl.dataflow.dflow INFO: Standard output for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a86efdff-4641-4ca5-9ea8-e50afcd0f1b5_isrForCrosstalkSources_855456_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a86efdff-4641-4ca5-9ea8-e50afcd0f1b5_isrForCrosstalkSources_855456_36.stderr parsl.dataflow.dflow DEBUG: Task 3431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3431 has memoization hash c58285a047bd4d210959fdc887247e7c parsl.dataflow.memoization INFO: Task 3431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3431 try 0 launched on executor multi with executor id 3432 parsl.dataflow.dflow INFO: Standard output for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/3700e20e-9474-46d8-87a3-3a67ff26a03c_isrForCrosstalkSources_855446_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/3700e20e-9474-46d8-87a3-3a67ff26a03c_isrForCrosstalkSources_855446_39.stderr parsl.dataflow.dflow DEBUG: Task 3432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3432 has memoization hash 69c2e47753e25f29c031e9f9842f5492 parsl.dataflow.memoization INFO: Task 3432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3432 try 0 launched on executor multi with executor id 3433 parsl.dataflow.dflow INFO: Standard output for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85cc5a34-b767-40e7-beb9-90d268ac4fae_isrForCrosstalkSources_855413_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85cc5a34-b767-40e7-beb9-90d268ac4fae_isrForCrosstalkSources_855413_23.stderr parsl.dataflow.dflow DEBUG: Task 3433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3433 has memoization hash c2559acdb34c2fa6bb759b31e04e4f91 parsl.dataflow.memoization INFO: Task 3433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3433 try 0 launched on executor multi with executor id 3434 parsl.dataflow.dflow INFO: Standard output for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/0237d247-1bec-4ac7-bf8c-5d55d0a380d7_isrForCrosstalkSources_855448_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/0237d247-1bec-4ac7-bf8c-5d55d0a380d7_isrForCrosstalkSources_855448_11.stderr parsl.dataflow.dflow DEBUG: Task 3434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3434 has memoization hash d9a297d98a38916735f194682fa8643a parsl.dataflow.memoization INFO: Task 3434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3434 try 0 launched on executor multi with executor id 3435 parsl.dataflow.dflow INFO: Standard output for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5095af1-3adf-46a7-a434-b4244137a1b4_isrForCrosstalkSources_855421_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5095af1-3adf-46a7-a434-b4244137a1b4_isrForCrosstalkSources_855421_34.stderr parsl.dataflow.dflow DEBUG: Task 3435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3435 has memoization hash ee8d4f74476e2c21b0c876df32e5f9df parsl.dataflow.memoization INFO: Task 3435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3435 try 0 launched on executor multi with executor id 3436 parsl.dataflow.dflow INFO: Standard output for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ba56838-af5b-4dab-8ece-4d94e75869b8_isrForCrosstalkSources_855442_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ba56838-af5b-4dab-8ece-4d94e75869b8_isrForCrosstalkSources_855442_51.stderr parsl.dataflow.dflow DEBUG: Task 3436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3436 has memoization hash f0f1bbd64d4a7eaa83c5b0a6cf4820ce parsl.dataflow.memoization INFO: Task 3436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3436 try 0 launched on executor multi with executor id 3437 parsl.dataflow.dflow INFO: Standard output for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8ecb4921-441c-463a-ae48-086d8c792534_isrForCrosstalkSources_855400_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8ecb4921-441c-463a-ae48-086d8c792534_isrForCrosstalkSources_855400_8.stderr parsl.dataflow.dflow DEBUG: Task 3437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3437 has memoization hash 9900fb052453001a91d60febd16f8ed0 parsl.dataflow.memoization INFO: Task 3437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3437 try 0 launched on executor multi with executor id 3438 parsl.dataflow.dflow INFO: Standard output for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0cd0dc86-c5da-4194-b0e8-69d7ff744a85_isrForCrosstalkSources_855383_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0cd0dc86-c5da-4194-b0e8-69d7ff744a85_isrForCrosstalkSources_855383_40.stderr parsl.dataflow.dflow DEBUG: Task 3438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3438 has memoization hash 9296e54d6a4ee15910d043df0eeb8552 parsl.dataflow.memoization INFO: Task 3438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3438 try 0 launched on executor multi with executor id 3439 parsl.dataflow.dflow INFO: Standard output for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7cfdac6c-1089-41a0-880a-78126cb2ca10_isrForCrosstalkSources_855414_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7cfdac6c-1089-41a0-880a-78126cb2ca10_isrForCrosstalkSources_855414_47.stderr parsl.dataflow.dflow DEBUG: Task 3439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3439 has memoization hash f87131eb4e2b5395e250cba6e2f5ef1f parsl.dataflow.memoization INFO: Task 3439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3439 try 0 launched on executor multi with executor id 3440 parsl.dataflow.dflow INFO: Standard output for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/affa7222-7884-44ab-b5db-d7cbf821b1b2_isrForCrosstalkSources_855390_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/affa7222-7884-44ab-b5db-d7cbf821b1b2_isrForCrosstalkSources_855390_19.stderr parsl.dataflow.dflow DEBUG: Task 3440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3440 has memoization hash 374ea5039284ace9c2497467980a0b01 parsl.dataflow.memoization INFO: Task 3440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3440 try 0 launched on executor multi with executor id 3441 parsl.dataflow.dflow INFO: Standard output for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/32f760e8-043c-4f7f-8699-200cda225b38_isrForCrosstalkSources_855428_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/32f760e8-043c-4f7f-8699-200cda225b38_isrForCrosstalkSources_855428_61.stderr parsl.dataflow.dflow DEBUG: Task 3441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3441 has memoization hash bc83e33c85d9061e250454222d74a19e parsl.dataflow.memoization INFO: Task 3441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3441 try 0 launched on executor multi with executor id 3442 parsl.dataflow.dflow INFO: Standard output for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/88f573b8-081a-4ddd-83d8-2cc1b595e838_isrForCrosstalkSources_855424_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/88f573b8-081a-4ddd-83d8-2cc1b595e838_isrForCrosstalkSources_855424_35.stderr parsl.dataflow.dflow DEBUG: Task 3442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3442 has memoization hash cfce970a5f0390beab6f04c0f56b947a parsl.dataflow.memoization INFO: Task 3442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5f1c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3442 try 0 launched on executor multi with executor id 3443 parsl.dataflow.dflow INFO: Standard output for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c8df001a-7dea-4ae0-948c-4bd62e1a8020_isrForCrosstalkSources_855413_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c8df001a-7dea-4ae0-948c-4bd62e1a8020_isrForCrosstalkSources_855413_12.stderr parsl.dataflow.dflow DEBUG: Task 3443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3443 has memoization hash 6834abf18b0fb9b57f8e3f3e5fd7c788 parsl.dataflow.memoization INFO: Task 3443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3443 try 0 launched on executor multi with executor id 3444 parsl.dataflow.dflow INFO: Standard output for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/5df8726b-a1f6-49dd-8b4e-5ef4b2af8857_isrForCrosstalkSources_855380_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/5df8726b-a1f6-49dd-8b4e-5ef4b2af8857_isrForCrosstalkSources_855380_57.stderr parsl.dataflow.dflow DEBUG: Task 3444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3444 has memoization hash d61fa030a397f79f380f61a3a10aafc5 parsl.dataflow.memoization INFO: Task 3444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3444 try 0 launched on executor multi with executor id 3445 parsl.dataflow.dflow INFO: Standard output for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/26c17532-63fa-41b7-b9b6-6d6f698c4999_isrForCrosstalkSources_855400_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/26c17532-63fa-41b7-b9b6-6d6f698c4999_isrForCrosstalkSources_855400_20.stderr parsl.dataflow.dflow DEBUG: Task 3445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3445 has memoization hash ee34f1245bd28546ce995dfc159df9ef parsl.dataflow.memoization INFO: Task 3445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3445 try 0 launched on executor multi with executor id 3446 parsl.dataflow.dflow INFO: Standard output for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c001cc44-4abc-4809-9f3d-ccd46e2116e5_isrForCrosstalkSources_855459_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c001cc44-4abc-4809-9f3d-ccd46e2116e5_isrForCrosstalkSources_855459_23.stderr parsl.dataflow.dflow DEBUG: Task 3446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3446 has memoization hash c53e03d4b73ce61f28f8eefc1293ff16 parsl.dataflow.memoization INFO: Task 3446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3446 try 0 launched on executor multi with executor id 3447 parsl.dataflow.dflow INFO: Standard output for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5b41249f-aefe-4c17-9173-a125efeaf69a_isrForCrosstalkSources_855445_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5b41249f-aefe-4c17-9173-a125efeaf69a_isrForCrosstalkSources_855445_19.stderr parsl.dataflow.dflow DEBUG: Task 3447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3447 has memoization hash 32ae092917af3a23a35e316d587f0fd3 parsl.dataflow.memoization INFO: Task 3447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3447 try 0 launched on executor multi with executor id 3448 parsl.dataflow.dflow INFO: Standard output for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d3102e7a-571b-4672-a616-481c6766a3a6_isrForCrosstalkSources_855407_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d3102e7a-571b-4672-a616-481c6766a3a6_isrForCrosstalkSources_855407_36.stderr parsl.dataflow.dflow DEBUG: Task 3448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3448 has memoization hash 9a878768a7d5e97fc2c7f5233ff7716b parsl.dataflow.memoization INFO: Task 3448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3448 try 0 launched on executor multi with executor id 3449 parsl.dataflow.dflow INFO: Standard output for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/12a9d19e-9e59-4625-92c1-44c1fc93a643_isrForCrosstalkSources_855428_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/12a9d19e-9e59-4625-92c1-44c1fc93a643_isrForCrosstalkSources_855428_5.stderr parsl.dataflow.dflow DEBUG: Task 3449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3449 has memoization hash bbef0870c140d8f47b13c6ccb746786d parsl.dataflow.memoization INFO: Task 3449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3449 try 0 launched on executor multi with executor id 3450 parsl.dataflow.dflow INFO: Standard output for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/bdc9a131-1688-40fa-8ddd-2da3ff2ecf7b_isrForCrosstalkSources_855449_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/bdc9a131-1688-40fa-8ddd-2da3ff2ecf7b_isrForCrosstalkSources_855449_43.stderr parsl.dataflow.dflow DEBUG: Task 3450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3450 has memoization hash 94d550d30dcc0a09aadd6bb3a3727193 parsl.dataflow.memoization INFO: Task 3450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c76014dd8aaa16c78db73880b182be with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e955e7c3-7083-4611-bff4-6f28e9c55bec_isrForCrosstalkSources_855402_12.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e955e7c3-7083-4611-bff4-6f28e9c55bec_isrForCrosstalkSources_855402_12.stderr parsl.dataflow.dflow INFO: Parsl task 3450 try 0 launched on executor multi with executor id 3451 parsl.dataflow.dflow INFO: Standard output for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3b2d67d5-15fd-4c28-a0db-616ff937f631_isrForCrosstalkSources_855429_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3b2d67d5-15fd-4c28-a0db-616ff937f631_isrForCrosstalkSources_855429_7.stderr parsl.dataflow.dflow DEBUG: Task 3451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3451 has memoization hash 6e6e827f052039d421127a7441931ad5 parsl.dataflow.memoization INFO: Task 3451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3451 try 0 launched on executor multi with executor id 3452 parsl.dataflow.dflow INFO: Standard output for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c2141cea-ebe4-47e8-8cef-eaaf6af81824_isrForCrosstalkSources_855393_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c2141cea-ebe4-47e8-8cef-eaaf6af81824_isrForCrosstalkSources_855393_29.stderr parsl.dataflow.dflow DEBUG: Task 3452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3452 has memoization hash d8b0d1957803033c0f03c191ebe478f9 parsl.dataflow.memoization INFO: Task 3452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3452 try 0 launched on executor multi with executor id 3453 parsl.dataflow.dflow INFO: Standard output for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/907572b2-f7cd-49a4-b7bb-c823864b0a57_isrForCrosstalkSources_855433_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/907572b2-f7cd-49a4-b7bb-c823864b0a57_isrForCrosstalkSources_855433_8.stderr parsl.dataflow.dflow DEBUG: Task 3453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3453 has memoization hash adef6f68d621306c40b31b7a5c4c028a parsl.dataflow.memoization INFO: Task 3453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3453 try 0 launched on executor multi with executor id 3454 parsl.dataflow.dflow INFO: Standard output for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/e19f3b26-cd27-4d92-ab3c-5dd203c8eade_isrForCrosstalkSources_855446_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/e19f3b26-cd27-4d92-ab3c-5dd203c8eade_isrForCrosstalkSources_855446_34.stderr parsl.dataflow.dflow DEBUG: Task 3454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3454 has memoization hash e5e6a5f12018921aff03dd2783c147b9 parsl.dataflow.memoization INFO: Task 3454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3454 try 0 launched on executor multi with executor id 3455 parsl.dataflow.dflow INFO: Standard output for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ed2b08-0c5c-42c7-a18e-a542a009413e_isrForCrosstalkSources_855396_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ed2b08-0c5c-42c7-a18e-a542a009413e_isrForCrosstalkSources_855396_11.stderr parsl.dataflow.dflow DEBUG: Task 3455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3455 has memoization hash b824932bb767db7acbc3842b0ca20291 parsl.dataflow.memoization INFO: Task 3455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3455 try 0 launched on executor multi with executor id 3456 parsl.dataflow.dflow INFO: Standard output for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/a505484b-7c28-465d-91d6-a9c0aec6a4a2_isrForCrosstalkSources_855425_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/a505484b-7c28-465d-91d6-a9c0aec6a4a2_isrForCrosstalkSources_855425_12.stderr parsl.dataflow.dflow DEBUG: Task 3456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3456 has memoization hash dae8e510434defca6f9dc92f0ba64cbb parsl.dataflow.memoization INFO: Task 3456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3456 try 0 launched on executor multi with executor id 3457 parsl.dataflow.dflow INFO: Standard output for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d56b2bf1-bef8-49a9-bca5-2c7b6dc28a36_isrForCrosstalkSources_855459_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d56b2bf1-bef8-49a9-bca5-2c7b6dc28a36_isrForCrosstalkSources_855459_30.stderr parsl.dataflow.dflow DEBUG: Task 3457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3457 has memoization hash 24ed8e3569eb5ff61cc7b9edc9ce3fd7 parsl.dataflow.memoization INFO: Task 3457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3457 try 0 launched on executor multi with executor id 3458 parsl.dataflow.dflow INFO: Standard output for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/166ca430-e38c-4a5b-a186-df6e3a4884e2_isrForCrosstalkSources_855444_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/166ca430-e38c-4a5b-a186-df6e3a4884e2_isrForCrosstalkSources_855444_33.stderr parsl.dataflow.dflow DEBUG: Task 3458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3458 has memoization hash f2317590b4d0c7f9e54fad753bcbfe0d parsl.dataflow.memoization INFO: Task 3458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3458 try 0 launched on executor multi with executor id 3459 parsl.dataflow.dflow INFO: Standard output for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/984f5a86-869f-4b02-a5d2-0b31ec7f7f89_isrForCrosstalkSources_855400_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/984f5a86-869f-4b02-a5d2-0b31ec7f7f89_isrForCrosstalkSources_855400_21.stderr parsl.dataflow.dflow DEBUG: Task 3459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3459 has memoization hash b52956f73a50e2fd8a9b928e90e32c27 parsl.dataflow.memoization INFO: Task 3459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3459 try 0 launched on executor multi with executor id 3460 parsl.dataflow.dflow INFO: Standard output for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3eb45eb1-2efd-48fa-9625-df920e9c0fe2_isrForCrosstalkSources_855402_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3eb45eb1-2efd-48fa-9625-df920e9c0fe2_isrForCrosstalkSources_855402_3.stderr parsl.dataflow.dflow DEBUG: Task 3460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3460 has memoization hash 02477c60fc3ec478012f4534c8f3cafc parsl.dataflow.memoization INFO: Task 3460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9de30d163c9e84bb2a11d462951bf04 with result from task 519 parsl.dataflow.dflow INFO: Parsl task 3460 try 0 launched on executor multi with executor id 3461 parsl.dataflow.dflow INFO: Standard output for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2deb78d0-1f66-4fcc-bbf8-b44cb7e5d958_isrForCrosstalkSources_855401_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2deb78d0-1f66-4fcc-bbf8-b44cb7e5d958_isrForCrosstalkSources_855401_35.stderr parsl.dataflow.dflow DEBUG: Task 3461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3461 has memoization hash 7cec3543fe957d33f680e0397cc9269e parsl.dataflow.memoization INFO: Task 3461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0a767632-35ea-4790-8677-532071a7d052_isrForCrosstalkSources_855403_18.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0a767632-35ea-4790-8677-532071a7d052_isrForCrosstalkSources_855403_18.stderr parsl.dataflow.dflow INFO: Parsl task 3461 try 0 launched on executor multi with executor id 3462 parsl.dataflow.dflow INFO: Standard output for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5a2cc04c-1889-489d-b5ed-199f4d7812c8_isrForCrosstalkSources_855424_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5a2cc04c-1889-489d-b5ed-199f4d7812c8_isrForCrosstalkSources_855424_12.stderr parsl.dataflow.dflow DEBUG: Task 3462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3462 has memoization hash 341f27e7db41c99e32172e1ecb21b843 parsl.dataflow.memoization INFO: Task 3462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3462 try 0 launched on executor multi with executor id 3463 parsl.dataflow.dflow INFO: Standard output for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a3f82cc8-a7d7-4585-aa70-e0e00d23ab7a_isrForCrosstalkSources_855426_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a3f82cc8-a7d7-4585-aa70-e0e00d23ab7a_isrForCrosstalkSources_855426_29.stderr parsl.dataflow.dflow DEBUG: Task 3463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3463 has memoization hash fd8e5bb911736a0e45ffd8be292af479 parsl.dataflow.memoization INFO: Task 3463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3463 try 0 launched on executor multi with executor id 3464 parsl.dataflow.dflow INFO: Standard output for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a5dcce0a-e390-44f2-8d39-250a7c2bb015_isrForCrosstalkSources_855398_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a5dcce0a-e390-44f2-8d39-250a7c2bb015_isrForCrosstalkSources_855398_47.stderr parsl.dataflow.dflow DEBUG: Task 3464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3464 has memoization hash 4d70387b413a9a7a9315dc8ba4bcef36 parsl.dataflow.memoization INFO: Task 3464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3464 try 0 launched on executor multi with executor id 3465 parsl.dataflow.dflow INFO: Standard output for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/517b2a07-83f5-468a-81bb-b79b7869a36c_isrForCrosstalkSources_855457_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/517b2a07-83f5-468a-81bb-b79b7869a36c_isrForCrosstalkSources_855457_23.stderr parsl.dataflow.dflow DEBUG: Task 3465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3465 has memoization hash 086ffe9f70b979effe04f5a28413047c parsl.dataflow.memoization INFO: Task 3465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3465 try 0 launched on executor multi with executor id 3466 parsl.dataflow.dflow INFO: Standard output for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f69c18bf-982b-48cb-9b13-7b4b9da6113b_isrForCrosstalkSources_855382_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f69c18bf-982b-48cb-9b13-7b4b9da6113b_isrForCrosstalkSources_855382_45.stderr parsl.dataflow.dflow DEBUG: Task 3466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3466 has memoization hash 644228c7152b17457666b4a835c24110 parsl.dataflow.memoization INFO: Task 3466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3466 try 0 launched on executor multi with executor id 3467 parsl.dataflow.dflow INFO: Standard output for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f8ae4a49-6851-4550-a6a1-98140540ad13_isrForCrosstalkSources_855455_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f8ae4a49-6851-4550-a6a1-98140540ad13_isrForCrosstalkSources_855455_16.stderr parsl.dataflow.dflow DEBUG: Task 3467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3467 has memoization hash 5b24f687a331f61497007f06d67a93d2 parsl.dataflow.memoization INFO: Task 3467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3467 try 0 launched on executor multi with executor id 3468 parsl.dataflow.dflow INFO: Standard output for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2a8bd97-0e06-4552-8b62-d9c87d48ad06_isrForCrosstalkSources_855402_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2a8bd97-0e06-4552-8b62-d9c87d48ad06_isrForCrosstalkSources_855402_13.stderr parsl.dataflow.dflow DEBUG: Task 3468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3468 has memoization hash 3af3e1825176e857bc280c3a9dd785d6 parsl.dataflow.memoization INFO: Task 3468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3468 try 0 launched on executor multi with executor id 3469 parsl.dataflow.dflow INFO: Standard output for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6279700-b9a7-4883-a26f-886b28501392_isrForCrosstalkSources_855412_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6279700-b9a7-4883-a26f-886b28501392_isrForCrosstalkSources_855412_28.stderr parsl.dataflow.dflow DEBUG: Task 3469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3469 has memoization hash cf76b745c8c4c92632a76a0f32497c11 parsl.dataflow.memoization INFO: Task 3469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3469 try 0 launched on executor multi with executor id 3470 parsl.dataflow.dflow INFO: Standard output for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/87d72053-8afe-4112-9021-03148a11cf2f_isrForCrosstalkSources_855430_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/87d72053-8afe-4112-9021-03148a11cf2f_isrForCrosstalkSources_855430_1.stderr parsl.dataflow.dflow DEBUG: Task 3470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3470 has memoization hash 5e628f8e1217b798d72b619e9fb34fdb parsl.dataflow.memoization INFO: Task 3470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3470 try 0 launched on executor multi with executor id 3471 parsl.dataflow.dflow INFO: Standard output for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/55ec0e6e-2eae-473a-98b4-bc4f9db61dfc_isrForCrosstalkSources_855400_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/55ec0e6e-2eae-473a-98b4-bc4f9db61dfc_isrForCrosstalkSources_855400_22.stderr parsl.dataflow.dflow DEBUG: Task 3471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3471 has memoization hash 8a6d1296afc89d901cc4bdf67c0e1a22 parsl.dataflow.memoization INFO: Task 3471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3471 try 0 launched on executor multi with executor id 3472 parsl.dataflow.dflow INFO: Standard output for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cc2091b6-682a-4b93-a569-37dd47227b22_isrForCrosstalkSources_855456_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cc2091b6-682a-4b93-a569-37dd47227b22_isrForCrosstalkSources_855456_33.stderr parsl.dataflow.dflow DEBUG: Task 3472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3472 has memoization hash 996df03a3c2dffd6cf3368926b4bc266 parsl.dataflow.memoization INFO: Task 3472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:35 proc_lsst.multi:170 [INFO] [multi] job_id 16964170 proc_lsst.multi INFO: [multi] job_id 16964170 2024-03-09 14:51:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-09 14:51:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964170 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964170 parsl.executors.status_handling DEBUG: Launched block 144 on executor multi with job ID 16964170 parsl.executors.status_handling INFO: Allocated block ID 145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-145 2024-03-09 14:51:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 2024-03-09 14:51:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-09 14:51:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3472 try 0 launched on executor multi with executor id 3473 parsl.dataflow.dflow INFO: Standard output for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cccb1036-0f3e-4259-b2b9-178cf974c4fa_isrForCrosstalkSources_855441_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cccb1036-0f3e-4259-b2b9-178cf974c4fa_isrForCrosstalkSources_855441_5.stderr parsl.dataflow.dflow DEBUG: Task 3473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3473 has memoization hash 0b7c17f8ce0f5d2551cd8f4ff89ab3f8 parsl.dataflow.memoization INFO: Task 3473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3473 try 0 launched on executor multi with executor id 3474 parsl.dataflow.dflow INFO: Standard output for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a4703adb-6136-45e1-ba0e-bf1b5b9da8d0_isrForCrosstalkSources_855388_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a4703adb-6136-45e1-ba0e-bf1b5b9da8d0_isrForCrosstalkSources_855388_29.stderr parsl.dataflow.dflow DEBUG: Task 3474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3474 has memoization hash 24cbb86368ee23dbe0db9b56f6651897 parsl.dataflow.memoization INFO: Task 3474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3474 try 0 launched on executor multi with executor id 3475 parsl.dataflow.dflow INFO: Standard output for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/50480b30-9bb0-4f43-8329-bf71cec85de3_isrForCrosstalkSources_855406_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/50480b30-9bb0-4f43-8329-bf71cec85de3_isrForCrosstalkSources_855406_36.stderr parsl.dataflow.dflow DEBUG: Task 3475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3475 has memoization hash e82ba1de454785fc730d684e98b305fd parsl.dataflow.memoization INFO: Task 3475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a61d76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3475 try 0 launched on executor multi with executor id 3476 parsl.dataflow.dflow INFO: Standard output for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3ee0f50e-03df-4478-88ea-1d3fbdf03441_isrForCrosstalkSources_855400_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3ee0f50e-03df-4478-88ea-1d3fbdf03441_isrForCrosstalkSources_855400_49.stderr parsl.dataflow.dflow DEBUG: Task 3476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3476 has memoization hash 5c0429bdd4d6acf5479b685bed3ac47d parsl.dataflow.memoization INFO: Task 3476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3476 try 0 launched on executor multi with executor id 3477 parsl.dataflow.dflow INFO: Standard output for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b9786dde-fdfd-44ee-808a-eca086ea3077_isrForCrosstalkSources_855439_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b9786dde-fdfd-44ee-808a-eca086ea3077_isrForCrosstalkSources_855439_29.stderr parsl.dataflow.dflow DEBUG: Task 3477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3477 has memoization hash ec742e97abc506df9b48ce382982e95e parsl.dataflow.memoization INFO: Task 3477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3477 try 0 launched on executor multi with executor id 3478 parsl.dataflow.dflow INFO: Standard output for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7050be6d-e5aa-4d90-ab5f-505446a2eb79_isrForCrosstalkSources_855405_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7050be6d-e5aa-4d90-ab5f-505446a2eb79_isrForCrosstalkSources_855405_18.stderr parsl.dataflow.dflow DEBUG: Task 3478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3478 has memoization hash 3f8a4f9bfc21352a58c7a915d6dccc70 parsl.dataflow.memoization INFO: Task 3478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3478 try 0 launched on executor multi with executor id 3479 parsl.dataflow.dflow INFO: Standard output for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/aaef5961-5040-4b4c-b788-79902e9bfd96_isrForCrosstalkSources_855414_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/aaef5961-5040-4b4c-b788-79902e9bfd96_isrForCrosstalkSources_855414_13.stderr parsl.dataflow.dflow DEBUG: Task 3479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3479 has memoization hash 890837931747e95cb8f5f5fb08ec5531 parsl.dataflow.memoization INFO: Task 3479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3479 try 0 launched on executor multi with executor id 3480 parsl.dataflow.dflow INFO: Standard output for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e2c96db1-e0e6-4420-a5a0-d7ebc64236ee_isrForCrosstalkSources_855408_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e2c96db1-e0e6-4420-a5a0-d7ebc64236ee_isrForCrosstalkSources_855408_62.stderr parsl.dataflow.dflow DEBUG: Task 3480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3480 has memoization hash 27036107d697c2c2c92b447cf8fda4f7 parsl.dataflow.memoization INFO: Task 3480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3480 try 0 launched on executor multi with executor id 3481 parsl.dataflow.dflow INFO: Standard output for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4226e1f4-21ea-462d-80ff-906cab3626b3_isrForCrosstalkSources_855440_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4226e1f4-21ea-462d-80ff-906cab3626b3_isrForCrosstalkSources_855440_54.stderr parsl.dataflow.dflow DEBUG: Task 3481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3481 has memoization hash bfc547e63a37c0fa4fecbbe1794bc1e5 parsl.dataflow.memoization INFO: Task 3481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3481 try 0 launched on executor multi with executor id 3482 parsl.dataflow.dflow INFO: Standard output for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/6c4cad36-72a7-4f78-af82-962dcd1d5c55_isrForCrosstalkSources_855406_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/6c4cad36-72a7-4f78-af82-962dcd1d5c55_isrForCrosstalkSources_855406_53.stderr parsl.dataflow.dflow DEBUG: Task 3482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3482 has memoization hash 6f6023aaeaa9d5cec887b04352067732 parsl.dataflow.memoization INFO: Task 3482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3482 try 0 launched on executor multi with executor id 3483 parsl.dataflow.dflow INFO: Standard output for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d7d1069f-117c-4885-9a91-d49179ad93dd_isrForCrosstalkSources_855454_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d7d1069f-117c-4885-9a91-d49179ad93dd_isrForCrosstalkSources_855454_38.stderr parsl.dataflow.dflow DEBUG: Task 3483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3483 has memoization hash 567246927b6881bbd324538badb876b9 parsl.dataflow.memoization INFO: Task 3483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3483 try 0 launched on executor multi with executor id 3484 parsl.dataflow.dflow INFO: Standard output for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/d3e366d4-fb5e-4d52-a0ce-3a1403076d7e_isrForCrosstalkSources_855452_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/d3e366d4-fb5e-4d52-a0ce-3a1403076d7e_isrForCrosstalkSources_855452_12.stderr parsl.dataflow.dflow DEBUG: Task 3484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3484 has memoization hash 79e36ab0b0c8b235a7133f00e26efdd1 parsl.dataflow.memoization INFO: Task 3484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3484 try 0 launched on executor multi with executor id 3485 parsl.dataflow.dflow INFO: Standard output for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4eaf419b-9dc1-4f88-beb7-8cda288fdbff_isrForCrosstalkSources_855404_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4eaf419b-9dc1-4f88-beb7-8cda288fdbff_isrForCrosstalkSources_855404_19.stderr parsl.dataflow.dflow DEBUG: Task 3485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3485 has memoization hash 1a252abe31bda9ed3fa235539b94b847 parsl.dataflow.memoization INFO: Task 3485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3485 try 0 launched on executor multi with executor id 3486 parsl.dataflow.dflow INFO: Standard output for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/27c5674b-b6d3-4df4-8545-dfd0a3f068af_isrForCrosstalkSources_855416_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/27c5674b-b6d3-4df4-8545-dfd0a3f068af_isrForCrosstalkSources_855416_2.stderr parsl.dataflow.dflow DEBUG: Task 3486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3486 has memoization hash 1037cc1fc6203348f490a1c4897a8862 parsl.dataflow.memoization INFO: Task 3486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3486 try 0 launched on executor multi with executor id 3487 parsl.dataflow.dflow INFO: Standard output for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/632c8718-46a1-4022-ad56-653d8f835739_isrForCrosstalkSources_855447_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/632c8718-46a1-4022-ad56-653d8f835739_isrForCrosstalkSources_855447_55.stderr parsl.dataflow.dflow DEBUG: Task 3487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3487 has memoization hash 7fdf80dfe8de7d54a4a8c3c62d281a26 parsl.dataflow.memoization INFO: Task 3487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3487 try 0 launched on executor multi with executor id 3488 parsl.dataflow.dflow INFO: Standard output for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/55de4cc2-73b9-48f2-90a2-d5d5d23a10c3_isrForCrosstalkSources_855425_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/55de4cc2-73b9-48f2-90a2-d5d5d23a10c3_isrForCrosstalkSources_855425_37.stderr parsl.dataflow.dflow DEBUG: Task 3488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3488 has memoization hash 3338eb778acf353eb32e9214e11dbbdf parsl.dataflow.memoization INFO: Task 3488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3488 try 0 launched on executor multi with executor id 3489 parsl.dataflow.dflow INFO: Standard output for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f453cbd2-37b3-40d7-b4cd-9b5c14045173_isrForCrosstalkSources_855444_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f453cbd2-37b3-40d7-b4cd-9b5c14045173_isrForCrosstalkSources_855444_17.stderr parsl.dataflow.dflow DEBUG: Task 3489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3489 has memoization hash 716d129426fa0275371f905d1de7e1eb parsl.dataflow.memoization INFO: Task 3489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49bc0752ca57d0360a446bfea5076bcf with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e92ace5e-9aff-4b84-87f1-d1fc27fa3029_isrForCrosstalkSources_855379_38.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e92ace5e-9aff-4b84-87f1-d1fc27fa3029_isrForCrosstalkSources_855379_38.stderr parsl.dataflow.dflow INFO: Parsl task 3489 try 0 launched on executor multi with executor id 3490 parsl.dataflow.dflow INFO: Standard output for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fbbf4bd7-8410-4125-951d-5810f14bcf0a_isrForCrosstalkSources_855382_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fbbf4bd7-8410-4125-951d-5810f14bcf0a_isrForCrosstalkSources_855382_12.stderr parsl.dataflow.dflow DEBUG: Task 3490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3490 has memoization hash 6b10fa431fc7c96307af9855dbbfc66c parsl.dataflow.memoization INFO: Task 3490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3490 try 0 launched on executor multi with executor id 3491 parsl.dataflow.dflow INFO: Standard output for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/31a51583-e414-4591-982f-c026508d2be3_isrForCrosstalkSources_855426_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/31a51583-e414-4591-982f-c026508d2be3_isrForCrosstalkSources_855426_25.stderr parsl.dataflow.dflow DEBUG: Task 3491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3491 has memoization hash 5cc9410951f80ff43b38fbfbd7d4aa7a parsl.dataflow.memoization INFO: Task 3491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3491 try 0 launched on executor multi with executor id 3492 parsl.dataflow.dflow INFO: Standard output for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/743af19f-3743-4b75-afd5-164e3c9afcbd_isrForCrosstalkSources_855401_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/743af19f-3743-4b75-afd5-164e3c9afcbd_isrForCrosstalkSources_855401_41.stderr parsl.dataflow.dflow DEBUG: Task 3492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3492 has memoization hash 1de74ced59c8e3eb92b567e8a3f40794 parsl.dataflow.memoization INFO: Task 3492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40083d46ce92ad6d8fc27c7e4c6b82f2 with result from task 512 parsl.dataflow.dflow INFO: Parsl task 3492 try 0 launched on executor multi with executor id 3493 parsl.dataflow.dflow INFO: Standard output for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04de1dc1-0b18-4abd-a233-8797ee17829e_isrForCrosstalkSources_855440_50.stdout parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Standard error for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04de1dc1-0b18-4abd-a233-8797ee17829e_isrForCrosstalkSources_855440_50.stderr parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 3493 will be sent to executor multi parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8f6ee23-809d-442a-9543-5b8fc8046b06_isrForCrosstalkSources_855399_52.stdout parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8f6ee23-809d-442a-9543-5b8fc8046b06_isrForCrosstalkSources_855399_52.stderr parsl.dataflow.dflow INFO: Task 3493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.dflow DEBUG: Task 3493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Storing app cache entry 340810548af47e36162b8eb4370c0ad1 with result from task 514 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3493 has memoization hash 7bca4474c3d2f4a09a15e0379ed0a5d0 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.memoization INFO: Task 3493 had no result in cache parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/769a141a-5f76-4e0f-8bf0-3b4ae4dd9c08_isrForCrosstalkSources_855403_28.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/769a141a-5f76-4e0f-8bf0-3b4ae4dd9c08_isrForCrosstalkSources_855403_28.stderr parsl.dataflow.dflow INFO: Parsl task 3493 try 0 launched on executor multi with executor id 3494 parsl.dataflow.dflow INFO: Standard output for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/be143c66-a5f8-4c88-89e1-2eb1aa80f41a_isrForCrosstalkSources_855392_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/be143c66-a5f8-4c88-89e1-2eb1aa80f41a_isrForCrosstalkSources_855392_8.stderr parsl.dataflow.dflow DEBUG: Task 3494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3494 has memoization hash d068899311ff4d963a61d7596619f095 parsl.dataflow.memoization INFO: Task 3494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3494 try 0 launched on executor multi with executor id 3495 parsl.dataflow.dflow INFO: Standard output for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2fb5ac56-f970-42af-a369-f419b1dd4308_isrForCrosstalkSources_855397_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2fb5ac56-f970-42af-a369-f419b1dd4308_isrForCrosstalkSources_855397_20.stderr parsl.dataflow.dflow DEBUG: Task 3495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3495 has memoization hash 6439b2d7dac99413fece1ee129700d7a parsl.dataflow.memoization INFO: Task 3495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3495 try 0 launched on executor multi with executor id 3496 parsl.dataflow.dflow INFO: Standard output for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/40c2a8cc-ea0a-497b-89e1-effeadf60cae_isrForCrosstalkSources_855438_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/40c2a8cc-ea0a-497b-89e1-effeadf60cae_isrForCrosstalkSources_855438_53.stderr parsl.dataflow.dflow DEBUG: Task 3496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3496 has memoization hash ddfa39f884035c92d8e744966e73d6ac parsl.dataflow.memoization INFO: Task 3496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3496 try 0 launched on executor multi with executor id 3497 parsl.dataflow.dflow INFO: Standard output for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7cfb1d5c-50ef-4557-a1bd-f7222b462dc3_isrForCrosstalkSources_855458_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7cfb1d5c-50ef-4557-a1bd-f7222b462dc3_isrForCrosstalkSources_855458_59.stderr parsl.dataflow.dflow DEBUG: Task 3497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3497 has memoization hash 8327a67250b072a5b7360743e403cd8d parsl.dataflow.memoization INFO: Task 3497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3497 try 0 launched on executor multi with executor id 3498 parsl.dataflow.dflow INFO: Standard output for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/eb9ae3e2-dc59-47a6-a9d3-4f95b0e01b33_isrForCrosstalkSources_855394_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/eb9ae3e2-dc59-47a6-a9d3-4f95b0e01b33_isrForCrosstalkSources_855394_17.stderr parsl.dataflow.dflow DEBUG: Task 3498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3498 has memoization hash 51d6a189002cfcd06e940ebbbeb20dd9 parsl.dataflow.memoization INFO: Task 3498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3498 try 0 launched on executor multi with executor id 3499 parsl.dataflow.dflow INFO: Standard output for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5db3f52f-effc-4255-81cd-c2e86405c5e6_isrForCrosstalkSources_855382_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5db3f52f-effc-4255-81cd-c2e86405c5e6_isrForCrosstalkSources_855382_18.stderr parsl.dataflow.dflow DEBUG: Task 3499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3499 has memoization hash ce5781cb2430f7a3e2fbe9f12f117ca0 parsl.dataflow.memoization INFO: Task 3499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3499 try 0 launched on executor multi with executor id 3500 parsl.dataflow.dflow INFO: Standard output for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9fa9e5ca-ca7d-4c04-8d26-932324bc785f_isrForCrosstalkSources_855427_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9fa9e5ca-ca7d-4c04-8d26-932324bc785f_isrForCrosstalkSources_855427_53.stderr parsl.dataflow.dflow DEBUG: Task 3500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3500 has memoization hash 3bbb3e7ce5179ef67b2d3577f7211103 parsl.dataflow.memoization INFO: Task 3500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3500 try 0 launched on executor multi with executor id 3501 parsl.dataflow.dflow INFO: Standard output for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f1b248ee-09cf-483d-8d15-e0a64f889688_isrForCrosstalkSources_855391_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f1b248ee-09cf-483d-8d15-e0a64f889688_isrForCrosstalkSources_855391_1.stderr parsl.dataflow.dflow DEBUG: Task 3501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3501 has memoization hash a3842905ff48376dffa3ff118e9f1af6 parsl.dataflow.memoization INFO: Task 3501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3501 try 0 launched on executor multi with executor id 3502 parsl.dataflow.dflow INFO: Standard output for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/495c4db4-2bcc-4f6e-aab3-b19195e05d2e_isrForCrosstalkSources_855415_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/495c4db4-2bcc-4f6e-aab3-b19195e05d2e_isrForCrosstalkSources_855415_39.stderr parsl.dataflow.dflow DEBUG: Task 3502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3502 has memoization hash ab8f7954a53c1c0744487bacbe8a2bba parsl.dataflow.memoization INFO: Task 3502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3502 try 0 launched on executor multi with executor id 3503 parsl.dataflow.dflow INFO: Standard output for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/736b0e7f-bfec-40a0-bfae-d210a796d396_isrForCrosstalkSources_855438_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/736b0e7f-bfec-40a0-bfae-d210a796d396_isrForCrosstalkSources_855438_39.stderr parsl.dataflow.dflow DEBUG: Task 3503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3503 has memoization hash cd828b32b13731f0411b840c28964edf parsl.dataflow.memoization INFO: Task 3503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3503 try 0 launched on executor multi with executor id 3504 parsl.dataflow.dflow INFO: Standard output for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d236dcdb-dade-4bcb-a89b-f5235cd57a2d_isrForCrosstalkSources_855430_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d236dcdb-dade-4bcb-a89b-f5235cd57a2d_isrForCrosstalkSources_855430_43.stderr parsl.dataflow.dflow DEBUG: Task 3504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3504 has memoization hash 521797f947f49d058c7772c1576ecb4d parsl.dataflow.memoization INFO: Task 3504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3504 try 0 launched on executor multi with executor id 3505 parsl.dataflow.dflow INFO: Standard output for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6dd07147-7195-40ea-b90c-41544a8dd14f_isrForCrosstalkSources_855397_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6dd07147-7195-40ea-b90c-41544a8dd14f_isrForCrosstalkSources_855397_6.stderr parsl.dataflow.dflow DEBUG: Task 3505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3505 has memoization hash de3e715d003dd378bcf58343ebc3f32e parsl.dataflow.memoization INFO: Task 3505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3505 try 0 launched on executor multi with executor id 3506 parsl.dataflow.dflow INFO: Standard output for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/63188aaa-4701-4f5e-8e8a-24fd74815db4_isrForCrosstalkSources_855392_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/63188aaa-4701-4f5e-8e8a-24fd74815db4_isrForCrosstalkSources_855392_33.stderr parsl.dataflow.dflow DEBUG: Task 3506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3506 has memoization hash b97629b05f8a0e2e10902f90ef6dc5c8 parsl.dataflow.memoization INFO: Task 3506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3506 try 0 launched on executor multi with executor id 3507 parsl.dataflow.dflow INFO: Standard output for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aa38a2f-5ac7-444e-8bc9-0ac99179b63a_isrForCrosstalkSources_855411_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aa38a2f-5ac7-444e-8bc9-0ac99179b63a_isrForCrosstalkSources_855411_4.stderr parsl.dataflow.dflow DEBUG: Task 3507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3507 has memoization hash bbbd6ccabef1ecd4134739345070d27a parsl.dataflow.memoization INFO: Task 3507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3507 try 0 launched on executor multi with executor id 3508 parsl.dataflow.dflow INFO: Standard output for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/91cebbaf-5877-4c1a-9bd9-0858e995f7dd_isrForCrosstalkSources_855419_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/91cebbaf-5877-4c1a-9bd9-0858e995f7dd_isrForCrosstalkSources_855419_52.stderr parsl.dataflow.dflow DEBUG: Task 3508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3508 has memoization hash 0e48a8091641ed7ef3250cd3b34d277b parsl.dataflow.memoization INFO: Task 3508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3508 try 0 launched on executor multi with executor id 3509 parsl.dataflow.dflow INFO: Standard output for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/14425dfb-5615-4049-901d-662a0203d68d_isrForCrosstalkSources_855388_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/14425dfb-5615-4049-901d-662a0203d68d_isrForCrosstalkSources_855388_33.stderr parsl.dataflow.dflow DEBUG: Task 3509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3509 has memoization hash 5e0480802544b04963c4138386bb7db7 parsl.dataflow.memoization INFO: Task 3509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3509 try 0 launched on executor multi with executor id 3510 parsl.dataflow.dflow INFO: Standard output for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e3ff23f6-1177-41b5-997d-1ab2a11ecafc_isrForCrosstalkSources_855380_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e3ff23f6-1177-41b5-997d-1ab2a11ecafc_isrForCrosstalkSources_855380_21.stderr parsl.dataflow.dflow DEBUG: Task 3510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3510 has memoization hash 584d02c33198b605250c57133261bfb9 parsl.dataflow.memoization INFO: Task 3510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3510 try 0 launched on executor multi with executor id 3511 parsl.dataflow.dflow INFO: Standard output for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/f9f94987-86a9-43d5-9c44-b46cf1e5ba04_isrForCrosstalkSources_855408_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/f9f94987-86a9-43d5-9c44-b46cf1e5ba04_isrForCrosstalkSources_855408_58.stderr parsl.dataflow.dflow DEBUG: Task 3511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3511 has memoization hash 5caffc90d45d69c47f0bdf0cd65fc5a7 parsl.dataflow.memoization INFO: Task 3511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3511 try 0 launched on executor multi with executor id 3512 parsl.dataflow.dflow INFO: Standard output for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/61041d6d-ffa2-4c31-b270-46e53784f74b_isrForCrosstalkSources_855434_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/61041d6d-ffa2-4c31-b270-46e53784f74b_isrForCrosstalkSources_855434_21.stderr parsl.dataflow.dflow DEBUG: Task 3512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3512 has memoization hash 2f51b4c827d51c95a8c0b92446f72d57 parsl.dataflow.memoization INFO: Task 3512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3512 try 0 launched on executor multi with executor id 3513 parsl.dataflow.dflow INFO: Standard output for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/eea59fe7-88b4-4995-b3f7-8f1897b38c09_isrForCrosstalkSources_855453_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/eea59fe7-88b4-4995-b3f7-8f1897b38c09_isrForCrosstalkSources_855453_3.stderr parsl.dataflow.dflow DEBUG: Task 3513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3513 has memoization hash b97704d3822803174cb7d90f4b149c08 parsl.dataflow.memoization INFO: Task 3513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3513 try 0 launched on executor multi with executor id 3514 parsl.dataflow.dflow INFO: Standard output for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/eac1223b-e185-44b6-b58b-3a104bbbb303_isrForCrosstalkSources_855390_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/eac1223b-e185-44b6-b58b-3a104bbbb303_isrForCrosstalkSources_855390_8.stderr parsl.dataflow.dflow DEBUG: Task 3514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3514 has memoization hash cbe9360d589d08c8380813ed9a3af113 parsl.dataflow.memoization INFO: Task 3514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3514 try 0 launched on executor multi with executor id 3515 parsl.dataflow.dflow INFO: Standard output for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6e5ff343-1f86-41a3-bcd0-a77f410dcbca_isrForCrosstalkSources_855457_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6e5ff343-1f86-41a3-bcd0-a77f410dcbca_isrForCrosstalkSources_855457_29.stderr parsl.dataflow.dflow DEBUG: Task 3515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3515 has memoization hash acfd8d940d5a3d68d77ea32934a2e5ea parsl.dataflow.memoization INFO: Task 3515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3515 try 0 launched on executor multi with executor id 3516 parsl.dataflow.dflow INFO: Standard output for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bcadaf5a-64cc-4f2e-9a60-0b935ffd5f72_isrForCrosstalkSources_855408_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bcadaf5a-64cc-4f2e-9a60-0b935ffd5f72_isrForCrosstalkSources_855408_44.stderr parsl.dataflow.dflow DEBUG: Task 3516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3516 has memoization hash 38e36b8a1fa54b8de35aa81f8a47a758 parsl.dataflow.memoization INFO: Task 3516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3516 try 0 launched on executor multi with executor id 3517 parsl.dataflow.dflow INFO: Standard output for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e722c02-c54f-461e-8449-4204f0b2280c_isrForCrosstalkSources_855451_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e722c02-c54f-461e-8449-4204f0b2280c_isrForCrosstalkSources_855451_38.stderr parsl.dataflow.dflow DEBUG: Task 3517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3517 has memoization hash a919e94faa609388a121a9b44aff372e parsl.dataflow.memoization INFO: Task 3517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3517 try 0 launched on executor multi with executor id 3518 parsl.dataflow.dflow INFO: Standard output for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/344b2869-e8bd-4393-8075-44ac5b673c93_isrForCrosstalkSources_855446_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/344b2869-e8bd-4393-8075-44ac5b673c93_isrForCrosstalkSources_855446_29.stderr parsl.dataflow.dflow DEBUG: Task 3518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3518 has memoization hash ba3ab4523f3463263ffa259af85f82d8 parsl.dataflow.memoization INFO: Task 3518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3518 try 0 launched on executor multi with executor id 3519 parsl.dataflow.dflow INFO: Standard output for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32ab099e-41e0-4e13-bb99-95bac65ae896_isrForCrosstalkSources_855447_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32ab099e-41e0-4e13-bb99-95bac65ae896_isrForCrosstalkSources_855447_25.stderr parsl.dataflow.dflow DEBUG: Task 3519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3519 has memoization hash c8584768beb4971c9fefdf58db1992e3 parsl.dataflow.memoization INFO: Task 3519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3519 try 0 launched on executor multi with executor id 3520 parsl.dataflow.dflow INFO: Standard output for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5ad89aa5-14cf-49af-adea-6a98351c8baf_isrForCrosstalkSources_855400_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5ad89aa5-14cf-49af-adea-6a98351c8baf_isrForCrosstalkSources_855400_58.stderr parsl.dataflow.dflow DEBUG: Task 3520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3520 has memoization hash 876ff2bfbecab3c276b25396b939fb95 parsl.dataflow.memoization INFO: Task 3520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3520 try 0 launched on executor multi with executor id 3521 parsl.dataflow.dflow INFO: Standard output for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0e60acd6-10ad-42c3-91eb-19969e15c325_isrForCrosstalkSources_855435_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0e60acd6-10ad-42c3-91eb-19969e15c325_isrForCrosstalkSources_855435_43.stderr parsl.dataflow.dflow DEBUG: Task 3521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3521 has memoization hash 4dc56c0e450b9570ca4279db566cbb74 parsl.dataflow.memoization INFO: Task 3521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3521 try 0 launched on executor multi with executor id 3522 parsl.dataflow.dflow INFO: Standard output for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a17d691-0212-47a4-b058-c5d07d721c20_isrForCrosstalkSources_855409_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a17d691-0212-47a4-b058-c5d07d721c20_isrForCrosstalkSources_855409_26.stderr parsl.dataflow.dflow DEBUG: Task 3522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3522 has memoization hash 063ac63f91996f374fb064bfaf2a500c parsl.dataflow.memoization INFO: Task 3522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3522 try 0 launched on executor multi with executor id 3523 parsl.dataflow.dflow INFO: Standard output for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/48b28375-c8bc-4f21-9b09-7ecb4f0b602d_isrForCrosstalkSources_855415_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/48b28375-c8bc-4f21-9b09-7ecb4f0b602d_isrForCrosstalkSources_855415_1.stderr parsl.dataflow.dflow DEBUG: Task 3523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3523 has memoization hash 9a9667e3bfc6759a5c914b6dd6f131fd parsl.dataflow.memoization INFO: Task 3523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3523 try 0 launched on executor multi with executor id 3524 parsl.dataflow.dflow INFO: Standard output for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f77b44e1-8895-47d7-b579-a4a6d9611a6a_isrForCrosstalkSources_855384_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f77b44e1-8895-47d7-b579-a4a6d9611a6a_isrForCrosstalkSources_855384_15.stderr parsl.dataflow.dflow DEBUG: Task 3524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3524 has memoization hash ff62663d6389d19b161c52d7c598eff3 parsl.dataflow.memoization INFO: Task 3524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3524 try 0 launched on executor multi with executor id 3525 parsl.dataflow.dflow INFO: Standard output for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b86a07a6-a973-4e8a-9e66-1c2bdf32e4e7_isrForCrosstalkSources_855407_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b86a07a6-a973-4e8a-9e66-1c2bdf32e4e7_isrForCrosstalkSources_855407_60.stderr parsl.dataflow.dflow DEBUG: Task 3525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3525 has memoization hash 9b13a05f02151b1adf5525c1c9fe977d parsl.dataflow.memoization INFO: Task 3525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3525 try 0 launched on executor multi with executor id 3526 parsl.dataflow.dflow INFO: Standard output for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1111bb95-9e35-4fc8-9955-e6df5be5db26_isrForCrosstalkSources_855380_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1111bb95-9e35-4fc8-9955-e6df5be5db26_isrForCrosstalkSources_855380_31.stderr parsl.dataflow.dflow DEBUG: Task 3526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3526 has memoization hash a7e2f326ef7344cddfaf9fe24deb2fe4 parsl.dataflow.memoization INFO: Task 3526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3526 try 0 launched on executor multi with executor id 3527 parsl.dataflow.dflow INFO: Standard output for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ec06bbbe-b3b1-4ae2-a987-e93ca6910221_isrForCrosstalkSources_855388_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ec06bbbe-b3b1-4ae2-a987-e93ca6910221_isrForCrosstalkSources_855388_39.stderr parsl.dataflow.dflow DEBUG: Task 3527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3527 has memoization hash d11211f495609cd0eeb11639939681bf parsl.dataflow.memoization INFO: Task 3527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3527 try 0 launched on executor multi with executor id 3528 parsl.dataflow.dflow INFO: Standard output for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/01e190a5-8570-4d91-b388-d3e6f2e5ecff_isrForCrosstalkSources_855393_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/01e190a5-8570-4d91-b388-d3e6f2e5ecff_isrForCrosstalkSources_855393_53.stderr parsl.dataflow.dflow DEBUG: Task 3528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3528 has memoization hash 21ae3d9f70288980de8e593597ff62cc parsl.dataflow.memoization INFO: Task 3528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3528 try 0 launched on executor multi with executor id 3529 parsl.dataflow.dflow INFO: Standard output for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/af05fd9f-5e99-4e0b-825c-60aeafeee911_isrForCrosstalkSources_855417_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/af05fd9f-5e99-4e0b-825c-60aeafeee911_isrForCrosstalkSources_855417_4.stderr parsl.dataflow.dflow DEBUG: Task 3529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3529 has memoization hash 17fb824cdcf30e0228f008a376f93139 parsl.dataflow.memoization INFO: Task 3529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3529 try 0 launched on executor multi with executor id 3530 parsl.dataflow.dflow INFO: Standard output for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff419ba9-31c3-4dbb-b78f-86a0def16b7b_isrForCrosstalkSources_855405_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff419ba9-31c3-4dbb-b78f-86a0def16b7b_isrForCrosstalkSources_855405_48.stderr parsl.dataflow.dflow DEBUG: Task 3530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3530 has memoization hash eddd0a428d85ab659a68d4efa0823707 parsl.dataflow.memoization INFO: Task 3530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3530 try 0 launched on executor multi with executor id 3531 parsl.dataflow.dflow INFO: Standard output for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/895aa623-9951-4abe-8e1d-af2d89568384_isrForCrosstalkSources_855395_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/895aa623-9951-4abe-8e1d-af2d89568384_isrForCrosstalkSources_855395_28.stderr parsl.dataflow.dflow DEBUG: Task 3531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3531 has memoization hash a94bcf387ac4be152f7988ec6ce7f947 parsl.dataflow.memoization INFO: Task 3531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3531 try 0 launched on executor multi with executor id 3532 parsl.dataflow.dflow INFO: Standard output for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/67987cbf-cd26-4f2a-8b0f-aa70c5536033_isrForCrosstalkSources_855407_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/67987cbf-cd26-4f2a-8b0f-aa70c5536033_isrForCrosstalkSources_855407_56.stderr parsl.dataflow.dflow DEBUG: Task 3532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3532 has memoization hash c0d408b854cfa3af84327fe8d46dbfde parsl.dataflow.memoization INFO: Task 3532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3532 try 0 launched on executor multi with executor id 3533 parsl.dataflow.dflow INFO: Standard output for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f4659f73-be9c-4973-9f14-499010998dc8_isrForCrosstalkSources_855397_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f4659f73-be9c-4973-9f14-499010998dc8_isrForCrosstalkSources_855397_48.stderr parsl.dataflow.dflow DEBUG: Task 3533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3533 has memoization hash 05581a07f84a4c5c98c3952a0933106f parsl.dataflow.memoization INFO: Task 3533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3533 try 0 launched on executor multi with executor id 3534 parsl.dataflow.dflow INFO: Standard output for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/9060c877-9179-42bc-81a5-50f3036ec3a1_isrForCrosstalkSources_855433_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/9060c877-9179-42bc-81a5-50f3036ec3a1_isrForCrosstalkSources_855433_9.stderr parsl.dataflow.dflow DEBUG: Task 3534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3534 has memoization hash fd685c9b0e2f72c4027f2836ccfcaaf5 parsl.dataflow.memoization INFO: Task 3534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3534 try 0 launched on executor multi with executor id 3535 parsl.dataflow.dflow INFO: Standard output for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/89d331b8-4c9f-4e05-a1f7-cff9fb66e961_isrForCrosstalkSources_855400_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/89d331b8-4c9f-4e05-a1f7-cff9fb66e961_isrForCrosstalkSources_855400_27.stderr parsl.dataflow.dflow DEBUG: Task 3535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3535 has memoization hash 08a509d5197e67ae505ba459b95d15a9 parsl.dataflow.memoization INFO: Task 3535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3535 try 0 launched on executor multi with executor id 3536 parsl.dataflow.dflow INFO: Standard output for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c14149e0-3e36-4cb6-97ad-222e666a73d5_isrForCrosstalkSources_855397_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c14149e0-3e36-4cb6-97ad-222e666a73d5_isrForCrosstalkSources_855397_29.stderr parsl.dataflow.dflow DEBUG: Task 3536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3536 has memoization hash 09f0fbbab9c657d6daf310f737e254b6 parsl.dataflow.memoization INFO: Task 3536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3536 try 0 launched on executor multi with executor id 3537 parsl.dataflow.dflow INFO: Standard output for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7dd824e9-7a35-42c7-a512-c948c8d5c366_isrForCrosstalkSources_855424_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7dd824e9-7a35-42c7-a512-c948c8d5c366_isrForCrosstalkSources_855424_6.stderr parsl.dataflow.dflow DEBUG: Task 3537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3537 has memoization hash 2cdff3e16df7242e0be439661726f0c2 parsl.dataflow.memoization INFO: Task 3537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3537 try 0 launched on executor multi with executor id 3538 parsl.dataflow.dflow INFO: Standard output for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/51fa4cd9-3234-4a73-8370-e09ca19c8616_isrForCrosstalkSources_855441_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/51fa4cd9-3234-4a73-8370-e09ca19c8616_isrForCrosstalkSources_855441_10.stderr parsl.dataflow.dflow DEBUG: Task 3538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3538 has memoization hash 21a686607d97a085dd73767fdaac511c parsl.dataflow.memoization INFO: Task 3538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3538 try 0 launched on executor multi with executor id 3539 parsl.dataflow.dflow INFO: Standard output for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c2bac645-4dd9-4b97-afe0-928ae0897ce5_isrForCrosstalkSources_855394_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c2bac645-4dd9-4b97-afe0-928ae0897ce5_isrForCrosstalkSources_855394_29.stderr parsl.dataflow.dflow DEBUG: Task 3539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3539 has memoization hash e2a75320714998ca176b7c5331a2248d parsl.dataflow.memoization INFO: Task 3539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3539 try 0 launched on executor multi with executor id 3540 parsl.dataflow.dflow INFO: Standard output for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58678676-eb43-4140-8bf3-169a63924fba_isrForCrosstalkSources_855407_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58678676-eb43-4140-8bf3-169a63924fba_isrForCrosstalkSources_855407_49.stderr parsl.dataflow.dflow DEBUG: Task 3540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3540 has memoization hash da9269b5c3eb446024e91be60d7af5cd parsl.dataflow.memoization INFO: Task 3540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3540 try 0 launched on executor multi with executor id 3541 parsl.dataflow.dflow INFO: Standard output for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ce9d494-2458-4e13-9242-cf909220e914_isrForCrosstalkSources_855396_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ce9d494-2458-4e13-9242-cf909220e914_isrForCrosstalkSources_855396_41.stderr parsl.dataflow.dflow DEBUG: Task 3541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3541 has memoization hash fb4668c77cd90236f1d6b79a00b335d8 parsl.dataflow.memoization INFO: Task 3541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3541 try 0 launched on executor multi with executor id 3542 parsl.dataflow.dflow INFO: Standard output for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/5f1dbe3c-d355-421c-83cb-f94940d348de_isrForCrosstalkSources_855459_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/5f1dbe3c-d355-421c-83cb-f94940d348de_isrForCrosstalkSources_855459_27.stderr parsl.dataflow.dflow DEBUG: Task 3542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3542 has memoization hash 3abd9d35777170fc77bbcaaf24717c99 parsl.dataflow.memoization INFO: Task 3542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3542 try 0 launched on executor multi with executor id 3543 parsl.dataflow.dflow INFO: Standard output for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/4d93cb8d-526e-496f-89c7-659ae8fe8e4b_isrForCrosstalkSources_855453_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/4d93cb8d-526e-496f-89c7-659ae8fe8e4b_isrForCrosstalkSources_855453_44.stderr parsl.dataflow.dflow DEBUG: Task 3543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3543 has memoization hash 11f506ba076b9d72b655bf2ee2baf826 parsl.dataflow.memoization INFO: Task 3543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3543 try 0 launched on executor multi with executor id 3544 parsl.dataflow.dflow INFO: Standard output for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b22d770a-7622-4780-bb88-8658c0181dab_isrForCrosstalkSources_855438_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b22d770a-7622-4780-bb88-8658c0181dab_isrForCrosstalkSources_855438_30.stderr parsl.dataflow.dflow DEBUG: Task 3544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3544 has memoization hash 823ad9d0f1be4ee1932e22b62c76d181 parsl.dataflow.memoization INFO: Task 3544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3544 try 0 launched on executor multi with executor id 3545 parsl.dataflow.dflow INFO: Standard output for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/74f4c5f0-49e0-4499-b999-7d4e2d1e34f4_isrForCrosstalkSources_855413_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/74f4c5f0-49e0-4499-b999-7d4e2d1e34f4_isrForCrosstalkSources_855413_9.stderr parsl.dataflow.dflow DEBUG: Task 3545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3545 has memoization hash c1d521f40a1bede78b30a318288d4b3a parsl.dataflow.memoization INFO: Task 3545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3545 try 0 launched on executor multi with executor id 3546 parsl.dataflow.dflow INFO: Standard output for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/95870a49-d685-4d7c-9156-d19edc8fede3_isrForCrosstalkSources_855435_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/95870a49-d685-4d7c-9156-d19edc8fede3_isrForCrosstalkSources_855435_14.stderr parsl.dataflow.dflow DEBUG: Task 3546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3546 has memoization hash 5d0120bea582f55b8d2a80f856ee306f parsl.dataflow.memoization INFO: Task 3546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3546 try 0 launched on executor multi with executor id 3547 parsl.dataflow.dflow INFO: Standard output for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2351cf1d-98db-4152-8904-b7d13e794f4f_isrForCrosstalkSources_855417_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2351cf1d-98db-4152-8904-b7d13e794f4f_isrForCrosstalkSources_855417_23.stderr parsl.dataflow.dflow DEBUG: Task 3547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3547 has memoization hash 08f9b3af4712f5e2ccb72791895ce1d6 parsl.dataflow.memoization INFO: Task 3547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3547 try 0 launched on executor multi with executor id 3548 parsl.dataflow.dflow INFO: Standard output for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c1b2f402-584f-489d-8f7d-9f9ef07d94d8_isrForCrosstalkSources_855417_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c1b2f402-584f-489d-8f7d-9f9ef07d94d8_isrForCrosstalkSources_855417_57.stderr parsl.dataflow.dflow DEBUG: Task 3548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3548 has memoization hash 34fb9123731e2584109df57386edb7f7 parsl.dataflow.memoization INFO: Task 3548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3548 try 0 launched on executor multi with executor id 3549 parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Standard output for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/075fb689-7fba-4377-bdab-2d5b3a9abe3b_isrForCrosstalkSources_855435_3.stdout parsl.dataflow.memoization DEBUG: Storing app cache entry 62b1f435396e6f70eaffe7803a954d3a with result from task 513 parsl.dataflow.dflow INFO: Standard error for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/075fb689-7fba-4377-bdab-2d5b3a9abe3b_isrForCrosstalkSources_855435_3.stderr parsl.dataflow.dflow DEBUG: Task 3549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3549 has memoization hash 735021a4a53e23025745f4845338f3cf parsl.dataflow.memoization INFO: Task 3549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ec00687d-c6ce-440d-a39b-676388a36be1_isrForCrosstalkSources_855440_61.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ec00687d-c6ce-440d-a39b-676388a36be1_isrForCrosstalkSources_855440_61.stderr parsl.dataflow.dflow INFO: Parsl task 3549 try 0 launched on executor multi with executor id 3550 parsl.dataflow.dflow INFO: Standard output for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2a1baa68-dcdc-49a9-8884-b7d84ed20cbd_isrForCrosstalkSources_855406_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2a1baa68-dcdc-49a9-8884-b7d84ed20cbd_isrForCrosstalkSources_855406_13.stderr parsl.dataflow.dflow DEBUG: Task 3550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3550 has memoization hash b01d771752b11fb35209640c32b6177f parsl.dataflow.memoization INFO: Task 3550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3550 try 0 launched on executor multi with executor id 3551 parsl.dataflow.dflow INFO: Standard output for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7140b809-35f7-44cb-95f4-c7500be2b39f_isrForCrosstalkSources_855412_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7140b809-35f7-44cb-95f4-c7500be2b39f_isrForCrosstalkSources_855412_2.stderr parsl.dataflow.dflow DEBUG: Task 3551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3551 has memoization hash f941ca4373482e8eefa580b62dc8dbea parsl.dataflow.memoization INFO: Task 3551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3551 try 0 launched on executor multi with executor id 3552 parsl.dataflow.dflow INFO: Standard output for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5002d2ee-b24d-4f20-afc9-2e0d4773e111_isrForCrosstalkSources_855425_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5002d2ee-b24d-4f20-afc9-2e0d4773e111_isrForCrosstalkSources_855425_20.stderr parsl.dataflow.dflow DEBUG: Task 3552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3552 has memoization hash 45a34999839717c48d585c977317934a parsl.dataflow.memoization INFO: Task 3552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3552 try 0 launched on executor multi with executor id 3553 parsl.dataflow.dflow INFO: Standard output for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f73e6ad1-cc06-4254-99d2-1bd4a07f4750_isrForCrosstalkSources_855420_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f73e6ad1-cc06-4254-99d2-1bd4a07f4750_isrForCrosstalkSources_855420_5.stderr parsl.dataflow.dflow DEBUG: Task 3553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3553 has memoization hash 590795d2b8881da06ad4de6db2e1ef5f parsl.dataflow.memoization INFO: Task 3553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3553 try 0 launched on executor multi with executor id 3554 parsl.dataflow.dflow INFO: Standard output for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/73675dac-6580-47b5-8c83-807d806e9c7b_isrForCrosstalkSources_855431_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/73675dac-6580-47b5-8c83-807d806e9c7b_isrForCrosstalkSources_855431_45.stderr parsl.dataflow.dflow DEBUG: Task 3554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3554 has memoization hash ab8c29f980dd63e7e5a81326ece4fa14 parsl.dataflow.memoization INFO: Task 3554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3554 try 0 launched on executor multi with executor id 3555 parsl.dataflow.dflow INFO: Standard output for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/423859e9-0d1d-49f9-97f4-e4c05fb38a6c_isrForCrosstalkSources_855393_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/423859e9-0d1d-49f9-97f4-e4c05fb38a6c_isrForCrosstalkSources_855393_55.stderr parsl.dataflow.dflow DEBUG: Task 3555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3555 has memoization hash bf4dbba610dd8e08b5a0890766ba3e52 parsl.dataflow.memoization INFO: Task 3555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3555 try 0 launched on executor multi with executor id 3556 parsl.dataflow.dflow INFO: Standard output for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cb313bb2-983a-4ad2-99b6-17d8b25f838e_isrForCrosstalkSources_855447_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cb313bb2-983a-4ad2-99b6-17d8b25f838e_isrForCrosstalkSources_855447_17.stderr parsl.dataflow.dflow DEBUG: Task 3556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3556 has memoization hash 5e67449ba4d73d7e5751bb0785edcc14 parsl.dataflow.memoization INFO: Task 3556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3556 try 0 launched on executor multi with executor id 3557 parsl.dataflow.dflow INFO: Standard output for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/83f2759d-27b2-4012-8972-d5a6ce1fd9bf_isrForCrosstalkSources_855450_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/83f2759d-27b2-4012-8972-d5a6ce1fd9bf_isrForCrosstalkSources_855450_4.stderr parsl.dataflow.dflow DEBUG: Task 3557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3557 has memoization hash b91873525f64f7f3064239a4ec5d2e5d parsl.dataflow.memoization INFO: Task 3557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3557 try 0 launched on executor multi with executor id 3558 parsl.dataflow.dflow INFO: Standard output for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e1877135-a346-405a-a1cc-b0b28b0fd76d_isrForCrosstalkSources_855409_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e1877135-a346-405a-a1cc-b0b28b0fd76d_isrForCrosstalkSources_855409_55.stderr parsl.dataflow.dflow DEBUG: Task 3558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3558 has memoization hash f62823c6030141fd828bbeb8c9ceb512 parsl.dataflow.memoization INFO: Task 3558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3558 try 0 launched on executor multi with executor id 3559 parsl.dataflow.dflow INFO: Standard output for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b017210-7aeb-4dfd-a6e7-511c7893ea6f_isrForCrosstalkSources_855398_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b017210-7aeb-4dfd-a6e7-511c7893ea6f_isrForCrosstalkSources_855398_3.stderr parsl.dataflow.dflow DEBUG: Task 3559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3559 has memoization hash 9a67c16c2b626211776ccaba5193f7b0 parsl.dataflow.memoization INFO: Task 3559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3559 try 0 launched on executor multi with executor id 3560 parsl.dataflow.dflow INFO: Standard output for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a67a79e6-b962-40ef-85da-aa440cb252da_isrForCrosstalkSources_855395_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a67a79e6-b962-40ef-85da-aa440cb252da_isrForCrosstalkSources_855395_27.stderr parsl.dataflow.dflow DEBUG: Task 3560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3560 has memoization hash b4511ede14faf5dd7ef8b53138facfc0 parsl.dataflow.memoization INFO: Task 3560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3560 try 0 launched on executor multi with executor id 3561 parsl.dataflow.dflow INFO: Standard output for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6aa510fc-88d0-4821-9e9d-06a8e3b8195c_isrForCrosstalkSources_855401_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6aa510fc-88d0-4821-9e9d-06a8e3b8195c_isrForCrosstalkSources_855401_7.stderr parsl.dataflow.dflow DEBUG: Task 3561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3561 has memoization hash 7a2c881f7ea3a974e19e2d173c794896 parsl.dataflow.memoization INFO: Task 3561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3561 try 0 launched on executor multi with executor id 3562 parsl.dataflow.dflow INFO: Standard output for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bf861680-9d92-4426-ade4-e0cb1dcf3923_isrForCrosstalkSources_855408_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bf861680-9d92-4426-ade4-e0cb1dcf3923_isrForCrosstalkSources_855408_60.stderr parsl.dataflow.dflow DEBUG: Task 3562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3562 has memoization hash 4401830838d6cb6b2351d6dde4136ac3 parsl.dataflow.memoization INFO: Task 3562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3562 try 0 launched on executor multi with executor id 3563 parsl.dataflow.dflow INFO: Standard output for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/84925ca2-d0c0-4191-8b0e-20ef9d9dcddf_isrForCrosstalkSources_855398_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/84925ca2-d0c0-4191-8b0e-20ef9d9dcddf_isrForCrosstalkSources_855398_30.stderr parsl.dataflow.dflow DEBUG: Task 3563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3563 has memoization hash a31e57451dec7b7705180a17f1bba377 parsl.dataflow.memoization INFO: Task 3563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3563 try 0 launched on executor multi with executor id 3564 parsl.dataflow.dflow INFO: Standard output for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c252f74f-be97-44fa-8d34-238a6ed5b288_isrForCrosstalkSources_855427_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c252f74f-be97-44fa-8d34-238a6ed5b288_isrForCrosstalkSources_855427_6.stderr parsl.dataflow.dflow DEBUG: Task 3564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3564 has memoization hash 5950776301b7e3ad5fc16129721ac462 parsl.dataflow.memoization INFO: Task 3564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3564 try 0 launched on executor multi with executor id 3565 parsl.dataflow.dflow INFO: Standard output for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/208529ab-15fa-47b2-bb59-119ca1e81cf3_isrForCrosstalkSources_855441_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/208529ab-15fa-47b2-bb59-119ca1e81cf3_isrForCrosstalkSources_855441_59.stderr parsl.dataflow.dflow DEBUG: Task 3565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3565 has memoization hash b83a9f80bbe2badac854acd088089de5 parsl.dataflow.memoization INFO: Task 3565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3565 try 0 launched on executor multi with executor id 3566 parsl.dataflow.dflow INFO: Standard output for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/69b8195e-cbb1-4d4c-99a4-686f9943f016_isrForCrosstalkSources_855448_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/69b8195e-cbb1-4d4c-99a4-686f9943f016_isrForCrosstalkSources_855448_38.stderr parsl.dataflow.dflow DEBUG: Task 3566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3566 has memoization hash ca7145ebc81e428ae3913ba23c219ccb parsl.dataflow.memoization INFO: Task 3566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3566 try 0 launched on executor multi with executor id 3567 parsl.dataflow.dflow INFO: Standard output for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7ad0a6e7-1c1e-48f6-97cf-95162de592c3_isrForCrosstalkSources_855379_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7ad0a6e7-1c1e-48f6-97cf-95162de592c3_isrForCrosstalkSources_855379_28.stderr parsl.dataflow.dflow DEBUG: Task 3567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3567 has memoization hash 7045c8084aa9f84da97a1491256be9c0 parsl.dataflow.memoization INFO: Task 3567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3567 try 0 launched on executor multi with executor id 3568 parsl.dataflow.dflow INFO: Standard output for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/008a57b0-f81f-4fbc-afd7-cebc141fdcd1_isrForCrosstalkSources_855436_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/008a57b0-f81f-4fbc-afd7-cebc141fdcd1_isrForCrosstalkSources_855436_14.stderr parsl.dataflow.dflow DEBUG: Task 3568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3568 has memoization hash 158598cf6f7e60e4f838e186e653d794 parsl.dataflow.memoization INFO: Task 3568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3568 try 0 launched on executor multi with executor id 3569 parsl.dataflow.dflow INFO: Standard output for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/07b204cc-c189-4076-a9af-9bc46e10d701_isrForCrosstalkSources_855445_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/07b204cc-c189-4076-a9af-9bc46e10d701_isrForCrosstalkSources_855445_17.stderr parsl.dataflow.dflow DEBUG: Task 3569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3569 has memoization hash bbd42eae016554e12883c64d09061b1a parsl.dataflow.memoization INFO: Task 3569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3569 try 0 launched on executor multi with executor id 3570 parsl.dataflow.dflow INFO: Standard output for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4094669c-c20f-4c4f-aec8-c2223058782b_isrForCrosstalkSources_855415_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4094669c-c20f-4c4f-aec8-c2223058782b_isrForCrosstalkSources_855415_51.stderr parsl.dataflow.dflow DEBUG: Task 3570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3570 has memoization hash b79c3e3e292bb19b73ba2e7be422f4d0 parsl.dataflow.memoization INFO: Task 3570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3570 try 0 launched on executor multi with executor id 3571 parsl.dataflow.dflow INFO: Standard output for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0f0900d6-bd6a-486b-919f-a20d0c421761_isrForCrosstalkSources_855446_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0f0900d6-bd6a-486b-919f-a20d0c421761_isrForCrosstalkSources_855446_48.stderr parsl.dataflow.dflow DEBUG: Task 3571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3571 has memoization hash 6236400eb79acfbcbd37d6b1675d2816 parsl.dataflow.memoization INFO: Task 3571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3571 try 0 launched on executor multi with executor id 3572 parsl.dataflow.dflow INFO: Standard output for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/486fcda9-77aa-490a-8097-91bcb88e125b_isrForCrosstalkSources_855394_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/486fcda9-77aa-490a-8097-91bcb88e125b_isrForCrosstalkSources_855394_59.stderr parsl.dataflow.dflow DEBUG: Task 3572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3572 has memoization hash 5273292c3d6009aee9f0e646ade35166 parsl.dataflow.memoization INFO: Task 3572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3572 try 0 launched on executor multi with executor id 3573 parsl.dataflow.dflow INFO: Standard output for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4371254d-56b5-418a-aa9e-a4f2d9f8ce5c_isrForCrosstalkSources_855403_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4371254d-56b5-418a-aa9e-a4f2d9f8ce5c_isrForCrosstalkSources_855403_25.stderr parsl.dataflow.dflow DEBUG: Task 3573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3573 has memoization hash 51f307bbb1a418a30c8dc56540fbc923 parsl.dataflow.memoization INFO: Task 3573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3573 try 0 launched on executor multi with executor id 3574 parsl.dataflow.dflow INFO: Standard output for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0486ccb4-6288-482b-a93b-c56becdb1963_isrForCrosstalkSources_855437_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0486ccb4-6288-482b-a93b-c56becdb1963_isrForCrosstalkSources_855437_60.stderr parsl.dataflow.dflow DEBUG: Task 3574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3574 has memoization hash 4210b4d964607f9290cd86ffb441648c parsl.dataflow.memoization INFO: Task 3574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3574 try 0 launched on executor multi with executor id 3575 parsl.dataflow.dflow INFO: Standard output for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ffc4a808-5dfc-482e-b4e1-b2d89266990f_isrForCrosstalkSources_855436_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ffc4a808-5dfc-482e-b4e1-b2d89266990f_isrForCrosstalkSources_855436_1.stderr parsl.dataflow.dflow DEBUG: Task 3575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3575 has memoization hash cd11729cbb66ab182b18de9a7d43b064 parsl.dataflow.memoization INFO: Task 3575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3575 try 0 launched on executor multi with executor id 3576 parsl.dataflow.dflow INFO: Standard output for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cc4e4b61-4e6b-4f25-af64-f6aa263b6174_isrForCrosstalkSources_855407_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cc4e4b61-4e6b-4f25-af64-f6aa263b6174_isrForCrosstalkSources_855407_58.stderr parsl.dataflow.dflow DEBUG: Task 3576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3576 has memoization hash ea4b0930ce5f3a935ae3c1b9927f6f2b parsl.dataflow.memoization INFO: Task 3576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3576 try 0 launched on executor multi with executor id 3577 parsl.dataflow.dflow INFO: Standard output for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fe837c7-635e-4a7e-a026-29ab8d6786c9_isrForCrosstalkSources_855387_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fe837c7-635e-4a7e-a026-29ab8d6786c9_isrForCrosstalkSources_855387_6.stderr parsl.dataflow.dflow DEBUG: Task 3577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3577 has memoization hash 9df794f2a47b4011d555471dff04e934 parsl.dataflow.memoization INFO: Task 3577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3577 try 0 launched on executor multi with executor id 3578 parsl.dataflow.dflow INFO: Standard output for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b2582232-2a09-48f6-8d8e-f7910c951302_isrForCrosstalkSources_855433_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b2582232-2a09-48f6-8d8e-f7910c951302_isrForCrosstalkSources_855433_29.stderr parsl.dataflow.dflow DEBUG: Task 3578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3578 has memoization hash 639c25de9362d342994e0c87218ac0ec parsl.dataflow.memoization INFO: Task 3578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3578 try 0 launched on executor multi with executor id 3579 parsl.dataflow.dflow INFO: Standard output for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/310f8fda-cfe4-4389-90f6-a642e22d3e8d_isrForCrosstalkSources_855459_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/310f8fda-cfe4-4389-90f6-a642e22d3e8d_isrForCrosstalkSources_855459_8.stderr parsl.dataflow.dflow DEBUG: Task 3579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3579 has memoization hash dc05e5c0cc8f2c507b26b982c8ffa238 parsl.dataflow.memoization INFO: Task 3579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3579 try 0 launched on executor multi with executor id 3580 parsl.dataflow.dflow INFO: Standard output for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8818c5cb-3daf-4b77-865e-cd6ce9817f08_isrForCrosstalkSources_855393_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8818c5cb-3daf-4b77-865e-cd6ce9817f08_isrForCrosstalkSources_855393_54.stderr parsl.dataflow.dflow DEBUG: Task 3580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3580 has memoization hash 08399a30416a83d4573a4ef73baa1be2 parsl.dataflow.memoization INFO: Task 3580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3580 try 0 launched on executor multi with executor id 3581 parsl.dataflow.dflow INFO: Standard output for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b7236540-7e34-4a25-9820-d653658c898f_isrForCrosstalkSources_855440_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b7236540-7e34-4a25-9820-d653658c898f_isrForCrosstalkSources_855440_13.stderr parsl.dataflow.dflow DEBUG: Task 3581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3581 has memoization hash 90c1b1e39887b3a60f55a8519e0b0992 parsl.dataflow.memoization INFO: Task 3581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3581 try 0 launched on executor multi with executor id 3582 parsl.dataflow.dflow INFO: Standard output for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c3b3cf12-5cfa-4b9f-9515-9a94fcc57dc2_isrForCrosstalkSources_855435_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c3b3cf12-5cfa-4b9f-9515-9a94fcc57dc2_isrForCrosstalkSources_855435_10.stderr parsl.dataflow.dflow DEBUG: Task 3582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3582 has memoization hash c6e948ae8b9440f6022e07e5135af956 parsl.dataflow.memoization INFO: Task 3582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1977ba673ccbdcfe999cad1528e0bc2 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b4ea3c07-95e4-414a-acb9-06572ff77d9c_isrForCrosstalkSources_855442_21.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b4ea3c07-95e4-414a-acb9-06572ff77d9c_isrForCrosstalkSources_855442_21.stderr parsl.dataflow.dflow INFO: Parsl task 3582 try 0 launched on executor multi with executor id 3583 parsl.dataflow.dflow INFO: Standard output for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3588ec91-e32a-45fe-a350-76619c1a2af7_isrForCrosstalkSources_855391_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3588ec91-e32a-45fe-a350-76619c1a2af7_isrForCrosstalkSources_855391_21.stderr parsl.dataflow.dflow DEBUG: Task 3583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3583 has memoization hash 182914a35ac0c7540a78c39d12a2c326 parsl.dataflow.memoization INFO: Task 3583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3583 try 0 launched on executor multi with executor id 3584 parsl.dataflow.dflow INFO: Standard output for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/53f2857f-37f0-4cd3-8982-11012bd07448_isrForCrosstalkSources_855436_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/53f2857f-37f0-4cd3-8982-11012bd07448_isrForCrosstalkSources_855436_44.stderr parsl.dataflow.dflow DEBUG: Task 3584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3584 has memoization hash 97c2aa90adb1d45acbc897b402386d07 parsl.dataflow.memoization INFO: Task 3584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3584 try 0 launched on executor multi with executor id 3585 parsl.dataflow.dflow INFO: Standard output for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f601338e-8b75-4c17-9d5e-7849db377a02_isrForCrosstalkSources_855425_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f601338e-8b75-4c17-9d5e-7849db377a02_isrForCrosstalkSources_855425_15.stderr parsl.dataflow.dflow DEBUG: Task 3585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3585 has memoization hash 62b1d68a523d7e268d0ea01ff51f33f5 parsl.dataflow.memoization INFO: Task 3585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3585 try 0 launched on executor multi with executor id 3586 parsl.dataflow.dflow INFO: Standard output for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/81c7a4c0-94e1-4a09-8a92-d16545778a93_isrForCrosstalkSources_855433_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/81c7a4c0-94e1-4a09-8a92-d16545778a93_isrForCrosstalkSources_855433_3.stderr parsl.dataflow.dflow DEBUG: Task 3586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3586 has memoization hash 1464b14b7b03dc216b988caa02939feb parsl.dataflow.memoization INFO: Task 3586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3586 try 0 launched on executor multi with executor id 3587 parsl.dataflow.dflow INFO: Standard output for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/6d359c1d-22f3-4381-82ad-a09a5fafbeee_isrForCrosstalkSources_855399_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/6d359c1d-22f3-4381-82ad-a09a5fafbeee_isrForCrosstalkSources_855399_53.stderr parsl.dataflow.dflow DEBUG: Task 3587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3587 has memoization hash 81332768d8426de860a928a31b651077 parsl.dataflow.memoization INFO: Task 3587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3587 try 0 launched on executor multi with executor id 3588 parsl.dataflow.dflow INFO: Standard output for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/225067e1-ab8d-42e9-801c-31458bc2c3c7_isrForCrosstalkSources_855381_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/225067e1-ab8d-42e9-801c-31458bc2c3c7_isrForCrosstalkSources_855381_1.stderr parsl.dataflow.dflow DEBUG: Task 3588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3588 has memoization hash 25a72272d662ffaaa59a6f73109d5a7f parsl.dataflow.memoization INFO: Task 3588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3588 try 0 launched on executor multi with executor id 3589 parsl.dataflow.dflow INFO: Standard output for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67ecb9fa-6d5d-43a8-a789-4c28949abdbf_isrForCrosstalkSources_855393_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67ecb9fa-6d5d-43a8-a789-4c28949abdbf_isrForCrosstalkSources_855393_42.stderr parsl.dataflow.dflow DEBUG: Task 3589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3589 has memoization hash d13a5ff29968f217933deef74a827f8d parsl.dataflow.memoization INFO: Task 3589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3589 try 0 launched on executor multi with executor id 3590 parsl.dataflow.dflow INFO: Standard output for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b8b5ea09-cd27-4eae-8b2b-109c0c88657d_isrForCrosstalkSources_855453_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b8b5ea09-cd27-4eae-8b2b-109c0c88657d_isrForCrosstalkSources_855453_56.stderr parsl.dataflow.dflow DEBUG: Task 3590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3590 has memoization hash f7dfc0b9d9c20cb7292bd20fb967bdd0 parsl.dataflow.memoization INFO: Task 3590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3590 try 0 launched on executor multi with executor id 3591 parsl.dataflow.dflow INFO: Standard output for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e0ea03d3-ad1c-4dc9-ab47-cd0a14766c9b_isrForCrosstalkSources_855385_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e0ea03d3-ad1c-4dc9-ab47-cd0a14766c9b_isrForCrosstalkSources_855385_52.stderr parsl.dataflow.dflow DEBUG: Task 3591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3591 has memoization hash dc58648d5b1f150fa1a32a9857f2a886 parsl.dataflow.memoization INFO: Task 3591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3591 try 0 launched on executor multi with executor id 3592 parsl.dataflow.dflow INFO: Standard output for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/75d74f6a-e9ff-41db-9647-5dcdee1c4c36_isrForCrosstalkSources_855411_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/75d74f6a-e9ff-41db-9647-5dcdee1c4c36_isrForCrosstalkSources_855411_22.stderr parsl.dataflow.dflow DEBUG: Task 3592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3592 has memoization hash 314732506efc4825ed01fb23ff0b6220 parsl.dataflow.memoization INFO: Task 3592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3592 try 0 launched on executor multi with executor id 3593 parsl.dataflow.dflow INFO: Standard output for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3df8bfa5-68d0-4b6b-87b3-73bc4846ffc5_isrForCrosstalkSources_855383_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3df8bfa5-68d0-4b6b-87b3-73bc4846ffc5_isrForCrosstalkSources_855383_41.stderr parsl.dataflow.dflow DEBUG: Task 3593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3593 has memoization hash 3aabdf3ce770d6b721eb641b86262216 parsl.dataflow.memoization INFO: Task 3593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3593 try 0 launched on executor multi with executor id 3594 parsl.dataflow.dflow INFO: Standard output for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4c7b5dea-8ad2-4be7-924a-56c0545dfbea_isrForCrosstalkSources_855443_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4c7b5dea-8ad2-4be7-924a-56c0545dfbea_isrForCrosstalkSources_855443_10.stderr parsl.dataflow.dflow DEBUG: Task 3594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3594 has memoization hash d0d95035a7d76d37f080f2e73e618d6b parsl.dataflow.memoization INFO: Task 3594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3594 try 0 launched on executor multi with executor id 3595 parsl.dataflow.dflow INFO: Standard output for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/3e25e837-931b-40f9-aae7-4147063e5388_isrForCrosstalkSources_855441_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/3e25e837-931b-40f9-aae7-4147063e5388_isrForCrosstalkSources_855441_11.stderr parsl.dataflow.dflow DEBUG: Task 3595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3595 has memoization hash 8b6002ea242aa1edf8a8fe8d1ad00406 parsl.dataflow.memoization INFO: Task 3595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3595 try 0 launched on executor multi with executor id 3596 parsl.dataflow.dflow INFO: Standard output for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4d8e5c40-5d78-4144-ad25-eb1a6a51d134_isrForCrosstalkSources_855439_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4d8e5c40-5d78-4144-ad25-eb1a6a51d134_isrForCrosstalkSources_855439_34.stderr parsl.dataflow.dflow DEBUG: Task 3596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3596 has memoization hash 0e11c718d290dcdb3e04c447231861e5 parsl.dataflow.memoization INFO: Task 3596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3425973303f5c7498a82d2ad5504ec with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/17e1e1b8-3fc3-4ca4-978e-01faa596a326_isrForCrosstalkSources_855438_12.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/17e1e1b8-3fc3-4ca4-978e-01faa596a326_isrForCrosstalkSources_855438_12.stderr parsl.dataflow.dflow INFO: Parsl task 3596 try 0 launched on executor multi with executor id 3597 parsl.dataflow.dflow INFO: Standard output for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e2ba2e16-92cf-4c29-8807-d70701f6f9f0_isrForCrosstalkSources_855458_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e2ba2e16-92cf-4c29-8807-d70701f6f9f0_isrForCrosstalkSources_855458_36.stderr parsl.dataflow.dflow DEBUG: Task 3597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3597 has memoization hash 8226cacd90c708f3fb569bfe5d032c08 parsl.dataflow.memoization INFO: Task 3597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3597 try 0 launched on executor multi with executor id 3598 parsl.dataflow.dflow INFO: Standard output for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/8aa592c5-09e8-40e7-9ed8-507a0cd55416_isrForCrosstalkSources_855451_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/8aa592c5-09e8-40e7-9ed8-507a0cd55416_isrForCrosstalkSources_855451_56.stderr parsl.dataflow.dflow DEBUG: Task 3598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3598 has memoization hash 4d73a6c4e34f9140287c1ee41652ef11 parsl.dataflow.memoization INFO: Task 3598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3598 try 0 launched on executor multi with executor id 3599 parsl.dataflow.dflow INFO: Standard output for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d367339f-46e3-4cdb-95c9-ea5bf38bf46c_isrForCrosstalkSources_855432_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d367339f-46e3-4cdb-95c9-ea5bf38bf46c_isrForCrosstalkSources_855432_37.stderr parsl.dataflow.dflow DEBUG: Task 3599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3599 has memoization hash ea287947c7261157783b2879a4048dc6 parsl.dataflow.memoization INFO: Task 3599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3599 try 0 launched on executor multi with executor id 3600 parsl.dataflow.dflow INFO: Standard output for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c5ec8882-563a-442e-8750-dfe7c440b9e6_isrForCrosstalkSources_855441_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c5ec8882-563a-442e-8750-dfe7c440b9e6_isrForCrosstalkSources_855441_54.stderr parsl.dataflow.dflow DEBUG: Task 3600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3600 has memoization hash a15bc0a003993be204f9fc196b10cdfe parsl.dataflow.memoization INFO: Task 3600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3600 try 0 launched on executor multi with executor id 3601 parsl.dataflow.dflow INFO: Standard output for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9db97005-ecae-42c7-baf8-3e623ce2c52b_isrForCrosstalkSources_855406_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9db97005-ecae-42c7-baf8-3e623ce2c52b_isrForCrosstalkSources_855406_22.stderr parsl.dataflow.dflow DEBUG: Task 3601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3601 has memoization hash 802944041e834a515f4bb7b21c42826a parsl.dataflow.memoization INFO: Task 3601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3601 try 0 launched on executor multi with executor id 3602 parsl.dataflow.dflow INFO: Standard output for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/07ba5436-0aaf-4711-bda9-7ef9717d2302_isrForCrosstalkSources_855430_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/07ba5436-0aaf-4711-bda9-7ef9717d2302_isrForCrosstalkSources_855430_42.stderr parsl.dataflow.dflow DEBUG: Task 3602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3602 has memoization hash bef1ea68c2f799b3cbc769e2023cc63d parsl.dataflow.memoization INFO: Task 3602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3602 try 0 launched on executor multi with executor id 3603 parsl.dataflow.dflow INFO: Standard output for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c61c0b86-4ece-4ed4-bed0-5c7aa4f364bc_isrForCrosstalkSources_855403_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c61c0b86-4ece-4ed4-bed0-5c7aa4f364bc_isrForCrosstalkSources_855403_49.stderr parsl.dataflow.dflow DEBUG: Task 3603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3603 has memoization hash e277ecb6a4ea8fc98f0586b6bd2cd568 parsl.dataflow.memoization INFO: Task 3603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3603 try 0 launched on executor multi with executor id 3604 parsl.dataflow.dflow INFO: Standard output for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/c54ab6a1-cc2f-483c-8b3c-8f8b6273198c_isrForCrosstalkSources_855386_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/c54ab6a1-cc2f-483c-8b3c-8f8b6273198c_isrForCrosstalkSources_855386_17.stderr parsl.dataflow.dflow DEBUG: Task 3604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3604 has memoization hash 2eaaf6efa5feabe380bd983e8cad212f parsl.dataflow.memoization INFO: Task 3604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8eb0031d5a7030454dc900fc115571 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e64fffb0-e180-4ff4-8c2c-e04278ac885e_isrForCrosstalkSources_855403_17.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e64fffb0-e180-4ff4-8c2c-e04278ac885e_isrForCrosstalkSources_855403_17.stderr parsl.dataflow.dflow INFO: Parsl task 3604 try 0 launched on executor multi with executor id 3605 parsl.dataflow.dflow INFO: Standard output for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/d8c9831b-620b-4c42-8a21-2af2dfdf7961_isrForCrosstalkSources_855405_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/d8c9831b-620b-4c42-8a21-2af2dfdf7961_isrForCrosstalkSources_855405_5.stderr parsl.dataflow.dflow DEBUG: Task 3605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3605 has memoization hash 8fd3f675c654b701074b35162ab8d938 parsl.dataflow.memoization INFO: Task 3605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3605 try 0 launched on executor multi with executor id 3606 parsl.dataflow.dflow INFO: Standard output for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/29a856ae-d1d3-4412-8fd0-95c7eb513b0e_isrForCrosstalkSources_855439_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/29a856ae-d1d3-4412-8fd0-95c7eb513b0e_isrForCrosstalkSources_855439_42.stderr parsl.dataflow.dflow DEBUG: Task 3606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3606 has memoization hash fb2e81080cb2963e56a097af71084a16 parsl.dataflow.memoization INFO: Task 3606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3606 try 0 launched on executor multi with executor id 3607 parsl.dataflow.dflow INFO: Standard output for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2dd6bafe-8a81-4a50-a436-e3f31f3c0929_isrForCrosstalkSources_855410_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2dd6bafe-8a81-4a50-a436-e3f31f3c0929_isrForCrosstalkSources_855410_58.stderr parsl.dataflow.dflow DEBUG: Task 3607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3607 has memoization hash 236f93b602b290eceb42682490d8b586 parsl.dataflow.memoization INFO: Task 3607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3607 try 0 launched on executor multi with executor id 3608 parsl.dataflow.dflow INFO: Standard output for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/48447d67-adbd-4710-bb27-d99c2a6b232b_isrForCrosstalkSources_855398_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/48447d67-adbd-4710-bb27-d99c2a6b232b_isrForCrosstalkSources_855398_36.stderr parsl.dataflow.dflow DEBUG: Task 3608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3608 has memoization hash 6918a7c52f38cbc727e8ee07d74296e4 parsl.dataflow.memoization INFO: Task 3608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3608 try 0 launched on executor multi with executor id 3609 parsl.dataflow.dflow INFO: Standard output for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/803aacb8-5f3f-4dd4-bc42-f043ca576d09_isrForCrosstalkSources_855446_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/803aacb8-5f3f-4dd4-bc42-f043ca576d09_isrForCrosstalkSources_855446_32.stderr parsl.dataflow.dflow DEBUG: Task 3609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3609 has memoization hash f96c346c02d23a4643f22e7fd811ffdd parsl.dataflow.memoization INFO: Task 3609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3609 try 0 launched on executor multi with executor id 3610 parsl.dataflow.dflow INFO: Standard output for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d3e78d10-6db5-4cd8-a155-28484e26f4e1_isrForCrosstalkSources_855457_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d3e78d10-6db5-4cd8-a155-28484e26f4e1_isrForCrosstalkSources_855457_31.stderr parsl.dataflow.dflow DEBUG: Task 3610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3610 has memoization hash fad977d3240861e7b67316c20a69b9d1 parsl.dataflow.memoization INFO: Task 3610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3610 try 0 launched on executor multi with executor id 3611 parsl.dataflow.dflow INFO: Standard output for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/951ff6d7-ed44-4970-aa25-8a99e4269034_isrForCrosstalkSources_855451_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/951ff6d7-ed44-4970-aa25-8a99e4269034_isrForCrosstalkSources_855451_6.stderr parsl.dataflow.dflow DEBUG: Task 3611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3611 has memoization hash 208a52d5f0b42ddbb3e80a7d2939909c parsl.dataflow.memoization INFO: Task 3611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e77f8bb675fa1a1003c89d0604bea3b1 with result from task 356 parsl.dataflow.dflow INFO: Parsl task 3611 try 0 launched on executor multi with executor id 3612 parsl.dataflow.dflow INFO: Standard output for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1c0d908-a620-4f0d-9025-150302b8e34e_isrForCrosstalkSources_855412_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1c0d908-a620-4f0d-9025-150302b8e34e_isrForCrosstalkSources_855412_25.stderr parsl.dataflow.dflow DEBUG: Task 3612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3612 has memoization hash 06f8e9481093c5bb6ffd92dc325a44d4 parsl.dataflow.memoization INFO: Task 3612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:38 proc_lsst.multi:170 [INFO] [multi] job_id 16964171 proc_lsst.multi INFO: [multi] job_id 16964171 2024-03-09 14:51:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-09 14:51:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964171 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964171 parsl.executors.status_handling DEBUG: Launched block 145 on executor multi with job ID 16964171 parsl.executors.status_handling INFO: Allocated block ID 146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-146 2024-03-09 14:51:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 2024-03-09 14:51:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-09 14:51:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b7fef9d0-aa1c-492e-b1b7-f3d241633f3a_isrForCrosstalkSources_855433_40.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b7fef9d0-aa1c-492e-b1b7-f3d241633f3a_isrForCrosstalkSources_855433_40.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb6c150dadbd1efb281545cef132057 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/56b31609-dd26-4eb9-89ae-75b246fa1306_isrForCrosstalkSources_855440_23.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/56b31609-dd26-4eb9-89ae-75b246fa1306_isrForCrosstalkSources_855440_23.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6063bae3650e39fc89c8adcdcc9d5c6 with result from task 359 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bdf00d32-9c37-4465-8eb2-947ec874e591_isrForCrosstalkSources_855438_36.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bdf00d32-9c37-4465-8eb2-947ec874e591_isrForCrosstalkSources_855438_36.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef9e7e6f96a5d947ed23413b5f3b586d with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/af9b66f0-fd96-47aa-937f-8a239544cb65_isrForCrosstalkSources_855394_47.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/af9b66f0-fd96-47aa-937f-8a239544cb65_isrForCrosstalkSources_855394_47.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b8ca94a726d8d9fec60b0ee8d52bc69 with result from task 360 parsl.dataflow.dflow INFO: Parsl task 3612 try 0 launched on executor multi with executor id 3613 parsl.dataflow.dflow INFO: Standard output for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/756c58c9-8fe8-463b-83f2-4f48a14029fc_isrForCrosstalkSources_855456_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/756c58c9-8fe8-463b-83f2-4f48a14029fc_isrForCrosstalkSources_855456_43.stderr parsl.dataflow.dflow DEBUG: Task 3613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3613 has memoization hash 44b4f863821c2c88aa46ed9e681acad2 parsl.dataflow.memoization INFO: Task 3613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a63d72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/121921d7-d0e7-4ab2-b11f-2ba15ee1b354_isrForCrosstalkSources_855386_13.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/121921d7-d0e7-4ab2-b11f-2ba15ee1b354_isrForCrosstalkSources_855386_13.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f3f478a9680073fe93e81943826c332 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a2320da4-3305-48dd-90db-fd961d1341b2_isrForCrosstalkSources_855388_10.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a2320da4-3305-48dd-90db-fd961d1341b2_isrForCrosstalkSources_855388_10.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 844dfe77f40dd659307d26deed9e463d with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6f3bc0c5-5347-4c3b-8735-493f809a26e2_isrForCrosstalkSources_855424_40.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6f3bc0c5-5347-4c3b-8735-493f809a26e2_isrForCrosstalkSources_855424_40.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b908de86fb7cae4590396a70e2416e44 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d3a8d4f8-e2c3-4c39-a066-ec870b0826d7_isrForCrosstalkSources_855417_38.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d3a8d4f8-e2c3-4c39-a066-ec870b0826d7_isrForCrosstalkSources_855417_38.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2df2b91aea0f26254a5115ebde8480bc with result from task 330 parsl.dataflow.dflow INFO: Parsl task 3613 try 0 launched on executor multi with executor id 3614 parsl.dataflow.dflow INFO: Standard output for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/d97144e7-8dad-4bf4-8b62-8e53dc047c1f_isrForCrosstalkSources_855441_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/d97144e7-8dad-4bf4-8b62-8e53dc047c1f_isrForCrosstalkSources_855441_44.stderr parsl.dataflow.dflow DEBUG: Task 3614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3614 has memoization hash dc699752903dfb01254500535aa15afa parsl.dataflow.memoization INFO: Task 3614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2baaa614-7be9-4456-8a0c-1bf632f4e57c_isrForCrosstalkSources_855424_11.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2baaa614-7be9-4456-8a0c-1bf632f4e57c_isrForCrosstalkSources_855424_11.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3672507f4b1e6abdc503e0a67637ad9a with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dd311b05-0593-4c4c-b333-ee9d94d1a709_isrForCrosstalkSources_855432_47.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dd311b05-0593-4c4c-b333-ee9d94d1a709_isrForCrosstalkSources_855432_47.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry febe58d86de650a5dc013b570b9e3ff7 with result from task 388 parsl.dataflow.dflow INFO: Parsl task 3614 try 0 launched on executor multi with executor id 3615 parsl.dataflow.dflow INFO: Standard output for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/4127ef1f-9cab-4a13-a5b5-4c17e1138b4c_isrForCrosstalkSources_855422_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/4127ef1f-9cab-4a13-a5b5-4c17e1138b4c_isrForCrosstalkSources_855422_11.stderr parsl.dataflow.dflow DEBUG: Task 3615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3615 has memoization hash b741b9774a9316b9d990d040daf8340d parsl.dataflow.memoization INFO: Task 3615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d519d9d1-3c63-4809-a63a-bef6d32ac91c_isrForCrosstalkSources_855450_36.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d519d9d1-3c63-4809-a63a-bef6d32ac91c_isrForCrosstalkSources_855450_36.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc30f673a92203d27b3c9633d6e1ca0 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e70c7127-6e14-49fd-b03e-f1ea0f7e381f_isrForCrosstalkSources_855418_18.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e70c7127-6e14-49fd-b03e-f1ea0f7e381f_isrForCrosstalkSources_855418_18.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6bb9283dda0fa4da542818af94afd60 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcac0e1-be24-4934-b97d-cf532d4a3b24_isrForCrosstalkSources_855409_16.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcac0e1-be24-4934-b97d-cf532d4a3b24_isrForCrosstalkSources_855409_16.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d33957d6be096836f825b1eb06b0b65 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e1784fd8-e44f-4691-a5ac-d7a819c5d35b_isrForCrosstalkSources_855402_56.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e1784fd8-e44f-4691-a5ac-d7a819c5d35b_isrForCrosstalkSources_855402_56.stderr parsl.dataflow.dflow INFO: Parsl task 3615 try 0 launched on executor multi with executor id 3616 parsl.dataflow.dflow INFO: Standard output for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/94a0f5da-c10c-4885-8355-0684ae6217ac_isrForCrosstalkSources_855441_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/94a0f5da-c10c-4885-8355-0684ae6217ac_isrForCrosstalkSources_855441_17.stderr parsl.dataflow.dflow DEBUG: Task 3616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3616 has memoization hash af9762e2e3f02e2ed267a90f5fb1b431 parsl.dataflow.memoization INFO: Task 3616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3616 try 0 launched on executor multi with executor id 3617 parsl.dataflow.dflow INFO: Standard output for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/964117aa-3d34-4831-9b72-95f633bd9a50_isrForCrosstalkSources_855380_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/964117aa-3d34-4831-9b72-95f633bd9a50_isrForCrosstalkSources_855380_49.stderr parsl.dataflow.dflow DEBUG: Task 3617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3617 has memoization hash 3600a8495f65cf7b7c2c47d3b1ffe72a parsl.dataflow.memoization INFO: Task 3617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3617 try 0 launched on executor multi with executor id 3618 parsl.dataflow.dflow INFO: Standard output for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e2e0fedc-9f2c-4304-ab20-fa65b91343aa_isrForCrosstalkSources_855404_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e2e0fedc-9f2c-4304-ab20-fa65b91343aa_isrForCrosstalkSources_855404_1.stderr parsl.dataflow.dflow DEBUG: Task 3618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3618 has memoization hash e7bbdb2980b69515fa064a1e837d095a parsl.dataflow.memoization INFO: Task 3618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3618 try 0 launched on executor multi with executor id 3619 parsl.dataflow.dflow INFO: Standard output for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9513d3c0-7f29-40d6-9809-71f4cd085536_isrForCrosstalkSources_855398_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9513d3c0-7f29-40d6-9809-71f4cd085536_isrForCrosstalkSources_855398_15.stderr parsl.dataflow.dflow DEBUG: Task 3619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3619 has memoization hash d9797786dc18bee3511a338d35cf04e2 parsl.dataflow.memoization INFO: Task 3619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9856b9b4e9efe10805110d4dba7b9203 with result from task 515 parsl.dataflow.dflow INFO: Parsl task 3619 try 0 launched on executor multi with executor id 3620 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Standard output for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6481ce8d-a74d-456d-bc24-292b583eb08d_isrForCrosstalkSources_855385_34.stdout parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard error for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6481ce8d-a74d-456d-bc24-292b583eb08d_isrForCrosstalkSources_855385_34.stderr parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ddbe6799-1c3b-48e7-8e55-2e4d8c04453b_isrForCrosstalkSources_855381_13.stdout parsl.dataflow.dflow DEBUG: Task 3620 will be sent to executor multi parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ddbe6799-1c3b-48e7-8e55-2e4d8c04453b_isrForCrosstalkSources_855381_13.stderr parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3620 has memoization hash 32c9a19f8aa5e21f1b91c06f85cf287d parsl.dataflow.memoization INFO: Task 3620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa3852712cf4d86f8a18430579f97f79 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/91fd216a-f8f1-433e-8004-d0e5f3f16e0c_isrForCrosstalkSources_855449_11.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/91fd216a-f8f1-433e-8004-d0e5f3f16e0c_isrForCrosstalkSources_855449_11.stderr parsl.dataflow.dflow INFO: Parsl task 3620 try 0 launched on executor multi with executor id 3621 parsl.dataflow.dflow INFO: Standard output for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/361861f7-6723-46fd-b49b-225de6392422_isrForCrosstalkSources_855388_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/361861f7-6723-46fd-b49b-225de6392422_isrForCrosstalkSources_855388_6.stderr parsl.dataflow.dflow DEBUG: Task 3621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3621 has memoization hash f755bd5f3384e9afafe662715fa804dc parsl.dataflow.memoization INFO: Task 3621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3621 try 0 launched on executor multi with executor id 3622 parsl.dataflow.dflow INFO: Standard output for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/85c96fc0-b25e-42de-bac7-3512a7624511_isrForCrosstalkSources_855389_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/85c96fc0-b25e-42de-bac7-3512a7624511_isrForCrosstalkSources_855389_40.stderr parsl.dataflow.dflow DEBUG: Task 3622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3622 has memoization hash d16d028d1f88ef2ac6115fd08ed18c12 parsl.dataflow.memoization INFO: Task 3622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3622 try 0 launched on executor multi with executor id 3623 parsl.dataflow.dflow INFO: Standard output for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5bdf7576-c584-4bbd-81f1-c195b6eafe2f_isrForCrosstalkSources_855458_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5bdf7576-c584-4bbd-81f1-c195b6eafe2f_isrForCrosstalkSources_855458_2.stderr parsl.dataflow.dflow DEBUG: Task 3623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3623 has memoization hash aa4352be8382c982086bcc12e048868a parsl.dataflow.memoization INFO: Task 3623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3623 try 0 launched on executor multi with executor id 3624 parsl.dataflow.dflow INFO: Standard output for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a8455573-1f91-4524-a812-21d0da8532ea_isrForCrosstalkSources_855397_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a8455573-1f91-4524-a812-21d0da8532ea_isrForCrosstalkSources_855397_11.stderr parsl.dataflow.dflow DEBUG: Task 3624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3624 has memoization hash a452bf647bbfbd66743d92a710df0536 parsl.dataflow.memoization INFO: Task 3624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3624 try 0 launched on executor multi with executor id 3625 parsl.dataflow.dflow INFO: Standard output for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1cef0e58-784c-4197-bb53-3d169b704f77_isrForCrosstalkSources_855399_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1cef0e58-784c-4197-bb53-3d169b704f77_isrForCrosstalkSources_855399_8.stderr parsl.dataflow.dflow DEBUG: Task 3625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3625 has memoization hash 3f592a8fa2f1d0672e83f2a5882a4a66 parsl.dataflow.memoization INFO: Task 3625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3625 try 0 launched on executor multi with executor id 3626 parsl.dataflow.dflow INFO: Standard output for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4b7e71-b2ec-4ff6-82ae-ec6c0b1c5c7b_isrForCrosstalkSources_855393_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4b7e71-b2ec-4ff6-82ae-ec6c0b1c5c7b_isrForCrosstalkSources_855393_32.stderr parsl.dataflow.dflow DEBUG: Task 3626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3626 has memoization hash b1e4567d430deb2314208ccad8cf786b parsl.dataflow.memoization INFO: Task 3626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3626 try 0 launched on executor multi with executor id 3627 parsl.dataflow.dflow INFO: Standard output for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a960ec8a-05a6-48d0-8c7d-3c80806f95a2_isrForCrosstalkSources_855422_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a960ec8a-05a6-48d0-8c7d-3c80806f95a2_isrForCrosstalkSources_855422_25.stderr parsl.dataflow.dflow DEBUG: Task 3627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3627 has memoization hash 912792969aefb417da22a7c4b32697dc parsl.dataflow.memoization INFO: Task 3627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3627 try 0 launched on executor multi with executor id 3628 parsl.dataflow.dflow INFO: Standard output for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1440ae3e-1b79-4a1d-83ee-3c5398968b62_isrForCrosstalkSources_855442_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1440ae3e-1b79-4a1d-83ee-3c5398968b62_isrForCrosstalkSources_855442_7.stderr parsl.dataflow.dflow DEBUG: Task 3628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3628 has memoization hash bd4d396e6fed55ea6b4ace4e42ebb230 parsl.dataflow.memoization INFO: Task 3628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3628 try 0 launched on executor multi with executor id 3629 parsl.dataflow.dflow INFO: Standard output for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f96a2162-9319-4e8f-ad6e-58435e03679f_isrForCrosstalkSources_855396_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f96a2162-9319-4e8f-ad6e-58435e03679f_isrForCrosstalkSources_855396_53.stderr parsl.dataflow.dflow DEBUG: Task 3629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3629 has memoization hash e108675d90f005934ad672d167356c4f parsl.dataflow.memoization INFO: Task 3629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3629 try 0 launched on executor multi with executor id 3630 parsl.dataflow.dflow INFO: Standard output for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fcd156-11b8-439a-9a66-9482c859bf71_isrForCrosstalkSources_855420_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fcd156-11b8-439a-9a66-9482c859bf71_isrForCrosstalkSources_855420_17.stderr parsl.dataflow.dflow DEBUG: Task 3630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3630 has memoization hash 7df1e4d7967ea05caf81bf7cf113d6b9 parsl.dataflow.memoization INFO: Task 3630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3630 try 0 launched on executor multi with executor id 3631 parsl.dataflow.dflow INFO: Standard output for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c8e393dc-da15-4b63-a472-287586098114_isrForCrosstalkSources_855418_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c8e393dc-da15-4b63-a472-287586098114_isrForCrosstalkSources_855418_38.stderr parsl.dataflow.dflow DEBUG: Task 3631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3631 has memoization hash c224cdfb51333b99cc592afbf6ab6768 parsl.dataflow.memoization INFO: Task 3631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3631 try 0 launched on executor multi with executor id 3632 parsl.dataflow.dflow INFO: Standard output for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/51c4dca3-02b4-4162-8080-65655e44d13c_isrForCrosstalkSources_855451_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/51c4dca3-02b4-4162-8080-65655e44d13c_isrForCrosstalkSources_855451_49.stderr parsl.dataflow.dflow DEBUG: Task 3632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3632 has memoization hash 9a45ad3e53784184add40a48d564be67 parsl.dataflow.memoization INFO: Task 3632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3632 try 0 launched on executor multi with executor id 3633 parsl.dataflow.dflow INFO: Standard output for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e508acd9-9f1b-4a00-8699-b9e218300c28_isrForCrosstalkSources_855420_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e508acd9-9f1b-4a00-8699-b9e218300c28_isrForCrosstalkSources_855420_55.stderr parsl.dataflow.dflow DEBUG: Task 3633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3633 has memoization hash fdd2ccbb6d802e1ff95e5a73ed155ff1 parsl.dataflow.memoization INFO: Task 3633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3633 try 0 launched on executor multi with executor id 3634 parsl.dataflow.dflow INFO: Standard output for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/84d3b206-b3fe-4a27-84d7-011ed74d0b07_isrForCrosstalkSources_855395_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/84d3b206-b3fe-4a27-84d7-011ed74d0b07_isrForCrosstalkSources_855395_48.stderr parsl.dataflow.dflow DEBUG: Task 3634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3634 has memoization hash a451bfd3dd56b443a95212d53e952384 parsl.dataflow.memoization INFO: Task 3634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3634 try 0 launched on executor multi with executor id 3635 parsl.dataflow.dflow INFO: Standard output for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d4fd2b5-de8e-4a34-83ef-536ef5bd0aab_isrForCrosstalkSources_855444_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d4fd2b5-de8e-4a34-83ef-536ef5bd0aab_isrForCrosstalkSources_855444_34.stderr parsl.dataflow.dflow DEBUG: Task 3635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3635 has memoization hash 77d14e30431d3c7f4bee7932b6534bdc parsl.dataflow.memoization INFO: Task 3635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3635 try 0 launched on executor multi with executor id 3636 parsl.dataflow.dflow INFO: Standard output for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cbf5e53b-33a7-4807-be5b-73c7c79a1c0b_isrForCrosstalkSources_855441_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cbf5e53b-33a7-4807-be5b-73c7c79a1c0b_isrForCrosstalkSources_855441_45.stderr parsl.dataflow.dflow DEBUG: Task 3636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3636 has memoization hash dfd5165d0b3e48f8fbb3d7167f54968c parsl.dataflow.memoization INFO: Task 3636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3636 try 0 launched on executor multi with executor id 3637 parsl.dataflow.dflow INFO: Standard output for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3b7f65dc-0e2a-46ab-af3e-04540511270f_isrForCrosstalkSources_855440_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3b7f65dc-0e2a-46ab-af3e-04540511270f_isrForCrosstalkSources_855440_47.stderr parsl.dataflow.dflow DEBUG: Task 3637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3637 has memoization hash 7bdc9391774816dbd43c5ef197caedf2 parsl.dataflow.memoization INFO: Task 3637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3637 try 0 launched on executor multi with executor id 3638 parsl.dataflow.dflow INFO: Standard output for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c7fe6c11-d608-49dd-85e5-bbed650e8d79_isrForCrosstalkSources_855402_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c7fe6c11-d608-49dd-85e5-bbed650e8d79_isrForCrosstalkSources_855402_55.stderr parsl.dataflow.dflow DEBUG: Task 3638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3638 has memoization hash 615a76989bcc72ea2c39c8897e8a48c6 parsl.dataflow.memoization INFO: Task 3638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3638 try 0 launched on executor multi with executor id 3639 parsl.dataflow.dflow INFO: Standard output for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c97f978a-0409-4831-a8bb-9bf3e6601adb_isrForCrosstalkSources_855452_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c97f978a-0409-4831-a8bb-9bf3e6601adb_isrForCrosstalkSources_855452_31.stderr parsl.dataflow.dflow DEBUG: Task 3639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3639 has memoization hash 7dbeae94f86e5b84f931c00f0fa93307 parsl.dataflow.memoization INFO: Task 3639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3639 try 0 launched on executor multi with executor id 3640 parsl.dataflow.dflow INFO: Standard output for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/624008d9-0a44-4c38-ac20-b12b4ff6fac2_isrForCrosstalkSources_855380_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/624008d9-0a44-4c38-ac20-b12b4ff6fac2_isrForCrosstalkSources_855380_61.stderr parsl.dataflow.dflow DEBUG: Task 3640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3640 has memoization hash e7449d6744c00678a5789a10a31af94f parsl.dataflow.memoization INFO: Task 3640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3640 try 0 launched on executor multi with executor id 3641 parsl.dataflow.dflow INFO: Standard output for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6a1a403-6f43-40f1-9c7e-cbedb4afd4d5_isrForCrosstalkSources_855380_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6a1a403-6f43-40f1-9c7e-cbedb4afd4d5_isrForCrosstalkSources_855380_25.stderr parsl.dataflow.dflow DEBUG: Task 3641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3641 has memoization hash e022231f3f1938cf2cf1dcfabf390107 parsl.dataflow.memoization INFO: Task 3641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3641 try 0 launched on executor multi with executor id 3642 parsl.dataflow.dflow INFO: Standard output for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8d7879fd-d442-4fa7-9f5b-7308057a5f54_isrForCrosstalkSources_855379_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8d7879fd-d442-4fa7-9f5b-7308057a5f54_isrForCrosstalkSources_855379_47.stderr parsl.dataflow.dflow DEBUG: Task 3642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3642 has memoization hash 938bd85f2f8970092d18b85b112e4362 parsl.dataflow.memoization INFO: Task 3642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3642 try 0 launched on executor multi with executor id 3643 parsl.dataflow.dflow INFO: Standard output for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7f7bf617-f0c3-4cf2-9d99-d6716fd37507_isrForCrosstalkSources_855406_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7f7bf617-f0c3-4cf2-9d99-d6716fd37507_isrForCrosstalkSources_855406_27.stderr parsl.dataflow.dflow DEBUG: Task 3643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3643 has memoization hash 04822c43e6fed03775da5a7dd30e86ab parsl.dataflow.memoization INFO: Task 3643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3643 try 0 launched on executor multi with executor id 3644 parsl.dataflow.dflow INFO: Standard output for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b8dfe603-054a-4ed5-84d2-7fb0f463e5e9_isrForCrosstalkSources_855422_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b8dfe603-054a-4ed5-84d2-7fb0f463e5e9_isrForCrosstalkSources_855422_56.stderr parsl.dataflow.dflow DEBUG: Task 3644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3644 has memoization hash b0c660db504a12acd9b19aca64207e77 parsl.dataflow.memoization INFO: Task 3644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3644 try 0 launched on executor multi with executor id 3645 parsl.dataflow.dflow INFO: Standard output for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0e4ac518-dbd7-48a7-8729-8601cc121784_isrForCrosstalkSources_855452_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0e4ac518-dbd7-48a7-8729-8601cc121784_isrForCrosstalkSources_855452_20.stderr parsl.dataflow.dflow DEBUG: Task 3645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3645 has memoization hash bd744de4b62302e02ebdf1cec156ada9 parsl.dataflow.memoization INFO: Task 3645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3645 try 0 launched on executor multi with executor id 3646 parsl.dataflow.dflow INFO: Standard output for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/18e5b8c2-5089-434f-8c7a-ea97415ec242_isrForCrosstalkSources_855459_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/18e5b8c2-5089-434f-8c7a-ea97415ec242_isrForCrosstalkSources_855459_15.stderr parsl.dataflow.dflow DEBUG: Task 3646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3646 has memoization hash 6b9af8d4d706879d7f3aa0df82303f4f parsl.dataflow.memoization INFO: Task 3646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5fb5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3646 try 0 launched on executor multi with executor id 3647 parsl.dataflow.dflow INFO: Standard output for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/8bff1e85-8a3e-4843-9f6f-8f7a436d1a84_isrForCrosstalkSources_855455_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/8bff1e85-8a3e-4843-9f6f-8f7a436d1a84_isrForCrosstalkSources_855455_17.stderr parsl.dataflow.dflow DEBUG: Task 3647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3647 has memoization hash ae7de885fb4002041f75b7d6f00e6adb parsl.dataflow.memoization INFO: Task 3647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3647 try 0 launched on executor multi with executor id 3648 parsl.dataflow.dflow INFO: Standard output for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f2ca3ed1-55b0-4bb2-8111-11ab11d27ef9_isrForCrosstalkSources_855400_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f2ca3ed1-55b0-4bb2-8111-11ab11d27ef9_isrForCrosstalkSources_855400_14.stderr parsl.dataflow.dflow DEBUG: Task 3648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3648 has memoization hash 4ea53f8fca935530e24df03bde79b34e parsl.dataflow.memoization INFO: Task 3648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3648 try 0 launched on executor multi with executor id 3649 parsl.dataflow.dflow INFO: Standard output for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a593f637-45aa-468a-8db6-1363e4dea858_isrForCrosstalkSources_855438_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a593f637-45aa-468a-8db6-1363e4dea858_isrForCrosstalkSources_855438_2.stderr parsl.dataflow.dflow DEBUG: Task 3649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3649 has memoization hash fbc16c87b619dd9006e1b817bf96eb98 parsl.dataflow.memoization INFO: Task 3649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3649 try 0 launched on executor multi with executor id 3650 parsl.dataflow.dflow INFO: Standard output for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b9857c0-c115-4896-a496-6ddc531b9808_isrForCrosstalkSources_855459_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b9857c0-c115-4896-a496-6ddc531b9808_isrForCrosstalkSources_855459_53.stderr parsl.dataflow.dflow DEBUG: Task 3650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3650 has memoization hash cf23c1bd1263327ac35662204d808c3b parsl.dataflow.memoization INFO: Task 3650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3650 try 0 launched on executor multi with executor id 3651 parsl.dataflow.dflow INFO: Standard output for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a6c2e588-65ba-429a-aa72-8324be19c7da_isrForCrosstalkSources_855391_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a6c2e588-65ba-429a-aa72-8324be19c7da_isrForCrosstalkSources_855391_48.stderr parsl.dataflow.dflow DEBUG: Task 3651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3651 has memoization hash 955096d4d7b6583bb7e0256f86cbc6d8 parsl.dataflow.memoization INFO: Task 3651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3651 try 0 launched on executor multi with executor id 3652 parsl.dataflow.dflow INFO: Standard output for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de14dc26-0d83-4764-9a4a-9c6b7bbaa183_isrForCrosstalkSources_855449_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de14dc26-0d83-4764-9a4a-9c6b7bbaa183_isrForCrosstalkSources_855449_31.stderr parsl.dataflow.dflow DEBUG: Task 3652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3652 has memoization hash 8c8e7bee939676d17bb67addfb4d8b59 parsl.dataflow.memoization INFO: Task 3652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3652 try 0 launched on executor multi with executor id 3653 parsl.dataflow.dflow INFO: Standard output for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a2e3da9f-ca02-49d1-ae0b-dc94973b8e29_isrForCrosstalkSources_855383_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a2e3da9f-ca02-49d1-ae0b-dc94973b8e29_isrForCrosstalkSources_855383_29.stderr parsl.dataflow.dflow DEBUG: Task 3653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3653 has memoization hash 3ea1b0df1e4de73b84c7f0df62049bd2 parsl.dataflow.memoization INFO: Task 3653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3653 try 0 launched on executor multi with executor id 3654 parsl.dataflow.dflow INFO: Standard output for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/287b80a6-0a20-43c1-9574-2df5fa453ecd_isrForCrosstalkSources_855423_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/287b80a6-0a20-43c1-9574-2df5fa453ecd_isrForCrosstalkSources_855423_53.stderr parsl.dataflow.dflow DEBUG: Task 3654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3654 has memoization hash 7ede7c766173553d562fa2494aef3769 parsl.dataflow.memoization INFO: Task 3654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3654 try 0 launched on executor multi with executor id 3655 parsl.dataflow.dflow INFO: Standard output for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3c0eb17a-e61b-4502-a15d-aa553523b4c0_isrForCrosstalkSources_855414_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3c0eb17a-e61b-4502-a15d-aa553523b4c0_isrForCrosstalkSources_855414_37.stderr parsl.dataflow.dflow DEBUG: Task 3655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3655 has memoization hash a2bd7a90549d7f08fde66ee71a72a11b parsl.dataflow.memoization INFO: Task 3655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3655 try 0 launched on executor multi with executor id 3656 parsl.dataflow.dflow INFO: Standard output for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/31d3ad1a-92ce-4006-bd57-3b1fb3f9dcaf_isrForCrosstalkSources_855437_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/31d3ad1a-92ce-4006-bd57-3b1fb3f9dcaf_isrForCrosstalkSources_855437_4.stderr parsl.dataflow.dflow DEBUG: Task 3656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3656 has memoization hash c3fa7a7b794472f536d1e144961e4a5e parsl.dataflow.memoization INFO: Task 3656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3656 try 0 launched on executor multi with executor id 3657 parsl.dataflow.dflow INFO: Standard output for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5b43d080-841d-4deb-bb4f-9ea7ddd8ed7e_isrForCrosstalkSources_855381_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5b43d080-841d-4deb-bb4f-9ea7ddd8ed7e_isrForCrosstalkSources_855381_15.stderr parsl.dataflow.dflow DEBUG: Task 3657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3657 has memoization hash c6171cdedae81e629649faa91b5475f6 parsl.dataflow.memoization INFO: Task 3657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3657 try 0 launched on executor multi with executor id 3658 parsl.dataflow.dflow INFO: Standard output for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9bec570f-aa8d-4332-b226-ed7c4208c339_isrForCrosstalkSources_855441_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9bec570f-aa8d-4332-b226-ed7c4208c339_isrForCrosstalkSources_855441_15.stderr parsl.dataflow.dflow DEBUG: Task 3658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3658 has memoization hash 5851a016220bb78daa373ec75e32bf67 parsl.dataflow.memoization INFO: Task 3658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3658 try 0 launched on executor multi with executor id 3659 parsl.dataflow.dflow INFO: Standard output for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9e7e049b-5ab0-4aee-a353-879b8874055e_isrForCrosstalkSources_855401_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9e7e049b-5ab0-4aee-a353-879b8874055e_isrForCrosstalkSources_855401_52.stderr parsl.dataflow.dflow DEBUG: Task 3659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3659 has memoization hash f438786142b857467189c2e8bb34ae4b parsl.dataflow.memoization INFO: Task 3659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3659 try 0 launched on executor multi with executor id 3660 parsl.dataflow.dflow INFO: Standard output for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/76e6d36b-3571-4a31-a572-99a87ea60b51_isrForCrosstalkSources_855440_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/76e6d36b-3571-4a31-a572-99a87ea60b51_isrForCrosstalkSources_855440_33.stderr parsl.dataflow.dflow DEBUG: Task 3660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3660 has memoization hash aa5f32f2176e7e7b1ac6e392cc9face9 parsl.dataflow.memoization INFO: Task 3660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3660 try 0 launched on executor multi with executor id 3661 parsl.dataflow.dflow INFO: Standard output for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/52e6869f-45a3-4b32-b15e-4db63f7d0353_isrForCrosstalkSources_855384_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/52e6869f-45a3-4b32-b15e-4db63f7d0353_isrForCrosstalkSources_855384_38.stderr parsl.dataflow.dflow DEBUG: Task 3661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3661 has memoization hash b96e040fddf015e0d30dbf2dbfabda0b parsl.dataflow.memoization INFO: Task 3661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3661 try 0 launched on executor multi with executor id 3662 parsl.dataflow.dflow INFO: Standard output for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6ecde8cb-b10e-46b4-b0ae-83e35798a744_isrForCrosstalkSources_855394_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6ecde8cb-b10e-46b4-b0ae-83e35798a744_isrForCrosstalkSources_855394_50.stderr parsl.dataflow.dflow DEBUG: Task 3662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3662 has memoization hash 809b252a0ae820aa768f82029f7126d6 parsl.dataflow.memoization INFO: Task 3662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3662 try 0 launched on executor multi with executor id 3663 parsl.dataflow.dflow INFO: Standard output for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bf2be93a-7038-4a03-9597-ae2e7d558cc0_isrForCrosstalkSources_855434_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bf2be93a-7038-4a03-9597-ae2e7d558cc0_isrForCrosstalkSources_855434_3.stderr parsl.dataflow.dflow DEBUG: Task 3663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3663 has memoization hash eb26f3867b79c4bd0ac4495002d4459a parsl.dataflow.memoization INFO: Task 3663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3663 try 0 launched on executor multi with executor id 3664 parsl.dataflow.dflow INFO: Standard output for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bc803bc7-51f8-4cfd-b956-03d6277874f3_isrForCrosstalkSources_855443_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bc803bc7-51f8-4cfd-b956-03d6277874f3_isrForCrosstalkSources_855443_7.stderr parsl.dataflow.dflow DEBUG: Task 3664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3664 has memoization hash daed0e3236e7bdf790db27ac7ab19e47 parsl.dataflow.memoization INFO: Task 3664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3664 try 0 launched on executor multi with executor id 3665 parsl.dataflow.dflow INFO: Standard output for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fe2f294e-7455-421d-a06e-422cb47ecbea_isrForCrosstalkSources_855420_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fe2f294e-7455-421d-a06e-422cb47ecbea_isrForCrosstalkSources_855420_52.stderr parsl.dataflow.dflow DEBUG: Task 3665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3665 has memoization hash 9714be20e5c68587687cdbf9f5955b6b parsl.dataflow.memoization INFO: Task 3665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3665 try 0 launched on executor multi with executor id 3666 parsl.dataflow.dflow INFO: Standard output for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/100a21d6-8233-43e4-8bd5-4178022cbdee_isrForCrosstalkSources_855427_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/100a21d6-8233-43e4-8bd5-4178022cbdee_isrForCrosstalkSources_855427_4.stderr parsl.dataflow.dflow DEBUG: Task 3666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3666 has memoization hash 120d044cc95f10458842e9f17f70f514 parsl.dataflow.memoization INFO: Task 3666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3666 try 0 launched on executor multi with executor id 3667 parsl.dataflow.dflow INFO: Standard output for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9f013ba4-f0d0-4f4e-97fb-d7a58bb9627a_isrForCrosstalkSources_855389_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9f013ba4-f0d0-4f4e-97fb-d7a58bb9627a_isrForCrosstalkSources_855389_43.stderr parsl.dataflow.dflow DEBUG: Task 3667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3667 has memoization hash 2c84cad47cdd572036ae0f094bfc6cbf parsl.dataflow.memoization INFO: Task 3667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3667 try 0 launched on executor multi with executor id 3668 parsl.dataflow.dflow INFO: Standard output for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/03bdc9fe-3513-4f76-9a59-2b2081246165_isrForCrosstalkSources_855437_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/03bdc9fe-3513-4f76-9a59-2b2081246165_isrForCrosstalkSources_855437_23.stderr parsl.dataflow.dflow DEBUG: Task 3668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3668 has memoization hash c7158d9a62b2c1e6d0a03c9b94dffbd0 parsl.dataflow.memoization INFO: Task 3668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ade03d73bd3295dead53a5995e400b8 with result from task 161 parsl.dataflow.dflow INFO: Parsl task 3668 try 0 launched on executor multi with executor id 3669 parsl.dataflow.dflow INFO: Standard output for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/47bbc988-fa0e-4c5e-9c2b-c82c2b3efa15_isrForCrosstalkSources_855422_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/47bbc988-fa0e-4c5e-9c2b-c82c2b3efa15_isrForCrosstalkSources_855422_49.stderr parsl.dataflow.dflow DEBUG: Task 3669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3669 has memoization hash 4b82e7c5473eca909a6c0ef1f2a01e60 parsl.dataflow.memoization INFO: Task 3669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3669 try 0 launched on executor multi with executor id 3670 parsl.dataflow.dflow INFO: Standard output for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/db281a04-69c2-4d5c-b52e-18933cd1f200_isrForCrosstalkSources_855439_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/db281a04-69c2-4d5c-b52e-18933cd1f200_isrForCrosstalkSources_855439_44.stderr parsl.dataflow.dflow DEBUG: Task 3670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3670 has memoization hash 2d49b47fffa0bd3ee0db4c691c1dd0d3 parsl.dataflow.memoization INFO: Task 3670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3670 try 0 launched on executor multi with executor id 3671 parsl.dataflow.dflow INFO: Standard output for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/02809a4b-1515-4766-9312-5508a9d521e9_isrForCrosstalkSources_855455_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/02809a4b-1515-4766-9312-5508a9d521e9_isrForCrosstalkSources_855455_31.stderr parsl.dataflow.dflow DEBUG: Task 3671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3671 has memoization hash 7d48ced4520c538053d66001bf0e9b7f parsl.dataflow.memoization INFO: Task 3671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3671 try 0 launched on executor multi with executor id 3672 parsl.dataflow.dflow INFO: Standard output for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/91b2ce67-441a-487a-ba59-070dc9e29924_isrForCrosstalkSources_855393_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/91b2ce67-441a-487a-ba59-070dc9e29924_isrForCrosstalkSources_855393_48.stderr parsl.dataflow.dflow DEBUG: Task 3672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3672 has memoization hash 51e37653e93a857f098e11719a78e0cd parsl.dataflow.memoization INFO: Task 3672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3672 try 0 launched on executor multi with executor id 3673 parsl.dataflow.dflow INFO: Standard output for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b80c660e-2db0-4f9e-b6cb-55162e5ddfff_isrForCrosstalkSources_855454_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b80c660e-2db0-4f9e-b6cb-55162e5ddfff_isrForCrosstalkSources_855454_35.stderr parsl.dataflow.dflow DEBUG: Task 3673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3673 has memoization hash d863b42bcb0408b39bf6f1c256c8dc72 parsl.dataflow.memoization INFO: Task 3673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fb262b4e-e3bd-4d6f-b12c-aba6f74d9855_isrForCrosstalkSources_855449_3.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fb262b4e-e3bd-4d6f-b12c-aba6f74d9855_isrForCrosstalkSources_855449_3.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d05ffd50accb7f144208cc974f74584 with result from task 500 parsl.dataflow.dflow INFO: Parsl task 3673 try 0 launched on executor multi with executor id 3674 parsl.dataflow.dflow INFO: Standard output for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f4c1e969-338d-42ac-ab38-64ef5a6e4796_isrForCrosstalkSources_855404_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f4c1e969-338d-42ac-ab38-64ef5a6e4796_isrForCrosstalkSources_855404_32.stderr parsl.dataflow.dflow DEBUG: Task 3674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3674 has memoization hash fa31053d8850f9df8676a0e75aebe34f parsl.dataflow.memoization INFO: Task 3674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3674 try 0 launched on executor multi with executor id 3675 parsl.dataflow.dflow INFO: Standard output for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2d25e4bb-2b3b-46dc-80f7-bc1f7b2aa659_isrForCrosstalkSources_855395_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2d25e4bb-2b3b-46dc-80f7-bc1f7b2aa659_isrForCrosstalkSources_855395_30.stderr parsl.dataflow.dflow DEBUG: Task 3675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3675 has memoization hash 792935a6f32144ca3653cb99d196273f parsl.dataflow.memoization INFO: Task 3675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/25da4179-1bc6-4d58-9adb-0cf0faac2163_isrForCrosstalkSources_855399_12.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/25da4179-1bc6-4d58-9adb-0cf0faac2163_isrForCrosstalkSources_855399_12.stderr parsl.dataflow.dflow INFO: Parsl task 3675 try 0 launched on executor multi with executor id 3676 parsl.dataflow.dflow INFO: Standard output for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/e7950baf-0b3c-413c-b50c-872095305331_isrForCrosstalkSources_855387_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/e7950baf-0b3c-413c-b50c-872095305331_isrForCrosstalkSources_855387_21.stderr parsl.dataflow.dflow DEBUG: Task 3676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3676 has memoization hash 25304cbe876261fb6fb3ac263a95fa43 parsl.dataflow.memoization INFO: Task 3676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3676 try 0 launched on executor multi with executor id 3677 parsl.dataflow.dflow INFO: Standard output for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3ed9d75d-9f36-4b53-a46a-af579e2902a5_isrForCrosstalkSources_855414_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3ed9d75d-9f36-4b53-a46a-af579e2902a5_isrForCrosstalkSources_855414_7.stderr parsl.dataflow.dflow DEBUG: Task 3677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3677 has memoization hash e5e312cfcc745e64b1f3bdbb2accbbc6 parsl.dataflow.memoization INFO: Task 3677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3677 try 0 launched on executor multi with executor id 3678 parsl.dataflow.dflow INFO: Standard output for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/affe1d0e-7d37-48fd-8ba2-df395102153e_isrForCrosstalkSources_855408_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/affe1d0e-7d37-48fd-8ba2-df395102153e_isrForCrosstalkSources_855408_23.stderr parsl.dataflow.dflow DEBUG: Task 3678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3678 has memoization hash 23bec19bef123b74a3c7762052f525ea parsl.dataflow.memoization INFO: Task 3678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3678 try 0 launched on executor multi with executor id 3679 parsl.dataflow.dflow INFO: Standard output for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/111b17ea-ca2b-401b-809a-d30dafcb5c89_isrForCrosstalkSources_855442_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/111b17ea-ca2b-401b-809a-d30dafcb5c89_isrForCrosstalkSources_855442_30.stderr parsl.dataflow.dflow DEBUG: Task 3679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3679 has memoization hash 27c06e11b71f747e356c2c223498bea1 parsl.dataflow.memoization INFO: Task 3679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3679 try 0 launched on executor multi with executor id 3680 parsl.dataflow.dflow INFO: Standard output for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d7ecc438-1bbb-4ff2-9869-3b96ed3cdda2_isrForCrosstalkSources_855430_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d7ecc438-1bbb-4ff2-9869-3b96ed3cdda2_isrForCrosstalkSources_855430_13.stderr parsl.dataflow.dflow DEBUG: Task 3680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3680 has memoization hash 2eed0d4777dbec5fb1898ec6e31bf094 parsl.dataflow.memoization INFO: Task 3680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3680 try 0 launched on executor multi with executor id 3681 parsl.dataflow.dflow INFO: Standard output for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fd26399e-a084-4376-97b0-38b019b24183_isrForCrosstalkSources_855423_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fd26399e-a084-4376-97b0-38b019b24183_isrForCrosstalkSources_855423_61.stderr parsl.dataflow.dflow DEBUG: Task 3681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3681 has memoization hash 1d8fb85c6be12d05478137cb430a91fb parsl.dataflow.memoization INFO: Task 3681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3681 try 0 launched on executor multi with executor id 3682 parsl.dataflow.dflow INFO: Standard output for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0fc3e8b7-67cb-4b67-8ac0-1af7a4198248_isrForCrosstalkSources_855384_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0fc3e8b7-67cb-4b67-8ac0-1af7a4198248_isrForCrosstalkSources_855384_47.stderr parsl.dataflow.dflow DEBUG: Task 3682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3682 has memoization hash 13e84fa4c3bcd6f372e9ac1cc9c21f56 parsl.dataflow.memoization INFO: Task 3682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3682 try 0 launched on executor multi with executor id 3683 parsl.dataflow.dflow INFO: Standard output for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/b88a13ad-47a9-42ba-a0f2-80751f47c78d_isrForCrosstalkSources_855409_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/b88a13ad-47a9-42ba-a0f2-80751f47c78d_isrForCrosstalkSources_855409_8.stderr parsl.dataflow.dflow DEBUG: Task 3683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3683 has memoization hash 92f5c3a656c4c3c803474316d178613e parsl.dataflow.memoization INFO: Task 3683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3683 try 0 launched on executor multi with executor id 3684 parsl.dataflow.dflow INFO: Standard output for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/fcac4def-d984-4b5a-8b8e-dd23aa33027f_isrForCrosstalkSources_855427_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/fcac4def-d984-4b5a-8b8e-dd23aa33027f_isrForCrosstalkSources_855427_21.stderr parsl.dataflow.dflow DEBUG: Task 3684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3684 has memoization hash 89254b701d80b51aa616bda927249103 parsl.dataflow.memoization INFO: Task 3684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3684 try 0 launched on executor multi with executor id 3685 parsl.dataflow.dflow INFO: Standard output for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6b0fb617-d215-4ebb-9f78-9637d0d1679f_isrForCrosstalkSources_855456_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6b0fb617-d215-4ebb-9f78-9637d0d1679f_isrForCrosstalkSources_855456_23.stderr parsl.dataflow.dflow DEBUG: Task 3685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3685 has memoization hash 43a8441356bd13f5c97377fe756e52fc parsl.dataflow.memoization INFO: Task 3685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3685 try 0 launched on executor multi with executor id 3686 parsl.dataflow.dflow INFO: Standard output for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/809d1d59-be25-4cd3-a4bd-b136451b76d9_isrForCrosstalkSources_855418_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/809d1d59-be25-4cd3-a4bd-b136451b76d9_isrForCrosstalkSources_855418_61.stderr parsl.dataflow.dflow DEBUG: Task 3686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3686 has memoization hash e0fc28f63d2e5efd8a73c2c87b18d181 parsl.dataflow.memoization INFO: Task 3686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3686 try 0 launched on executor multi with executor id 3687 parsl.dataflow.dflow INFO: Standard output for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/67291f22-13fc-42f8-afdf-1e423eb845cd_isrForCrosstalkSources_855456_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/67291f22-13fc-42f8-afdf-1e423eb845cd_isrForCrosstalkSources_855456_39.stderr parsl.dataflow.dflow DEBUG: Task 3687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3687 has memoization hash 193c4d7b8c97b211f6ddf521dc2879f2 parsl.dataflow.memoization INFO: Task 3687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3687 try 0 launched on executor multi with executor id 3688 parsl.dataflow.dflow INFO: Standard output for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1fc7e0e1-8ec7-4559-bb59-0dd3dd1432d8_isrForCrosstalkSources_855393_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1fc7e0e1-8ec7-4559-bb59-0dd3dd1432d8_isrForCrosstalkSources_855393_39.stderr parsl.dataflow.dflow DEBUG: Task 3688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3688 has memoization hash 834d7a2c8971f3db807b005f6b542476 parsl.dataflow.memoization INFO: Task 3688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3688 try 0 launched on executor multi with executor id 3689 parsl.dataflow.dflow INFO: Standard output for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ba7535b-d66f-4604-8883-3a40613fad69_isrForCrosstalkSources_855437_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ba7535b-d66f-4604-8883-3a40613fad69_isrForCrosstalkSources_855437_24.stderr parsl.dataflow.dflow DEBUG: Task 3689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3689 has memoization hash 044529d537f36712c4582133f5cbf6f6 parsl.dataflow.memoization INFO: Task 3689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3689 try 0 launched on executor multi with executor id 3690 parsl.dataflow.dflow INFO: Standard output for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d09a11e2-7715-4992-953b-9cfe38f890f8_isrForCrosstalkSources_855432_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d09a11e2-7715-4992-953b-9cfe38f890f8_isrForCrosstalkSources_855432_6.stderr parsl.dataflow.dflow DEBUG: Task 3690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3690 has memoization hash 49c9085075d49645ed10559e485287b4 parsl.dataflow.memoization INFO: Task 3690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3690 try 0 launched on executor multi with executor id 3691 parsl.dataflow.dflow INFO: Standard output for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/cac471ea-97b7-4dd4-9c0c-ce32958feeec_isrForCrosstalkSources_855459_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/cac471ea-97b7-4dd4-9c0c-ce32958feeec_isrForCrosstalkSources_855459_5.stderr parsl.dataflow.dflow DEBUG: Task 3691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3691 has memoization hash 8d5b0b2e9a2c92d15097f7590d72a83f parsl.dataflow.memoization INFO: Task 3691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3691 try 0 launched on executor multi with executor id 3692 parsl.dataflow.dflow INFO: Standard output for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2023b467-1887-40cd-88b0-d78af183575c_isrForCrosstalkSources_855392_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2023b467-1887-40cd-88b0-d78af183575c_isrForCrosstalkSources_855392_41.stderr parsl.dataflow.dflow DEBUG: Task 3692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3692 has memoization hash 7deb1a38a98a3cc286d7fd667b8c6f0a parsl.dataflow.memoization INFO: Task 3692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3692 try 0 launched on executor multi with executor id 3693 parsl.dataflow.dflow INFO: Standard output for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3c483bc5-b8c8-427f-b564-b2da634349ca_isrForCrosstalkSources_855453_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3c483bc5-b8c8-427f-b564-b2da634349ca_isrForCrosstalkSources_855453_39.stderr parsl.dataflow.dflow DEBUG: Task 3693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3693 has memoization hash 138e8e27fa0e4063098e46295c74e355 parsl.dataflow.memoization INFO: Task 3693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3693 try 0 launched on executor multi with executor id 3694 parsl.dataflow.dflow INFO: Standard output for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f462d030-b462-4440-b154-c4c02d0e3b9b_isrForCrosstalkSources_855434_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f462d030-b462-4440-b154-c4c02d0e3b9b_isrForCrosstalkSources_855434_17.stderr parsl.dataflow.dflow DEBUG: Task 3694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3694 has memoization hash c8243bb10441d3ca98afe2208b064b90 parsl.dataflow.memoization INFO: Task 3694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3694 try 0 launched on executor multi with executor id 3695 parsl.dataflow.dflow INFO: Standard output for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2d320fa2-ff81-4d6b-9c41-09c7a3047275_isrForCrosstalkSources_855406_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2d320fa2-ff81-4d6b-9c41-09c7a3047275_isrForCrosstalkSources_855406_48.stderr parsl.dataflow.dflow DEBUG: Task 3695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3695 has memoization hash 560ee2ddb802a537c20b4627dd9658f7 parsl.dataflow.memoization INFO: Task 3695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3695 try 0 launched on executor multi with executor id 3696 parsl.dataflow.dflow INFO: Standard output for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a09acec7-1983-4e3d-9ed9-275e072b6bcb_isrForCrosstalkSources_855451_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a09acec7-1983-4e3d-9ed9-275e072b6bcb_isrForCrosstalkSources_855451_35.stderr parsl.dataflow.dflow DEBUG: Task 3696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3696 has memoization hash eac64945621ad8b06b2a40d02acd05fa parsl.dataflow.memoization INFO: Task 3696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eedf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3696 try 0 launched on executor multi with executor id 3697 parsl.dataflow.dflow INFO: Standard output for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/16489366-a461-4e72-90b8-cb749b8dea48_isrForCrosstalkSources_855416_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/16489366-a461-4e72-90b8-cb749b8dea48_isrForCrosstalkSources_855416_49.stderr parsl.dataflow.dflow DEBUG: Task 3697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3697 has memoization hash 21a7995c5e6ae474cfda4382bac0b87f parsl.dataflow.memoization INFO: Task 3697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3697 try 0 launched on executor multi with executor id 3698 parsl.dataflow.dflow INFO: Standard output for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d365306a-dd7e-4585-ba1f-23b3656d3180_isrForCrosstalkSources_855446_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d365306a-dd7e-4585-ba1f-23b3656d3180_isrForCrosstalkSources_855446_7.stderr parsl.dataflow.dflow DEBUG: Task 3698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3698 has memoization hash 1ef5270fe9037f256d6ad2550b6de4d7 parsl.dataflow.memoization INFO: Task 3698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3698 try 0 launched on executor multi with executor id 3699 parsl.dataflow.dflow INFO: Standard output for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f52a0f10-3780-4bc4-af41-23ef9c2dff81_isrForCrosstalkSources_855400_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f52a0f10-3780-4bc4-af41-23ef9c2dff81_isrForCrosstalkSources_855400_19.stderr parsl.dataflow.dflow DEBUG: Task 3699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3699 has memoization hash 5f4437d9b8a99161c34ab9403fe0a9da parsl.dataflow.memoization INFO: Task 3699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3699 try 0 launched on executor multi with executor id 3700 parsl.dataflow.dflow INFO: Standard output for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2d7c856f-a111-4e6e-bc69-085301876a75_isrForCrosstalkSources_855429_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2d7c856f-a111-4e6e-bc69-085301876a75_isrForCrosstalkSources_855429_20.stderr parsl.dataflow.dflow DEBUG: Task 3700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3700 has memoization hash 1457d983bd03030b0745d7be635ad95b parsl.dataflow.memoization INFO: Task 3700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3700 try 0 launched on executor multi with executor id 3701 parsl.dataflow.dflow INFO: Standard output for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c6a30af4-2100-4441-b261-5947c242bbb9_isrForCrosstalkSources_855431_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c6a30af4-2100-4441-b261-5947c242bbb9_isrForCrosstalkSources_855431_9.stderr parsl.dataflow.dflow DEBUG: Task 3701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3701 has memoization hash 328ad4b63958eb2023054a57a7a9a8c9 parsl.dataflow.memoization INFO: Task 3701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2cd0938e34da127eedb0ba9d89e6fc with result from task 265 parsl.dataflow.dflow INFO: Parsl task 3701 try 0 launched on executor multi with executor id 3702 parsl.dataflow.dflow INFO: Standard output for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24d447f6-88fd-41c9-b28e-d07a26db4fb0_isrForCrosstalkSources_855390_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24d447f6-88fd-41c9-b28e-d07a26db4fb0_isrForCrosstalkSources_855390_40.stderr parsl.dataflow.dflow DEBUG: Task 3702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3702 has memoization hash 4c5f190815c9a3620f672f11a86e21cd parsl.dataflow.memoization INFO: Task 3702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3702 try 0 launched on executor multi with executor id 3703 parsl.dataflow.dflow INFO: Standard output for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/0eca8f25-9839-45e9-ab70-4c817da2aeb8_isrForCrosstalkSources_855392_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/0eca8f25-9839-45e9-ab70-4c817da2aeb8_isrForCrosstalkSources_855392_35.stderr parsl.dataflow.dflow DEBUG: Task 3703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3703 has memoization hash e8a8bd3b1c772a7c1ac1572fb0c14e33 parsl.dataflow.memoization INFO: Task 3703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/95b0323a-2656-4f78-aeba-44a4111f8aec_isrForCrosstalkSources_855405_56.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/95b0323a-2656-4f78-aeba-44a4111f8aec_isrForCrosstalkSources_855405_56.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb8daf38235073560903f778bc0bdec1 with result from task 258 parsl.dataflow.dflow INFO: Parsl task 3703 try 0 launched on executor multi with executor id 3704 parsl.dataflow.dflow INFO: Standard output for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7bc3ca23-8093-4078-8f74-887287bada26_isrForCrosstalkSources_855413_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7bc3ca23-8093-4078-8f74-887287bada26_isrForCrosstalkSources_855413_25.stderr parsl.dataflow.dflow DEBUG: Task 3704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3704 has memoization hash a7a2e3523ae325749eeafc2841703a19 parsl.dataflow.memoization INFO: Task 3704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3704 try 0 launched on executor multi with executor id 3705 parsl.dataflow.dflow INFO: Standard output for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0a1973c5-f59c-4ee4-b72f-7bdbbe4ae6ef_isrForCrosstalkSources_855418_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0a1973c5-f59c-4ee4-b72f-7bdbbe4ae6ef_isrForCrosstalkSources_855418_32.stderr parsl.dataflow.dflow DEBUG: Task 3705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3705 has memoization hash 430520341d01ab757f4b6ef98a32e598 parsl.dataflow.memoization INFO: Task 3705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/286e38d4-aece-4b84-b69a-df21c0302a72_isrForCrosstalkSources_855453_41.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/286e38d4-aece-4b84-b69a-df21c0302a72_isrForCrosstalkSources_855453_41.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3073e66221165daaac671b6422cfde with result from task 262 parsl.dataflow.dflow INFO: Parsl task 3705 try 0 launched on executor multi with executor id 3706 parsl.dataflow.dflow INFO: Standard output for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3bfd0259-b5e5-4d82-98e6-3b0b40fdf1f5_isrForCrosstalkSources_855433_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3bfd0259-b5e5-4d82-98e6-3b0b40fdf1f5_isrForCrosstalkSources_855433_24.stderr parsl.dataflow.dflow DEBUG: Task 3706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3706 has memoization hash a4ef6eaf421a3566b789f0fc2fb68885 parsl.dataflow.memoization INFO: Task 3706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3706 try 0 launched on executor multi with executor id 3707 parsl.dataflow.dflow INFO: Standard output for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9592d78c-c0ae-4bf9-bb81-20db09af7e2c_isrForCrosstalkSources_855441_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9592d78c-c0ae-4bf9-bb81-20db09af7e2c_isrForCrosstalkSources_855441_32.stderr parsl.dataflow.dflow DEBUG: Task 3707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3707 has memoization hash 3ba65ec4f967d6655fe999e6b58415d0 parsl.dataflow.memoization INFO: Task 3707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e6089ac2-c25b-4691-8e08-66c412fe5201_isrForCrosstalkSources_855433_2.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e6089ac2-c25b-4691-8e08-66c412fe5201_isrForCrosstalkSources_855433_2.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b2675408bf315f2c1f1a1db11b98d0b with result from task 260 parsl.dataflow.dflow INFO: Parsl task 3707 try 0 launched on executor multi with executor id 3708 parsl.dataflow.dflow INFO: Standard output for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/22a3f58b-811c-4a2c-9610-daf3aabc0cb5_isrForCrosstalkSources_855385_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/22a3f58b-811c-4a2c-9610-daf3aabc0cb5_isrForCrosstalkSources_855385_59.stderr parsl.dataflow.dflow DEBUG: Task 3708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3708 has memoization hash 6c2315cda6074b543c04f900d256bb97 parsl.dataflow.memoization INFO: Task 3708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a04179ca-0eff-4b4e-ba43-cb487aa8ac19_isrForCrosstalkSources_855453_33.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a04179ca-0eff-4b4e-ba43-cb487aa8ac19_isrForCrosstalkSources_855453_33.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8ebfbe40efc34b8f76d8e16c6028b34 with result from task 264 parsl.dataflow.dflow INFO: Parsl task 3708 try 0 launched on executor multi with executor id 3709 parsl.dataflow.dflow INFO: Standard output for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c78972bb-7cc9-477d-894b-1575b4a4220e_isrForCrosstalkSources_855451_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c78972bb-7cc9-477d-894b-1575b4a4220e_isrForCrosstalkSources_855451_1.stderr parsl.dataflow.dflow DEBUG: Task 3709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3709 has memoization hash 8f5643839584be6733ef050754bbbb5d parsl.dataflow.memoization INFO: Task 3709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/57f871ff-f46b-4167-9982-6a3d74027b7f_isrForCrosstalkSources_855456_40.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/57f871ff-f46b-4167-9982-6a3d74027b7f_isrForCrosstalkSources_855456_40.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514597f0fe96bf62b63983a7dcef2c63 with result from task 263 parsl.dataflow.dflow INFO: Parsl task 3709 try 0 launched on executor multi with executor id 3710 parsl.dataflow.dflow INFO: Standard output for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/94b18464-8413-4f87-8696-2e62dd69e1d7_isrForCrosstalkSources_855418_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/94b18464-8413-4f87-8696-2e62dd69e1d7_isrForCrosstalkSources_855418_57.stderr parsl.dataflow.dflow DEBUG: Task 3710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3710 has memoization hash e6011e958e67cd6aaea12e58e2a7f190 parsl.dataflow.memoization INFO: Task 3710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b133a2c6-2f8c-4caf-aec3-135c7cf0211f_isrForCrosstalkSources_855423_22.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b133a2c6-2f8c-4caf-aec3-135c7cf0211f_isrForCrosstalkSources_855423_22.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0096d34ba2cec20e25ec4d7f2f77d3 with result from task 259 parsl.dataflow.dflow INFO: Parsl task 3710 try 0 launched on executor multi with executor id 3711 parsl.dataflow.dflow INFO: Standard output for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/52b109a9-4641-49ac-8251-341ab42110c2_isrForCrosstalkSources_855429_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/52b109a9-4641-49ac-8251-341ab42110c2_isrForCrosstalkSources_855429_4.stderr parsl.dataflow.dflow DEBUG: Task 3711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3711 has memoization hash 1ff14a87bcc5aa20181a3ed7d7a073a2 parsl.dataflow.memoization INFO: Task 3711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/20e56840-32d7-42e6-afeb-205bf415ef6c_isrForCrosstalkSources_855403_56.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/20e56840-32d7-42e6-afeb-205bf415ef6c_isrForCrosstalkSources_855403_56.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2624183d66f8ecd7ded506be66357116 with result from task 261 parsl.dataflow.dflow INFO: Parsl task 3711 try 0 launched on executor multi with executor id 3712 parsl.dataflow.dflow INFO: Standard output for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/77832b1e-1cdf-444c-bbdf-964668af3bb7_isrForCrosstalkSources_855459_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/77832b1e-1cdf-444c-bbdf-964668af3bb7_isrForCrosstalkSources_855459_42.stderr parsl.dataflow.dflow DEBUG: Task 3712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3712 has memoization hash b904f4be2bf273c2a332c20c68979630 parsl.dataflow.memoization INFO: Task 3712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5924b07e-c0e6-44c5-827c-82fbfdbd6081_isrForCrosstalkSources_855454_24.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5924b07e-c0e6-44c5-827c-82fbfdbd6081_isrForCrosstalkSources_855454_24.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522d4fa73bc0ca8b74b89c28cbbe8ac7 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e421317-f91e-4a51-9450-67d7a5bc317c_isrForCrosstalkSources_855449_29.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e421317-f91e-4a51-9450-67d7a5bc317c_isrForCrosstalkSources_855449_29.stderr parsl.dataflow.dflow INFO: Parsl task 3712 try 0 launched on executor multi with executor id 3713 parsl.dataflow.dflow INFO: Standard output for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f764e681-afb0-45ff-be79-27cadb5c2dff_isrForCrosstalkSources_855390_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f764e681-afb0-45ff-be79-27cadb5c2dff_isrForCrosstalkSources_855390_45.stderr parsl.dataflow.dflow DEBUG: Task 3713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3713 has memoization hash 0313692aa505c34c25a485f7228ccc57 parsl.dataflow.memoization INFO: Task 3713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3713 try 0 launched on executor multi with executor id 3714 parsl.dataflow.dflow INFO: Standard output for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2c954487-cb98-400e-afa2-33b989965dbd_isrForCrosstalkSources_855401_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2c954487-cb98-400e-afa2-33b989965dbd_isrForCrosstalkSources_855401_8.stderr parsl.dataflow.dflow DEBUG: Task 3714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3714 has memoization hash 51ef061fa3e5f1f4b38470e3edb41e83 parsl.dataflow.memoization INFO: Task 3714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf1673bb2373590124c50dbe88abaff1 with result from task 302 2024-03-09 14:51:39 proc_lsst.multi:170 [INFO] [multi] job_id 16964172 proc_lsst.multi INFO: [multi] job_id 16964172 2024-03-09 14:51:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-09 14:51:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964172 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964172 parsl.executors.status_handling DEBUG: Launched block 146 on executor multi with job ID 16964172 parsl.executors.status_handling INFO: Allocated block ID 147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-147 2024-03-09 14:51:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 2024-03-09 14:51:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-09 14:51:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3714 try 0 launched on executor multi with executor id 3715 parsl.dataflow.dflow INFO: Standard output for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/faf00e26-00ba-41a5-ba11-895d293b50d2_isrForCrosstalkSources_855412_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/faf00e26-00ba-41a5-ba11-895d293b50d2_isrForCrosstalkSources_855412_42.stderr parsl.dataflow.dflow DEBUG: Task 3715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3715 has memoization hash a7200cc01d49f7cd1e9cb65973e3bd6f parsl.dataflow.memoization INFO: Task 3715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3715 try 0 launched on executor multi with executor id 3716 parsl.dataflow.dflow INFO: Standard output for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/2a1f095c-8609-4104-b46b-2cd9e783560f_isrForCrosstalkSources_855427_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/2a1f095c-8609-4104-b46b-2cd9e783560f_isrForCrosstalkSources_855427_61.stderr parsl.dataflow.dflow DEBUG: Task 3716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3716 has memoization hash ee223419cebe908b715ccc6e5e7c4888 parsl.dataflow.memoization INFO: Task 3716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d9a91b7d-2012-4be8-b1f9-6c44b8342830_isrForCrosstalkSources_855415_7.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d9a91b7d-2012-4be8-b1f9-6c44b8342830_isrForCrosstalkSources_855415_7.stderr parsl.dataflow.dflow INFO: Parsl task 3716 try 0 launched on executor multi with executor id 3717 parsl.dataflow.dflow INFO: Standard output for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9676fe8c-31eb-4aa4-b319-e946100f08fb_isrForCrosstalkSources_855382_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9676fe8c-31eb-4aa4-b319-e946100f08fb_isrForCrosstalkSources_855382_31.stderr parsl.dataflow.dflow DEBUG: Task 3717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3717 has memoization hash 9f7908931096277309f752456913aa71 parsl.dataflow.memoization INFO: Task 3717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3717 try 0 launched on executor multi with executor id 3718 parsl.dataflow.dflow INFO: Standard output for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e5b7a2f0-5d66-4286-8ce4-a2f6867ec9da_isrForCrosstalkSources_855416_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e5b7a2f0-5d66-4286-8ce4-a2f6867ec9da_isrForCrosstalkSources_855416_46.stderr parsl.dataflow.dflow DEBUG: Task 3718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3718 has memoization hash c6799ef143b7abe1d3d30e1a4dc24b7e parsl.dataflow.memoization INFO: Task 3718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3718 try 0 launched on executor multi with executor id 3719 parsl.dataflow.dflow INFO: Standard output for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3bb48352-9f4a-44c6-a25b-a9312ca93808_isrForCrosstalkSources_855424_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3bb48352-9f4a-44c6-a25b-a9312ca93808_isrForCrosstalkSources_855424_21.stderr parsl.dataflow.dflow DEBUG: Task 3719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3719 has memoization hash 8e598bbf23e2b62cce3067e6e6cab31c parsl.dataflow.memoization INFO: Task 3719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3719 try 0 launched on executor multi with executor id 3720 parsl.dataflow.dflow INFO: Standard output for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1f49fccc-4475-4eee-a32c-d8eb0f8166e0_isrForCrosstalkSources_855442_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1f49fccc-4475-4eee-a32c-d8eb0f8166e0_isrForCrosstalkSources_855442_55.stderr parsl.dataflow.dflow DEBUG: Task 3720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3720 has memoization hash e89bc0fd8cb681efb6b9213b6b93f13f parsl.dataflow.memoization INFO: Task 3720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3720 try 0 launched on executor multi with executor id 3721 parsl.dataflow.dflow INFO: Standard output for task 3720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f607274c-ad8c-43e0-b2c6-3ee34b1f3f22_isrForCrosstalkSources_855392_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f607274c-ad8c-43e0-b2c6-3ee34b1f3f22_isrForCrosstalkSources_855392_60.stderr parsl.dataflow.dflow DEBUG: Task 3721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3721 has memoization hash 5a8917b5a019fa9897c6e7d271a95f06 parsl.dataflow.memoization INFO: Task 3721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3721 try 0 launched on executor multi with executor id 3722 parsl.dataflow.dflow INFO: Standard output for task 3721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/303570a2-2ad2-447f-8747-626876a23a62_isrForCrosstalkSources_855403_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/303570a2-2ad2-447f-8747-626876a23a62_isrForCrosstalkSources_855403_6.stderr parsl.dataflow.dflow DEBUG: Task 3722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3722 has memoization hash 4d99b3de7b2be3d940e73c189c9993a1 parsl.dataflow.memoization INFO: Task 3722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5e4eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3722 try 0 launched on executor multi with executor id 3723 parsl.dataflow.dflow INFO: Standard output for task 3722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6682204e-c77b-400b-9493-94da19955df6_isrForCrosstalkSources_855397_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6682204e-c77b-400b-9493-94da19955df6_isrForCrosstalkSources_855397_52.stderr parsl.dataflow.dflow DEBUG: Task 3723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3723 has memoization hash 89aa5ddbf08e3cb6643e367721296516 parsl.dataflow.memoization INFO: Task 3723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3723 try 0 launched on executor multi with executor id 3724 parsl.dataflow.dflow INFO: Standard output for task 3723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a4b72e4a-c459-4ac1-abbe-8ef3b2b28836_isrForCrosstalkSources_855393_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a4b72e4a-c459-4ac1-abbe-8ef3b2b28836_isrForCrosstalkSources_855393_52.stderr parsl.dataflow.dflow DEBUG: Task 3724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3724 has memoization hash cc456167cf2dfbabc77909a854e29808 parsl.dataflow.memoization INFO: Task 3724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d623e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3724 try 0 launched on executor multi with executor id 3725 parsl.dataflow.dflow INFO: Standard output for task 3724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/769e8593-9407-4164-8975-bdd9838dd333_isrForCrosstalkSources_855397_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/769e8593-9407-4164-8975-bdd9838dd333_isrForCrosstalkSources_855397_3.stderr parsl.dataflow.dflow DEBUG: Task 3725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3725 has memoization hash 87558b40b4afab03ec75c2be071385b6 parsl.dataflow.memoization INFO: Task 3725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3725 try 0 launched on executor multi with executor id 3726 parsl.dataflow.dflow INFO: Standard output for task 3725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f6e01602-c6fc-4aa7-8323-570b7bca8900_isrForCrosstalkSources_855453_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f6e01602-c6fc-4aa7-8323-570b7bca8900_isrForCrosstalkSources_855453_1.stderr parsl.dataflow.dflow DEBUG: Task 3726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3726 has memoization hash 39306a411c439cded272ecd9c76a0c5a parsl.dataflow.memoization INFO: Task 3726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3726 try 0 launched on executor multi with executor id 3727 parsl.dataflow.dflow INFO: Standard output for task 3726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/cd2e5160-3def-4d85-84dc-b86cd8bbf21f_isrForCrosstalkSources_855388_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/cd2e5160-3def-4d85-84dc-b86cd8bbf21f_isrForCrosstalkSources_855388_28.stderr parsl.dataflow.dflow DEBUG: Task 3727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3727 has memoization hash 332a791bd3e3747227b5c97e75dfbe14 parsl.dataflow.memoization INFO: Task 3727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3727 try 0 launched on executor multi with executor id 3728 parsl.dataflow.dflow INFO: Standard output for task 3727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a1b8fdf7-fd6a-4664-8e2a-4d942857ab34_isrForCrosstalkSources_855421_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a1b8fdf7-fd6a-4664-8e2a-4d942857ab34_isrForCrosstalkSources_855421_56.stderr parsl.dataflow.dflow DEBUG: Task 3728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3728 has memoization hash c61d2ee20d4bafcbb1d84a3eef9abfa6 parsl.dataflow.memoization INFO: Task 3728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3728 try 0 launched on executor multi with executor id 3729 parsl.dataflow.dflow INFO: Standard output for task 3728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6b3420bc-2a5a-4417-9a46-ad60a6ba3da9_isrForCrosstalkSources_855394_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6b3420bc-2a5a-4417-9a46-ad60a6ba3da9_isrForCrosstalkSources_855394_24.stderr parsl.dataflow.dflow DEBUG: Task 3729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3729 has memoization hash 8dd29172854b6ddbe03fd6bfab2a5500 parsl.dataflow.memoization INFO: Task 3729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3729 try 0 launched on executor multi with executor id 3730 parsl.dataflow.dflow INFO: Standard output for task 3729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e1fa3ac0-e44a-4b97-8b7b-9a30e8f90198_isrForCrosstalkSources_855432_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e1fa3ac0-e44a-4b97-8b7b-9a30e8f90198_isrForCrosstalkSources_855432_31.stderr parsl.dataflow.dflow DEBUG: Task 3730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3730 has memoization hash 77137b5ff0adb5fdb63972b806f6501a parsl.dataflow.memoization INFO: Task 3730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3730 try 0 launched on executor multi with executor id 3731 parsl.dataflow.dflow INFO: Standard output for task 3730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8d7cef6a-99bb-444d-8c0d-d5db03543bff_isrForCrosstalkSources_855449_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8d7cef6a-99bb-444d-8c0d-d5db03543bff_isrForCrosstalkSources_855449_52.stderr parsl.dataflow.dflow DEBUG: Task 3731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3731 has memoization hash 18828405c4f55ce14a1cc6d2df27515f parsl.dataflow.memoization INFO: Task 3731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 281c23e0ef7191627f5df6115f7d7956 with result from task 331 parsl.dataflow.dflow INFO: Parsl task 3731 try 0 launched on executor multi with executor id 3732 parsl.dataflow.dflow INFO: Standard output for task 3731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f49f7637-c71d-4b0f-ab4c-1e82f0af05d5_isrForCrosstalkSources_855459_46.stdout parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Standard error for task 3731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f49f7637-c71d-4b0f-ab4c-1e82f0af05d5_isrForCrosstalkSources_855459_46.stderr parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 3732 will be sent to executor multi parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8772ef1d-239c-465b-966c-37bd15c1d545_isrForCrosstalkSources_855397_28.stdout parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8772ef1d-239c-465b-966c-37bd15c1d545_isrForCrosstalkSources_855397_28.stderr parsl.dataflow.dflow INFO: Task 3732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3732 has memoization hash 2f713da4a9e811d5e1e87fa99392ed95 parsl.dataflow.memoization INFO: Task 3732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3732 try 0 launched on executor multi with executor id 3733 parsl.dataflow.dflow INFO: Standard output for task 3732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e0ee264b-610f-472f-8f1a-abcb8c8ba4dd_isrForCrosstalkSources_855430_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e0ee264b-610f-472f-8f1a-abcb8c8ba4dd_isrForCrosstalkSources_855430_6.stderr parsl.dataflow.dflow DEBUG: Task 3733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3733 has memoization hash c4d6904bda48ee553a7f6ee799f2729d parsl.dataflow.memoization INFO: Task 3733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3733 try 0 launched on executor multi with executor id 3734 parsl.dataflow.dflow INFO: Standard output for task 3733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c4f0401b-7289-4aea-80e7-b48f3b90071b_isrForCrosstalkSources_855429_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c4f0401b-7289-4aea-80e7-b48f3b90071b_isrForCrosstalkSources_855429_62.stderr parsl.dataflow.dflow DEBUG: Task 3734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3734 has memoization hash 3ee3ccb953c4ea60224f6de4ee93a6ce parsl.dataflow.memoization INFO: Task 3734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3734 try 0 launched on executor multi with executor id 3735 parsl.dataflow.dflow INFO: Standard output for task 3734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a8106ae8-255d-4e0c-b4ff-7260a627ee17_isrForCrosstalkSources_855443_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a8106ae8-255d-4e0c-b4ff-7260a627ee17_isrForCrosstalkSources_855443_1.stderr parsl.dataflow.dflow DEBUG: Task 3735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3735 has memoization hash 6badcdf0ab28edbf4e519bd3c905aa5a parsl.dataflow.memoization INFO: Task 3735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3735 try 0 launched on executor multi with executor id 3736 parsl.dataflow.dflow INFO: Standard output for task 3735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b8814759-4238-45ba-b39d-6106eacd30ac_isrForCrosstalkSources_855454_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b8814759-4238-45ba-b39d-6106eacd30ac_isrForCrosstalkSources_855454_36.stderr parsl.dataflow.dflow DEBUG: Task 3736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3736 has memoization hash e9a7f89dedacf4ce31ceefe1a3863057 parsl.dataflow.memoization INFO: Task 3736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3736 try 0 launched on executor multi with executor id 3737 parsl.dataflow.dflow INFO: Standard output for task 3736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4278d7c9-a5f5-4a98-9137-119a3b5a09de_isrForCrosstalkSources_855413_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4278d7c9-a5f5-4a98-9137-119a3b5a09de_isrForCrosstalkSources_855413_7.stderr parsl.dataflow.dflow DEBUG: Task 3737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3737 has memoization hash b59e0222590f0ffcdc0df098e5c205da parsl.dataflow.memoization INFO: Task 3737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 188feb473dc555cea717b1d64d4e6d49 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/58369c1e-e131-4696-8d47-fa956fd0d244_isrForCrosstalkSources_855399_2.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/58369c1e-e131-4696-8d47-fa956fd0d244_isrForCrosstalkSources_855399_2.stderr parsl.dataflow.dflow INFO: Parsl task 3737 try 0 launched on executor multi with executor id 3738 parsl.dataflow.dflow INFO: Standard output for task 3737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/95e18f2a-ba45-4f40-b320-d392c1c86488_isrForCrosstalkSources_855413_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/95e18f2a-ba45-4f40-b320-d392c1c86488_isrForCrosstalkSources_855413_19.stderr parsl.dataflow.dflow DEBUG: Task 3738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3738 has memoization hash d075057fd3a58c595bf41ac3fea70ab3 parsl.dataflow.memoization INFO: Task 3738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3738 try 0 launched on executor multi with executor id 3739 parsl.dataflow.dflow INFO: Standard output for task 3738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c2b9261e-6131-4b40-b8fe-aa91a383047f_isrForCrosstalkSources_855420_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c2b9261e-6131-4b40-b8fe-aa91a383047f_isrForCrosstalkSources_855420_15.stderr parsl.dataflow.dflow DEBUG: Task 3739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3739 has memoization hash f8c429f3c1d7776b69dd6028de0b08f8 parsl.dataflow.memoization INFO: Task 3739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3739 try 0 launched on executor multi with executor id 3740 parsl.dataflow.dflow INFO: Standard output for task 3739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2fab58c4-f35d-4a91-a3c4-3ff704e00a1e_isrForCrosstalkSources_855411_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2fab58c4-f35d-4a91-a3c4-3ff704e00a1e_isrForCrosstalkSources_855411_55.stderr parsl.dataflow.dflow DEBUG: Task 3740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3740 has memoization hash f0407c41cb4f80d87450563246895d27 parsl.dataflow.memoization INFO: Task 3740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3740 try 0 launched on executor multi with executor id 3741 parsl.dataflow.dflow INFO: Standard output for task 3740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/916fe9ac-3cf7-456d-bdbc-52602123b326_isrForCrosstalkSources_855417_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/916fe9ac-3cf7-456d-bdbc-52602123b326_isrForCrosstalkSources_855417_24.stderr parsl.dataflow.dflow DEBUG: Task 3741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3741 has memoization hash 89ecd28768fe3e6fc2f0b9ff6a160e01 parsl.dataflow.memoization INFO: Task 3741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3741 try 0 launched on executor multi with executor id 3742 parsl.dataflow.dflow INFO: Standard output for task 3741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/eef2184e-84ae-44b0-9f8a-1d660abd38f8_isrForCrosstalkSources_855455_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/eef2184e-84ae-44b0-9f8a-1d660abd38f8_isrForCrosstalkSources_855455_4.stderr parsl.dataflow.dflow DEBUG: Task 3742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3742 has memoization hash f6dbd0053ae90306a613ee2577c9153a parsl.dataflow.memoization INFO: Task 3742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3742 try 0 launched on executor multi with executor id 3743 parsl.dataflow.dflow INFO: Standard output for task 3742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e63ee2f9-c6c0-4915-bdee-3042666095e3_isrForCrosstalkSources_855419_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e63ee2f9-c6c0-4915-bdee-3042666095e3_isrForCrosstalkSources_855419_29.stderr parsl.dataflow.dflow DEBUG: Task 3743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3743 has memoization hash d7ca162e0d3c9093a305a4c38809a4a7 parsl.dataflow.memoization INFO: Task 3743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3743 try 0 launched on executor multi with executor id 3744 parsl.dataflow.dflow INFO: Standard output for task 3743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/1c7e0d12-f2d0-4801-9192-1cecba439eba_isrForCrosstalkSources_855451_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/1c7e0d12-f2d0-4801-9192-1cecba439eba_isrForCrosstalkSources_855451_22.stderr parsl.dataflow.dflow DEBUG: Task 3744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3744 has memoization hash 047323f6a756cc69a3b0c321c4d53717 parsl.dataflow.memoization INFO: Task 3744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3744 try 0 launched on executor multi with executor id 3745 parsl.dataflow.dflow INFO: Standard output for task 3744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f9fd9eec-a6a5-49e8-aba5-786025ea8f55_isrForCrosstalkSources_855387_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f9fd9eec-a6a5-49e8-aba5-786025ea8f55_isrForCrosstalkSources_855387_54.stderr parsl.dataflow.dflow DEBUG: Task 3745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3745 has memoization hash 12b3020f43ed89070e513407f190bf98 parsl.dataflow.memoization INFO: Task 3745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3745 try 0 launched on executor multi with executor id 3746 parsl.dataflow.dflow INFO: Standard output for task 3745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/242d89e7-fbf2-47cf-af04-980f457ba055_isrForCrosstalkSources_855429_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/242d89e7-fbf2-47cf-af04-980f457ba055_isrForCrosstalkSources_855429_6.stderr parsl.dataflow.dflow DEBUG: Task 3746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3746 has memoization hash a02f2658e29229d7456d5f7e49e82cbd parsl.dataflow.memoization INFO: Task 3746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3746 try 0 launched on executor multi with executor id 3747 parsl.dataflow.dflow INFO: Standard output for task 3746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fe684390-86b3-4ce8-a576-f6506e97f75c_isrForCrosstalkSources_855417_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fe684390-86b3-4ce8-a576-f6506e97f75c_isrForCrosstalkSources_855417_55.stderr parsl.dataflow.dflow DEBUG: Task 3747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3747 has memoization hash 31295d33864bae17e716ca05c996ddf7 parsl.dataflow.memoization INFO: Task 3747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3747 try 0 launched on executor multi with executor id 3748 parsl.dataflow.dflow INFO: Standard output for task 3747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a569e825-fa61-4f88-85fb-7316bb89e279_isrForCrosstalkSources_855450_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a569e825-fa61-4f88-85fb-7316bb89e279_isrForCrosstalkSources_855450_7.stderr parsl.dataflow.dflow DEBUG: Task 3748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3748 has memoization hash 586281e4b55be6cbe340c256cc15cd50 parsl.dataflow.memoization INFO: Task 3748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3748 try 0 launched on executor multi with executor id 3749 parsl.dataflow.dflow INFO: Standard output for task 3748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5e490d57-bd2d-4210-a2af-03a5f9f24e57_isrForCrosstalkSources_855405_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5e490d57-bd2d-4210-a2af-03a5f9f24e57_isrForCrosstalkSources_855405_21.stderr parsl.dataflow.dflow DEBUG: Task 3749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3749 has memoization hash ddcc8eeb7a1f0cc37643d44e3a4acd1b parsl.dataflow.memoization INFO: Task 3749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3749 try 0 launched on executor multi with executor id 3750 parsl.dataflow.dflow INFO: Standard output for task 3749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6475bc93-494c-4cc4-b274-4a08b542f8b9_isrForCrosstalkSources_855387_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6475bc93-494c-4cc4-b274-4a08b542f8b9_isrForCrosstalkSources_855387_58.stderr parsl.dataflow.dflow DEBUG: Task 3750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3750 has memoization hash 132f737c1a8e139a74ad1a63ba2d8227 parsl.dataflow.memoization INFO: Task 3750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eefec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3750 try 0 launched on executor multi with executor id 3751 parsl.dataflow.dflow INFO: Standard output for task 3750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f043d470-8efd-4c2e-b4fe-e93cda53fd49_isrForCrosstalkSources_855417_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f043d470-8efd-4c2e-b4fe-e93cda53fd49_isrForCrosstalkSources_855417_48.stderr parsl.dataflow.dflow DEBUG: Task 3751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3751 has memoization hash 855f14479cf9a024762a8496caab9cae parsl.dataflow.memoization INFO: Task 3751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3751 try 0 launched on executor multi with executor id 3752 parsl.dataflow.dflow INFO: Standard output for task 3751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/36b9270b-0b00-4d7b-bf62-540e5714f30f_isrForCrosstalkSources_855431_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/36b9270b-0b00-4d7b-bf62-540e5714f30f_isrForCrosstalkSources_855431_59.stderr parsl.dataflow.dflow DEBUG: Task 3752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3752 has memoization hash 91d997f418ad0c23d9718ac088dbf630 parsl.dataflow.memoization INFO: Task 3752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3752 try 0 launched on executor multi with executor id 3753 parsl.dataflow.dflow INFO: Standard output for task 3752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/2ee62b5e-619e-4de4-8762-00fcfe02efcf_isrForCrosstalkSources_855445_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/2ee62b5e-619e-4de4-8762-00fcfe02efcf_isrForCrosstalkSources_855445_10.stderr parsl.dataflow.dflow DEBUG: Task 3753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3753 has memoization hash 3413995975bfef0027007e2a874b0abd parsl.dataflow.memoization INFO: Task 3753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d600e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3753 try 0 launched on executor multi with executor id 3754 parsl.dataflow.dflow INFO: Standard output for task 3753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c2c2d003-5a87-4ab2-a93e-60d5fb18893e_isrForCrosstalkSources_855435_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c2c2d003-5a87-4ab2-a93e-60d5fb18893e_isrForCrosstalkSources_855435_56.stderr parsl.dataflow.dflow DEBUG: Task 3754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3754 has memoization hash 74752565bf728611cb185d23f3c8963e parsl.dataflow.memoization INFO: Task 3754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3754 try 0 launched on executor multi with executor id 3755 parsl.dataflow.dflow INFO: Standard output for task 3754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2ba87b9a-ce05-4e39-ab1e-5ae37c126d2d_isrForCrosstalkSources_855398_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2ba87b9a-ce05-4e39-ab1e-5ae37c126d2d_isrForCrosstalkSources_855398_48.stderr parsl.dataflow.dflow DEBUG: Task 3755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3755 has memoization hash fd97916757bb659bf39b1e6a335e98af parsl.dataflow.memoization INFO: Task 3755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3755 try 0 launched on executor multi with executor id 3756 parsl.dataflow.dflow INFO: Standard output for task 3755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/51afda2f-366a-49db-8572-c2459bd61021_isrForCrosstalkSources_855391_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/51afda2f-366a-49db-8572-c2459bd61021_isrForCrosstalkSources_855391_42.stderr parsl.dataflow.dflow DEBUG: Task 3756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3756 has memoization hash 93f9bf49ede719842c94e49fa6e2a75d parsl.dataflow.memoization INFO: Task 3756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3756 try 0 launched on executor multi with executor id 3757 parsl.dataflow.dflow INFO: Standard output for task 3756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/058f818a-dc13-4dd4-a4c5-0527ccc87076_isrForCrosstalkSources_855443_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/058f818a-dc13-4dd4-a4c5-0527ccc87076_isrForCrosstalkSources_855443_25.stderr parsl.dataflow.dflow DEBUG: Task 3757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3757 has memoization hash f718345f41f85ffc6d40888716e3c849 parsl.dataflow.memoization INFO: Task 3757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3757 try 0 launched on executor multi with executor id 3758 parsl.dataflow.dflow INFO: Standard output for task 3757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/cdfe4257-f6d3-4856-b654-0f0d92699c34_isrForCrosstalkSources_855395_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/cdfe4257-f6d3-4856-b654-0f0d92699c34_isrForCrosstalkSources_855395_59.stderr parsl.dataflow.dflow DEBUG: Task 3758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3758 has memoization hash a0288e9d240cee722634af1a486c3b0a parsl.dataflow.memoization INFO: Task 3758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3758 try 0 launched on executor multi with executor id 3759 parsl.dataflow.dflow INFO: Standard output for task 3758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f5a2027a-54e5-47fa-b725-b7d1af199f11_isrForCrosstalkSources_855379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f5a2027a-54e5-47fa-b725-b7d1af199f11_isrForCrosstalkSources_855379_2.stderr parsl.dataflow.dflow DEBUG: Task 3759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3759 has memoization hash f26e52240903da6b31dfae881b54933b parsl.dataflow.memoization INFO: Task 3759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d602c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3759 try 0 launched on executor multi with executor id 3760 parsl.dataflow.dflow INFO: Standard output for task 3759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f658c58d-e18e-451c-9859-20543777b603_isrForCrosstalkSources_855431_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f658c58d-e18e-451c-9859-20543777b603_isrForCrosstalkSources_855431_39.stderr parsl.dataflow.dflow DEBUG: Task 3760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3760 has memoization hash c7973393121f7472bfa295425c609321 parsl.dataflow.memoization INFO: Task 3760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3760 try 0 launched on executor multi with executor id 3761 parsl.dataflow.dflow INFO: Standard output for task 3760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3c871020-392a-4320-a18b-f12ef6d3c61d_isrForCrosstalkSources_855433_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3c871020-392a-4320-a18b-f12ef6d3c61d_isrForCrosstalkSources_855433_39.stderr parsl.dataflow.dflow DEBUG: Task 3761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3761 has memoization hash f23e2d018f29b1ec361f0d5086392e60 parsl.dataflow.memoization INFO: Task 3761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3761 try 0 launched on executor multi with executor id 3762 parsl.dataflow.dflow INFO: Standard output for task 3761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5038f7bc-b53d-4687-995b-f63c6661aae3_isrForCrosstalkSources_855439_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5038f7bc-b53d-4687-995b-f63c6661aae3_isrForCrosstalkSources_855439_54.stderr parsl.dataflow.dflow DEBUG: Task 3762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3762 has memoization hash bcbd8b1e14a8d31c928140ff0c2e713c parsl.dataflow.memoization INFO: Task 3762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3762 try 0 launched on executor multi with executor id 3763 parsl.dataflow.dflow INFO: Standard output for task 3762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/93dfa094-786e-49ef-8dd9-ff1b28428fa6_isrForCrosstalkSources_855403_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/93dfa094-786e-49ef-8dd9-ff1b28428fa6_isrForCrosstalkSources_855403_11.stderr parsl.dataflow.dflow DEBUG: Task 3763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3763 has memoization hash 154ba167c547dcf3248d0507d9c453ef parsl.dataflow.memoization INFO: Task 3763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:40 proc_lsst.multi:170 [INFO] [multi] job_id 16964173 proc_lsst.multi INFO: [multi] job_id 16964173 2024-03-09 14:51:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-09 14:51:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964173 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964173 parsl.executors.status_handling DEBUG: Launched block 147 on executor multi with job ID 16964173 parsl.executors.status_handling INFO: Allocated block ID 148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-148 2024-03-09 14:51:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a52d186d772819871bc9dd27875ab3e with result from task 323 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a8e320e-018e-417f-b24a-271456defc16_isrForCrosstalkSources_855452_2.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a8e320e-018e-417f-b24a-271456defc16_isrForCrosstalkSources_855452_2.stderr parsl.dataflow.dflow INFO: Parsl task 3763 try 0 launched on executor multi with executor id 3764 parsl.dataflow.dflow INFO: Standard output for task 3763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/215fd0b5-7cbd-45f5-bcfe-ddf06d192f94_isrForCrosstalkSources_855426_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/215fd0b5-7cbd-45f5-bcfe-ddf06d192f94_isrForCrosstalkSources_855426_61.stderr parsl.dataflow.dflow DEBUG: Task 3764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3764 has memoization hash c088da5c38c8c6d718913d85e5d1a78b parsl.dataflow.memoization INFO: Task 3764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3764 try 0 launched on executor multi with executor id 3765 parsl.dataflow.dflow INFO: Standard output for task 3764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2028ab19-a252-4c76-b389-18de0df5fb8f_isrForCrosstalkSources_855442_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2028ab19-a252-4c76-b389-18de0df5fb8f_isrForCrosstalkSources_855442_37.stderr parsl.dataflow.dflow DEBUG: Task 3765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3765 has memoization hash e1bbf644f50c66d3f0dbf4b42a0989c4 parsl.dataflow.memoization INFO: Task 3765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d605e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3765 try 0 launched on executor multi with executor id 3766 parsl.dataflow.dflow INFO: Standard output for task 3765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b5e498ab-0fb7-483f-a7af-b77f814b9d23_isrForCrosstalkSources_855415_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b5e498ab-0fb7-483f-a7af-b77f814b9d23_isrForCrosstalkSources_855415_55.stderr parsl.dataflow.dflow DEBUG: Task 3766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3766 has memoization hash d28400259f11418164c362c2b40363fa parsl.dataflow.memoization INFO: Task 3766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3766 try 0 launched on executor multi with executor id 3767 parsl.dataflow.dflow INFO: Standard output for task 3766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b63987e-e9e7-4863-8239-b8c175189c2d_isrForCrosstalkSources_855449_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b63987e-e9e7-4863-8239-b8c175189c2d_isrForCrosstalkSources_855449_32.stderr parsl.dataflow.dflow DEBUG: Task 3767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3767 has memoization hash 826c03120526513386571500a302daf3 parsl.dataflow.memoization INFO: Task 3767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3767 try 0 launched on executor multi with executor id 3768 parsl.dataflow.dflow INFO: Standard output for task 3767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/0c5a1922-d189-4299-a447-7c12ea6dcaa3_isrForCrosstalkSources_855410_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/0c5a1922-d189-4299-a447-7c12ea6dcaa3_isrForCrosstalkSources_855410_2.stderr parsl.dataflow.dflow DEBUG: Task 3768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3768 has memoization hash 0f3793fb52ed984752baac9306726aa0 parsl.dataflow.memoization INFO: Task 3768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3768 try 0 launched on executor multi with executor id 3769 parsl.dataflow.dflow INFO: Standard output for task 3768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/01f69a39-0f20-4196-a731-bfc0a717d2db_isrForCrosstalkSources_855436_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/01f69a39-0f20-4196-a731-bfc0a717d2db_isrForCrosstalkSources_855436_55.stderr parsl.dataflow.dflow DEBUG: Task 3769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3769 has memoization hash 91ec19610f0da8aac5dbe7e6919229f0 parsl.dataflow.memoization INFO: Task 3769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3769 try 0 launched on executor multi with executor id 3770 parsl.dataflow.dflow INFO: Standard output for task 3769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/64b8e42c-9657-4d76-9915-879f701b73cc_isrForCrosstalkSources_855388_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/64b8e42c-9657-4d76-9915-879f701b73cc_isrForCrosstalkSources_855388_52.stderr parsl.dataflow.dflow DEBUG: Task 3770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3770 has memoization hash 87be5e0de037315cd7911703b3e84bc7 parsl.dataflow.memoization INFO: Task 3770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3770 try 0 launched on executor multi with executor id 3771 parsl.dataflow.dflow INFO: Standard output for task 3770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9a20a477-3f55-4090-891f-5d6ba793daf1_isrForCrosstalkSources_855456_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9a20a477-3f55-4090-891f-5d6ba793daf1_isrForCrosstalkSources_855456_48.stderr parsl.dataflow.dflow DEBUG: Task 3771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3771 has memoization hash 78896fbba3d1c813c976d246ed9d3bca parsl.dataflow.memoization INFO: Task 3771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3771 try 0 launched on executor multi with executor id 3772 parsl.dataflow.dflow INFO: Standard output for task 3771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/1e8a9629-ac39-47c9-b75e-ac54a0aca148_isrForCrosstalkSources_855404_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/1e8a9629-ac39-47c9-b75e-ac54a0aca148_isrForCrosstalkSources_855404_33.stderr parsl.dataflow.dflow DEBUG: Task 3772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3772 has memoization hash 6b182cc80b20244d0779fc05d2a7744b parsl.dataflow.memoization INFO: Task 3772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d609a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3772 try 0 launched on executor multi with executor id 3773 parsl.dataflow.dflow INFO: Standard output for task 3772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/48977c39-e3d3-483a-bb6c-3255ab139dc6_isrForCrosstalkSources_855450_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/48977c39-e3d3-483a-bb6c-3255ab139dc6_isrForCrosstalkSources_855450_14.stderr parsl.dataflow.dflow DEBUG: Task 3773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3773 has memoization hash 3ca4359238bb02f1504203b456fa162a parsl.dataflow.memoization INFO: Task 3773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3773 try 0 launched on executor multi with executor id 3774 parsl.dataflow.dflow INFO: Standard output for task 3773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68c66a67-ea5f-4f13-83cc-2817db5d66f9_isrForCrosstalkSources_855399_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68c66a67-ea5f-4f13-83cc-2817db5d66f9_isrForCrosstalkSources_855399_49.stderr parsl.dataflow.dflow DEBUG: Task 3774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3774 has memoization hash 642029ad24556e07e61013cdb42c1edb parsl.dataflow.memoization INFO: Task 3774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3774 try 0 launched on executor multi with executor id 3775 parsl.dataflow.dflow INFO: Standard output for task 3774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/12fb37dd-f6a6-43af-b23c-cc9df7d54232_isrForCrosstalkSources_855386_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/12fb37dd-f6a6-43af-b23c-cc9df7d54232_isrForCrosstalkSources_855386_33.stderr parsl.dataflow.dflow DEBUG: Task 3775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3775 has memoization hash 76f4bf6899817b23c991356f3f3a5a21 parsl.dataflow.memoization INFO: Task 3775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3775 try 0 launched on executor multi with executor id 3776 parsl.dataflow.dflow INFO: Standard output for task 3775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b91eb5fc-cb97-4b05-b39b-95fece5d075f_isrForCrosstalkSources_855442_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b91eb5fc-cb97-4b05-b39b-95fece5d075f_isrForCrosstalkSources_855442_42.stderr parsl.dataflow.dflow DEBUG: Task 3776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3776 has memoization hash ab3cd72556a86c9d91a188364b38511c parsl.dataflow.memoization INFO: Task 3776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3776 try 0 launched on executor multi with executor id 3777 parsl.dataflow.dflow INFO: Standard output for task 3776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/111f52e3-e56b-40e6-87bc-354b29b3d88d_isrForCrosstalkSources_855440_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/111f52e3-e56b-40e6-87bc-354b29b3d88d_isrForCrosstalkSources_855440_16.stderr parsl.dataflow.dflow DEBUG: Task 3777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3777 has memoization hash 27fab50465900de36aa0579af59b2af9 parsl.dataflow.memoization INFO: Task 3777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e89ef0466ffd89426f657c2abfb9c59 with result from task 176 parsl.dataflow.dflow INFO: Parsl task 3777 try 0 launched on executor multi with executor id 3778 parsl.dataflow.dflow INFO: Standard output for task 3777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0ccba4cc-6240-4e55-9098-830f3c94ed16_isrForCrosstalkSources_855452_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0ccba4cc-6240-4e55-9098-830f3c94ed16_isrForCrosstalkSources_855452_5.stderr parsl.dataflow.dflow DEBUG: Task 3778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3778 has memoization hash 61c986802c4f03f72e88d11de88c07e2 parsl.dataflow.memoization INFO: Task 3778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/1273549f-7a51-4e9e-a2af-ba8d29a70f78_isrForCrosstalkSources_855414_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/1273549f-7a51-4e9e-a2af-ba8d29a70f78_isrForCrosstalkSources_855414_59.stderr parsl.dataflow.dflow INFO: Parsl task 3778 try 0 launched on executor multi with executor id 3779 parsl.dataflow.dflow INFO: Standard output for task 3778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a952f553-4488-4dbb-b39d-96c6303871c0_isrForCrosstalkSources_855455_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a952f553-4488-4dbb-b39d-96c6303871c0_isrForCrosstalkSources_855455_33.stderr parsl.dataflow.dflow DEBUG: Task 3779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3779 has memoization hash 256f70a23d06502bc0aabd0234d2a14d parsl.dataflow.memoization INFO: Task 3779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3779 try 0 launched on executor multi with executor id 3780 parsl.dataflow.dflow INFO: Standard output for task 3779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5aab5506-fe4e-48cc-aeb9-6165b6fd194f_isrForCrosstalkSources_855434_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5aab5506-fe4e-48cc-aeb9-6165b6fd194f_isrForCrosstalkSources_855434_45.stderr parsl.dataflow.dflow DEBUG: Task 3780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3780 has memoization hash a5f37b16c77515de5fa35488bac0bf2a parsl.dataflow.memoization INFO: Task 3780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3780 try 0 launched on executor multi with executor id 3781 parsl.dataflow.dflow INFO: Standard output for task 3780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/30a69108-c795-4c15-ac81-88c4d7887f23_isrForCrosstalkSources_855406_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/30a69108-c795-4c15-ac81-88c4d7887f23_isrForCrosstalkSources_855406_51.stderr parsl.dataflow.dflow DEBUG: Task 3781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3781 has memoization hash 50a91dd64a23930f8ce1be0b44ff936f parsl.dataflow.memoization INFO: Task 3781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d622a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3781 try 0 launched on executor multi with executor id 3782 parsl.dataflow.dflow INFO: Standard output for task 3781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/1c5a289b-9930-4005-a318-33039e735fab_isrForCrosstalkSources_855386_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/1c5a289b-9930-4005-a318-33039e735fab_isrForCrosstalkSources_855386_27.stderr parsl.dataflow.dflow DEBUG: Task 3782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3782 has memoization hash be8ccf0b6dc85abbd9535660f0d84702 parsl.dataflow.memoization INFO: Task 3782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3782 try 0 launched on executor multi with executor id 3783 parsl.dataflow.dflow INFO: Standard output for task 3782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6f28eb13-32ba-4b2c-8bff-34ed32550661_isrForCrosstalkSources_855400_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6f28eb13-32ba-4b2c-8bff-34ed32550661_isrForCrosstalkSources_855400_12.stderr parsl.dataflow.dflow DEBUG: Task 3783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3783 has memoization hash 575b79d991849d1f6c08e62505726937 parsl.dataflow.memoization INFO: Task 3783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3783 try 0 launched on executor multi with executor id 3784 parsl.dataflow.dflow INFO: Standard output for task 3783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/aae7a606-71a1-4a8f-bfc2-3295d6010b43_isrForCrosstalkSources_855428_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/aae7a606-71a1-4a8f-bfc2-3295d6010b43_isrForCrosstalkSources_855428_11.stderr parsl.dataflow.dflow DEBUG: Task 3784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3784 has memoization hash 50b556f0d28f8bd1ab33de64053619a5 parsl.dataflow.memoization INFO: Task 3784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3784 try 0 launched on executor multi with executor id 3785 parsl.dataflow.dflow INFO: Standard output for task 3784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4b22bbfb-97df-4e59-9cc5-5736aa77ed0c_isrForCrosstalkSources_855403_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4b22bbfb-97df-4e59-9cc5-5736aa77ed0c_isrForCrosstalkSources_855403_54.stderr parsl.dataflow.dflow DEBUG: Task 3785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3785 has memoization hash 57f8c31c38077216bef3d543aab3a6f1 parsl.dataflow.memoization INFO: Task 3785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3785 try 0 launched on executor multi with executor id 3786 parsl.dataflow.dflow INFO: Standard output for task 3785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/07f0fa20-5457-485c-816e-ad4f31061d11_isrForCrosstalkSources_855457_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/07f0fa20-5457-485c-816e-ad4f31061d11_isrForCrosstalkSources_855457_57.stderr parsl.dataflow.dflow DEBUG: Task 3786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3786 has memoization hash 6b86fd16817e133ce9527bca53e2b0ac parsl.dataflow.memoization INFO: Task 3786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3786 try 0 launched on executor multi with executor id 3787 parsl.dataflow.dflow INFO: Standard output for task 3786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/3a640860-b691-44dc-a624-4073f4b23904_isrForCrosstalkSources_855459_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/3a640860-b691-44dc-a624-4073f4b23904_isrForCrosstalkSources_855459_18.stderr parsl.dataflow.dflow DEBUG: Task 3787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3787 has memoization hash cd7e8a052b9dd04502989c6be111e695 parsl.dataflow.memoization INFO: Task 3787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3787 try 0 launched on executor multi with executor id 3788 parsl.dataflow.dflow INFO: Standard output for task 3787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bebd6491-40ac-4b9f-acce-9984648602cd_isrForCrosstalkSources_855427_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bebd6491-40ac-4b9f-acce-9984648602cd_isrForCrosstalkSources_855427_55.stderr parsl.dataflow.dflow DEBUG: Task 3788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3788 has memoization hash e65b163fa59be9f41af589288ec505e4 parsl.dataflow.memoization INFO: Task 3788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3788 try 0 launched on executor multi with executor id 3789 parsl.dataflow.dflow INFO: Standard output for task 3788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e76eca89-3e0d-400e-b3af-e136275b5f4d_isrForCrosstalkSources_855456_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e76eca89-3e0d-400e-b3af-e136275b5f4d_isrForCrosstalkSources_855456_60.stderr parsl.dataflow.dflow DEBUG: Task 3789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3789 has memoization hash 618b74762ad5f07d29374fe67d828245 parsl.dataflow.memoization INFO: Task 3789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3789 try 0 launched on executor multi with executor id 3790 parsl.dataflow.dflow INFO: Standard output for task 3789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bd1844c6-3d71-4a22-916f-dffec30e8ebf_isrForCrosstalkSources_855405_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bd1844c6-3d71-4a22-916f-dffec30e8ebf_isrForCrosstalkSources_855405_23.stderr parsl.dataflow.dflow DEBUG: Task 3790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3790 has memoization hash 5fe0999efd091330cb21cd9a0a8b148f parsl.dataflow.memoization INFO: Task 3790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3790 try 0 launched on executor multi with executor id 3791 parsl.dataflow.dflow INFO: Standard output for task 3790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2b4c7ed7-c700-40e5-ab7f-69054c89d8d0_isrForCrosstalkSources_855408_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2b4c7ed7-c700-40e5-ab7f-69054c89d8d0_isrForCrosstalkSources_855408_52.stderr parsl.dataflow.dflow DEBUG: Task 3791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3791 has memoization hash 26d429ea5f4a943146667e27ad0f1a48 parsl.dataflow.memoization INFO: Task 3791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3791 try 0 launched on executor multi with executor id 3792 parsl.dataflow.dflow INFO: Standard output for task 3791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/990c8d36-02f4-4de2-b6fa-04bdd7d8f0d2_isrForCrosstalkSources_855407_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/990c8d36-02f4-4de2-b6fa-04bdd7d8f0d2_isrForCrosstalkSources_855407_11.stderr parsl.dataflow.dflow DEBUG: Task 3792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3792 has memoization hash d74617ee7d7fd5512035598ed3481822 parsl.dataflow.memoization INFO: Task 3792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d619e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3792 try 0 launched on executor multi with executor id 3793 parsl.dataflow.dflow INFO: Standard output for task 3792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0819a946-de57-4596-bdd8-4155d3023363_isrForCrosstalkSources_855427_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0819a946-de57-4596-bdd8-4155d3023363_isrForCrosstalkSources_855427_48.stderr parsl.dataflow.dflow DEBUG: Task 3793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3793 has memoization hash 3ec34e7dfefb9ceaa99fade11d6800c4 parsl.dataflow.memoization INFO: Task 3793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3793 try 0 launched on executor multi with executor id 3794 parsl.dataflow.dflow INFO: Standard output for task 3793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ce39d21-6640-40ac-b73e-d98592f25aab_isrForCrosstalkSources_855383_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ce39d21-6640-40ac-b73e-d98592f25aab_isrForCrosstalkSources_855383_61.stderr parsl.dataflow.dflow DEBUG: Task 3794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3794 has memoization hash 9a1f35403f47ac6151390008fa7d8f36 parsl.dataflow.memoization INFO: Task 3794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3794 try 0 launched on executor multi with executor id 3795 parsl.dataflow.dflow INFO: Standard output for task 3794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/117c9559-6110-4eee-b836-7a575ca368b2_isrForCrosstalkSources_855440_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/117c9559-6110-4eee-b836-7a575ca368b2_isrForCrosstalkSources_855440_8.stderr parsl.dataflow.dflow DEBUG: Task 3795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3795 has memoization hash 03029e4cdaf260c5a2edcaafd75c6a71 parsl.dataflow.memoization INFO: Task 3795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3795 try 0 launched on executor multi with executor id 3796 parsl.dataflow.dflow INFO: Standard output for task 3795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/aba3bf78-b6f3-4a93-b48e-dfff5f420b65_isrForCrosstalkSources_855409_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/aba3bf78-b6f3-4a93-b48e-dfff5f420b65_isrForCrosstalkSources_855409_59.stderr parsl.dataflow.dflow DEBUG: Task 3796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3796 has memoization hash 6843b24ebcf7511a85e88528d6a587e3 parsl.dataflow.memoization INFO: Task 3796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d618a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c2b84d45e0ec401d2bfa336728aa19 with result from task 233 parsl.dataflow.dflow INFO: Parsl task 3796 try 0 launched on executor multi with executor id 3797 parsl.dataflow.dflow INFO: Standard output for task 3796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4872d037-545d-425c-8516-cc49679bcd37_isrForCrosstalkSources_855403_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4872d037-545d-425c-8516-cc49679bcd37_isrForCrosstalkSources_855403_34.stderr parsl.dataflow.dflow DEBUG: Task 3797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3797 has memoization hash 86d00c88276396933bf07e4218d6bbbf parsl.dataflow.memoization INFO: Task 3797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3797 try 0 launched on executor multi with executor id 3798 parsl.dataflow.dflow INFO: Standard output for task 3797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/baacdf0f-d9e5-4855-a171-1812ecb59d55_isrForCrosstalkSources_855432_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/baacdf0f-d9e5-4855-a171-1812ecb59d55_isrForCrosstalkSources_855432_49.stderr parsl.dataflow.dflow DEBUG: Task 3798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3798 has memoization hash 379a68878cb9c0d54dbb220878ec20ed parsl.dataflow.memoization INFO: Task 3798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/bf458f0a-e311-4217-9af2-8488489674e4_isrForCrosstalkSources_855446_27.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/bf458f0a-e311-4217-9af2-8488489674e4_isrForCrosstalkSources_855446_27.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b37b9e6c8bc2c0d7db2e9797663359d6 with result from task 231 parsl.dataflow.dflow INFO: Parsl task 3798 try 0 launched on executor multi with executor id 3799 parsl.dataflow.dflow INFO: Standard output for task 3798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4c18d2ab-1d33-4b07-917a-16bf3ce487e2_isrForCrosstalkSources_855388_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4c18d2ab-1d33-4b07-917a-16bf3ce487e2_isrForCrosstalkSources_855388_31.stderr parsl.dataflow.dflow DEBUG: Task 3799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3799 has memoization hash 53e96b0adbbd554b7fb77e44b763f5c6 parsl.dataflow.memoization INFO: Task 3799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9b262aa1-6f44-48e6-aace-9d238b40be64_isrForCrosstalkSources_855438_7.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9b262aa1-6f44-48e6-aace-9d238b40be64_isrForCrosstalkSources_855438_7.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d5613b4e1161d56f380e4594c9ed560 with result from task 232 parsl.dataflow.dflow INFO: Parsl task 3799 try 0 launched on executor multi with executor id 3800 parsl.dataflow.dflow INFO: Standard output for task 3799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f78e8ad2-4cc4-4f9f-bf7a-c2da06d9e304_isrForCrosstalkSources_855431_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f78e8ad2-4cc4-4f9f-bf7a-c2da06d9e304_isrForCrosstalkSources_855431_18.stderr parsl.dataflow.dflow DEBUG: Task 3800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3800 has memoization hash cfef7d89c75d2ed091b909dcbf780f94 parsl.dataflow.memoization INFO: Task 3800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3800 try 0 launched on executor multi with executor id 3801 parsl.dataflow.dflow INFO: Standard output for task 3800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/07114559-394e-46d1-bfc2-998b30af29b2_isrForCrosstalkSources_855411_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/07114559-394e-46d1-bfc2-998b30af29b2_isrForCrosstalkSources_855411_24.stderr parsl.dataflow.dflow DEBUG: Task 3801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3801 has memoization hash 0f7b09b319f1911b03d778459aa22b49 parsl.dataflow.memoization INFO: Task 3801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3801 try 0 launched on executor multi with executor id 3802 parsl.dataflow.dflow INFO: Standard output for task 3801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e283ec3a-5306-483e-b9f9-282f664c9e23_isrForCrosstalkSources_855432_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e283ec3a-5306-483e-b9f9-282f664c9e23_isrForCrosstalkSources_855432_13.stderr parsl.dataflow.dflow DEBUG: Task 3802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3802 has memoization hash 4f176934ea529f0f454b7e5a61cace74 parsl.dataflow.memoization INFO: Task 3802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d613a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c2437ef3-aeba-4f52-9a69-771c2cd98b06_isrForCrosstalkSources_855451_30.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c2437ef3-aeba-4f52-9a69-771c2cd98b06_isrForCrosstalkSources_855451_30.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 098c0e3f9266482d4ade78e064a9a06b with result from task 227 2024-03-09 14:51:40 proc_lsst.multi:170 [INFO] [multi] job_id 16964174 proc_lsst.multi INFO: [multi] job_id 16964174 2024-03-09 14:51:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-09 14:51:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964174 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964174 parsl.executors.status_handling DEBUG: Launched block 148 on executor multi with job ID 16964174 parsl.executors.status_handling INFO: Allocated block ID 149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-149 2024-03-09 14:51:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-09 14:51:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3802 try 0 launched on executor multi with executor id 3803 parsl.dataflow.dflow INFO: Standard output for task 3802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3a7851cd-8928-4942-9a38-4f6700939632_isrForCrosstalkSources_855394_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3a7851cd-8928-4942-9a38-4f6700939632_isrForCrosstalkSources_855394_51.stderr parsl.dataflow.dflow DEBUG: Task 3803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3803 has memoization hash 311b45230269869cf404c08ed410e901 parsl.dataflow.memoization INFO: Task 3803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3803 try 0 launched on executor multi with executor id 3804 parsl.dataflow.dflow INFO: Standard output for task 3803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/2e00787b-1950-4275-a669-3e90a156cc9f_isrForCrosstalkSources_855383_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/2e00787b-1950-4275-a669-3e90a156cc9f_isrForCrosstalkSources_855383_52.stderr parsl.dataflow.dflow DEBUG: Task 3804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3804 has memoization hash 1b0bd970a8314224c2e18677f85d76cf parsl.dataflow.memoization INFO: Task 3804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/78b13cb2-68d9-4e1a-a1f2-372ef14794ab_isrForCrosstalkSources_855413_14.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/78b13cb2-68d9-4e1a-a1f2-372ef14794ab_isrForCrosstalkSources_855413_14.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e1df0269a1ca43b679091c408c0100 with result from task 230 parsl.dataflow.dflow INFO: Parsl task 3804 try 0 launched on executor multi with executor id 3805 parsl.dataflow.dflow INFO: Standard output for task 3804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e62ce6ff-1418-4c3e-a1f6-8c4b5b3efe2d_isrForCrosstalkSources_855444_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e62ce6ff-1418-4c3e-a1f6-8c4b5b3efe2d_isrForCrosstalkSources_855444_2.stderr parsl.dataflow.dflow DEBUG: Task 3805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3805 has memoization hash 10c48bedacfa66f2cf682f0b9de02b9f parsl.dataflow.memoization INFO: Task 3805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3805 try 0 launched on executor multi with executor id 3806 parsl.dataflow.dflow INFO: Standard output for task 3805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b930ed94-cf4a-42ff-af08-564080c44837_isrForCrosstalkSources_855427_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b930ed94-cf4a-42ff-af08-564080c44837_isrForCrosstalkSources_855427_17.stderr parsl.dataflow.dflow DEBUG: Task 3806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3806 has memoization hash a03252983ab240d44fd85e5d1706b535 parsl.dataflow.memoization INFO: Task 3806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d614e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54b634f9-a38e-4e15-a65f-77dd5e2f94f5_isrForCrosstalkSources_855382_7.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54b634f9-a38e-4e15-a65f-77dd5e2f94f5_isrForCrosstalkSources_855382_7.stderr parsl.dataflow.dflow INFO: Parsl task 3806 try 0 launched on executor multi with executor id 3807 parsl.dataflow.dflow INFO: Standard output for task 3806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e923f0e8-97df-4366-82b5-94043665c4f5_isrForCrosstalkSources_855431_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e923f0e8-97df-4366-82b5-94043665c4f5_isrForCrosstalkSources_855431_37.stderr parsl.dataflow.dflow DEBUG: Task 3807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3807 has memoization hash 5609c2eda301da29abde8ba142e889ad parsl.dataflow.memoization INFO: Task 3807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d611c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7076bb34fd02334e17dfcae22f65f1f with result from task 520 parsl.dataflow.dflow INFO: Parsl task 3807 try 0 launched on executor multi with executor id 3808 parsl.dataflow.dflow INFO: Standard output for task 3807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a9f8169d-9e52-40db-95b6-007b700ed966_isrForCrosstalkSources_855401_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a9f8169d-9e52-40db-95b6-007b700ed966_isrForCrosstalkSources_855401_46.stderr parsl.dataflow.dflow DEBUG: Task 3808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3808 has memoization hash 6eb38a0e900c778c89026ac1486b0c3d parsl.dataflow.memoization INFO: Task 3808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ea43d1f7-72ab-4d9d-82d2-7687178c9fb3_isrForCrosstalkSources_855430_36.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ea43d1f7-72ab-4d9d-82d2-7687178c9fb3_isrForCrosstalkSources_855430_36.stderr parsl.dataflow.dflow INFO: Parsl task 3808 try 0 launched on executor multi with executor id 3809 parsl.dataflow.dflow INFO: Standard output for task 3808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9fdffaab-1b15-41f9-a627-ac4f59d8d086_isrForCrosstalkSources_855393_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9fdffaab-1b15-41f9-a627-ac4f59d8d086_isrForCrosstalkSources_855393_19.stderr parsl.dataflow.dflow DEBUG: Task 3809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3809 has memoization hash 4d424dc49a72931c0a53a06cd71d76cf parsl.dataflow.memoization INFO: Task 3809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3809 try 0 launched on executor multi with executor id 3810 parsl.dataflow.dflow INFO: Standard output for task 3809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b29adea0-8dbe-4282-9806-486e2e364c3e_isrForCrosstalkSources_855456_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b29adea0-8dbe-4282-9806-486e2e364c3e_isrForCrosstalkSources_855456_57.stderr parsl.dataflow.dflow DEBUG: Task 3810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3810 has memoization hash 97e0b37b0ddbe5d0ed366df77aa29bea parsl.dataflow.memoization INFO: Task 3810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d61120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3810 try 0 launched on executor multi with executor id 3811 parsl.dataflow.dflow INFO: Standard output for task 3810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8e1c73f4-cd8b-4b4f-b5d0-b636c1bf85e9_isrForCrosstalkSources_855381_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8e1c73f4-cd8b-4b4f-b5d0-b636c1bf85e9_isrForCrosstalkSources_855381_12.stderr parsl.dataflow.dflow DEBUG: Task 3811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3811 has memoization hash 86744de8acc3b2fa5f229b1c657419a1 parsl.dataflow.memoization INFO: Task 3811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3811 try 0 launched on executor multi with executor id 3812 parsl.dataflow.dflow INFO: Standard output for task 3811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8820a789-a76e-4d3b-9cee-fb4321b37691_isrForCrosstalkSources_855441_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8820a789-a76e-4d3b-9cee-fb4321b37691_isrForCrosstalkSources_855441_56.stderr parsl.dataflow.dflow DEBUG: Task 3812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3812 has memoization hash 9ce27384ae9c318ae0aa8530b6c7cf79 parsl.dataflow.memoization INFO: Task 3812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2acc64a343856de7a7fd4bd4936912 with result from task 229 parsl.dataflow.dflow INFO: Parsl task 3812 try 0 launched on executor multi with executor id 3813 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Standard output for task 3812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8464ad49-f405-4e87-8f54-42f2d8ed6c39_isrForCrosstalkSources_855459_33.stdout parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard error for task 3812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8464ad49-f405-4e87-8f54-42f2d8ed6c39_isrForCrosstalkSources_855459_33.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/39b2a43d-b75b-42b2-b1c7-dcef2cfdf011_isrForCrosstalkSources_855379_34.stdout parsl.dataflow.dflow DEBUG: Task 3813 will be sent to executor multi parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/39b2a43d-b75b-42b2-b1c7-dcef2cfdf011_isrForCrosstalkSources_855379_34.stderr parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3813 has memoization hash f863efc6c47143923cd81962d08caa12 parsl.dataflow.memoization INFO: Task 3813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d627a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e3be51ee76089fc7327595e77cf5e9c with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e10b9409-f8a8-4f51-b0d7-85e0b111923a_isrForCrosstalkSources_855411_18.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e10b9409-f8a8-4f51-b0d7-85e0b111923a_isrForCrosstalkSources_855411_18.stderr parsl.dataflow.dflow INFO: Parsl task 3813 try 0 launched on executor multi with executor id 3814 parsl.dataflow.dflow INFO: Standard output for task 3813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/47dbfa3d-0b3f-42e5-a845-4a36b9634961_isrForCrosstalkSources_855454_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/47dbfa3d-0b3f-42e5-a845-4a36b9634961_isrForCrosstalkSources_855454_46.stderr parsl.dataflow.dflow DEBUG: Task 3814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3814 has memoization hash 24987f2215b2b02ff8221d83f141c576 parsl.dataflow.memoization INFO: Task 3814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3814 try 0 launched on executor multi with executor id 3815 parsl.dataflow.dflow INFO: Standard output for task 3814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/36dbcbc0-5c49-417e-b8d7-30976f3dd49f_isrForCrosstalkSources_855433_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/36dbcbc0-5c49-417e-b8d7-30976f3dd49f_isrForCrosstalkSources_855433_14.stderr parsl.dataflow.dflow DEBUG: Task 3815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3815 has memoization hash ca76ee64b542beb08d24d99ebb005915 parsl.dataflow.memoization INFO: Task 3815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ca9f88f7ab0b54a5c92602d2edb2d23 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/22f075d6-b8e0-4d48-ae8c-b8f16d2222b1_isrForCrosstalkSources_855456_12.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/22f075d6-b8e0-4d48-ae8c-b8f16d2222b1_isrForCrosstalkSources_855456_12.stderr parsl.dataflow.dflow INFO: Parsl task 3815 try 0 launched on executor multi with executor id 3816 parsl.dataflow.dflow INFO: Standard output for task 3815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf370c1b-c292-4f48-aa6d-9456af94e630_isrForCrosstalkSources_855432_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf370c1b-c292-4f48-aa6d-9456af94e630_isrForCrosstalkSources_855432_34.stderr parsl.dataflow.dflow DEBUG: Task 3816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3816 has memoization hash 411377895562c073c4596d07f399c6a6 parsl.dataflow.memoization INFO: Task 3816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d628e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3816 try 0 launched on executor multi with executor id 3817 parsl.dataflow.dflow INFO: Standard output for task 3816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20eb1eec-5638-4781-af19-aa2038dba493_isrForCrosstalkSources_855418_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20eb1eec-5638-4781-af19-aa2038dba493_isrForCrosstalkSources_855418_50.stderr parsl.dataflow.dflow DEBUG: Task 3817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3817 has memoization hash eee3849e858497b46139274fba5d5ff0 parsl.dataflow.memoization INFO: Task 3817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3817 try 0 launched on executor multi with executor id 3818 parsl.dataflow.dflow INFO: Standard output for task 3817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ebd4b793-2eea-4270-8543-f5119d83ccf9_isrForCrosstalkSources_855381_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ebd4b793-2eea-4270-8543-f5119d83ccf9_isrForCrosstalkSources_855381_25.stderr parsl.dataflow.dflow DEBUG: Task 3818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3818 has memoization hash f256dcd0b2e774485886d893e72bb739 parsl.dataflow.memoization INFO: Task 3818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3818 try 0 launched on executor multi with executor id 3819 parsl.dataflow.dflow INFO: Standard output for task 3818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e609f697-9c6d-4ffb-afc6-ac044324e0aa_isrForCrosstalkSources_855413_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e609f697-9c6d-4ffb-afc6-ac044324e0aa_isrForCrosstalkSources_855413_21.stderr parsl.dataflow.dflow DEBUG: Task 3819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3819 has memoization hash 7605fd66cc96c5d42b83fe61c714155d parsl.dataflow.memoization INFO: Task 3819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3819 try 0 launched on executor multi with executor id 3820 parsl.dataflow.dflow INFO: Standard output for task 3819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ff2ecbd1-79b5-4919-9f0e-cef076aa97b5_isrForCrosstalkSources_855440_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ff2ecbd1-79b5-4919-9f0e-cef076aa97b5_isrForCrosstalkSources_855440_15.stderr parsl.dataflow.dflow DEBUG: Task 3820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3820 has memoization hash e339d8a636ebe1ec6f4dcae35e8c6cad parsl.dataflow.memoization INFO: Task 3820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b39b4a91c24709c64d38d1f3ed35a0 with result from task 159 parsl.dataflow.dflow INFO: Parsl task 3820 try 0 launched on executor multi with executor id 3821 parsl.dataflow.dflow INFO: Standard output for task 3820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/6c9840a7-9939-4dda-ba02-96def72af1dc_isrForCrosstalkSources_855430_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/6c9840a7-9939-4dda-ba02-96def72af1dc_isrForCrosstalkSources_855430_51.stderr parsl.dataflow.dflow DEBUG: Task 3821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3821 has memoization hash 8192d8ebbfb23601dd77b8de75bd5505 parsl.dataflow.memoization INFO: Task 3821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3821 try 0 launched on executor multi with executor id 3822 parsl.dataflow.dflow INFO: Standard output for task 3821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b714aea0-5899-4c5d-a664-bcedf0f37dc5_isrForCrosstalkSources_855453_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b714aea0-5899-4c5d-a664-bcedf0f37dc5_isrForCrosstalkSources_855453_52.stderr parsl.dataflow.dflow DEBUG: Task 3822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3822 has memoization hash b5da492c30873ee35b87319d1bdb1760 parsl.dataflow.memoization INFO: Task 3822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3822 try 0 launched on executor multi with executor id 3823 parsl.dataflow.dflow INFO: Standard output for task 3822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1208293f-d019-4d7d-9d4b-ae3b888274f1_isrForCrosstalkSources_855427_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1208293f-d019-4d7d-9d4b-ae3b888274f1_isrForCrosstalkSources_855427_19.stderr parsl.dataflow.dflow DEBUG: Task 3823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3823 has memoization hash 901c878dad4863635590f4bca11661ca parsl.dataflow.memoization INFO: Task 3823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d639c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3823 try 0 launched on executor multi with executor id 3824 parsl.dataflow.dflow INFO: Standard output for task 3823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/960f2db5-de65-40a3-bc8c-ed80f8ecdba9_isrForCrosstalkSources_855415_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/960f2db5-de65-40a3-bc8c-ed80f8ecdba9_isrForCrosstalkSources_855415_52.stderr parsl.dataflow.dflow DEBUG: Task 3824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3824 has memoization hash fcd45c34b4ae53c7ffa09735a829ae3a parsl.dataflow.memoization INFO: Task 3824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d637e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3ef22b8e-088e-498c-b2c8-fe7e293bb78d_isrForCrosstalkSources_855404_35.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3ef22b8e-088e-498c-b2c8-fe7e293bb78d_isrForCrosstalkSources_855404_35.stderr parsl.dataflow.dflow INFO: Parsl task 3824 try 0 launched on executor multi with executor id 3825 parsl.dataflow.dflow INFO: Standard output for task 3824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/88a3c085-308d-4f9d-b67f-b6120e96e3ec_isrForCrosstalkSources_855420_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/88a3c085-308d-4f9d-b67f-b6120e96e3ec_isrForCrosstalkSources_855420_16.stderr parsl.dataflow.dflow DEBUG: Task 3825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3825 has memoization hash 373ed84cfff436a6268f583518d74b8e parsl.dataflow.memoization INFO: Task 3825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d0a7691c8498685d7a12fb4634bff34 with result from task 234 parsl.dataflow.dflow INFO: Parsl task 3825 try 0 launched on executor multi with executor id 3826 parsl.dataflow.dflow INFO: Standard output for task 3825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/36652692-738e-40bb-ae44-13897b6b5b20_isrForCrosstalkSources_855437_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/36652692-738e-40bb-ae44-13897b6b5b20_isrForCrosstalkSources_855437_44.stderr parsl.dataflow.dflow DEBUG: Task 3826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3826 has memoization hash a92cfc811edf01676da2dda3bc70b396 parsl.dataflow.memoization INFO: Task 3826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3826 try 0 launched on executor multi with executor id 3827 parsl.dataflow.dflow INFO: Standard output for task 3826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ed0cb72d-ccec-4dd5-8338-09253664f229_isrForCrosstalkSources_855403_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ed0cb72d-ccec-4dd5-8338-09253664f229_isrForCrosstalkSources_855403_36.stderr parsl.dataflow.dflow DEBUG: Task 3827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3827 has memoization hash 5a8cc7e6dff7200707f541b6d2e4bfd4 parsl.dataflow.memoization INFO: Task 3827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3827 try 0 launched on executor multi with executor id 3828 parsl.dataflow.dflow INFO: Standard output for task 3827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c17bffe7-b847-4100-b555-7d75941258de_isrForCrosstalkSources_855414_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c17bffe7-b847-4100-b555-7d75941258de_isrForCrosstalkSources_855414_58.stderr parsl.dataflow.dflow DEBUG: Task 3828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3828 has memoization hash 42f4be8c8d698ae1a489ed28ebc1ee62 parsl.dataflow.memoization INFO: Task 3828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d634c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7b13b6a5-521f-45b4-b241-225cc3775812_isrForCrosstalkSources_855406_5.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7b13b6a5-521f-45b4-b241-225cc3775812_isrForCrosstalkSources_855406_5.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b1d4448e2daf055579b5c674debe23 with result from task 240 parsl.dataflow.dflow INFO: Parsl task 3828 try 0 launched on executor multi with executor id 3829 parsl.dataflow.dflow INFO: Standard output for task 3828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/dbbf3121-53e4-411f-b4b9-4378f0ddd7e6_isrForCrosstalkSources_855380_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/dbbf3121-53e4-411f-b4b9-4378f0ddd7e6_isrForCrosstalkSources_855380_45.stderr parsl.dataflow.dflow DEBUG: Task 3829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3829 has memoization hash 8b3b914d5d8f366a42e2eebc93954684 parsl.dataflow.memoization INFO: Task 3829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3829 try 0 launched on executor multi with executor id 3830 parsl.dataflow.dflow INFO: Standard output for task 3829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/09350d9d-37d0-461a-b799-44ddd6e2b05d_isrForCrosstalkSources_855447_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/09350d9d-37d0-461a-b799-44ddd6e2b05d_isrForCrosstalkSources_855447_4.stderr parsl.dataflow.dflow DEBUG: Task 3830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3830 has memoization hash 6dbe7a156e43f84a4fae4f13f0ace8b5 parsl.dataflow.memoization INFO: Task 3830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3830 try 0 launched on executor multi with executor id 3831 parsl.dataflow.dflow INFO: Standard output for task 3830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/34116808-5bb7-4637-bad6-15e92009af6c_isrForCrosstalkSources_855454_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/34116808-5bb7-4637-bad6-15e92009af6c_isrForCrosstalkSources_855454_55.stderr parsl.dataflow.dflow DEBUG: Task 3831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3831 has memoization hash e18b3bda6978031c2ea449d8c2a07198 parsl.dataflow.memoization INFO: Task 3831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d632e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e7b77c0f-afc6-43fb-b7c6-cce6da95e5f2_isrForCrosstalkSources_855441_16.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e7b77c0f-afc6-43fb-b7c6-cce6da95e5f2_isrForCrosstalkSources_855441_16.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5abeeb953dee93c0f5dd2e6905b7864a with result from task 235 parsl.dataflow.dflow INFO: Parsl task 3831 try 0 launched on executor multi with executor id 3832 parsl.dataflow.dflow INFO: Standard output for task 3831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/49473dc9-3190-4d54-8ca2-c45bb54f7681_isrForCrosstalkSources_855410_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/49473dc9-3190-4d54-8ca2-c45bb54f7681_isrForCrosstalkSources_855410_13.stderr parsl.dataflow.dflow DEBUG: Task 3832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3832 has memoization hash 4f168c71766e97e7b86a9876cb6c2ac5 parsl.dataflow.memoization INFO: Task 3832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3832 try 0 launched on executor multi with executor id 3833 parsl.dataflow.dflow INFO: Standard output for task 3832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/01ee78d2-3b72-4676-9ca8-86f8eb0dad85_isrForCrosstalkSources_855404_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/01ee78d2-3b72-4676-9ca8-86f8eb0dad85_isrForCrosstalkSources_855404_17.stderr parsl.dataflow.dflow DEBUG: Task 3833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3833 has memoization hash 66da9028cc6859f6c46d538df029112d parsl.dataflow.memoization INFO: Task 3833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3833 try 0 launched on executor multi with executor id 3834 parsl.dataflow.dflow INFO: Standard output for task 3833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ef72c06f-44cc-46b4-8942-333f84e32b6a_isrForCrosstalkSources_855390_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ef72c06f-44cc-46b4-8942-333f84e32b6a_isrForCrosstalkSources_855390_38.stderr parsl.dataflow.dflow DEBUG: Task 3834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3834 has memoization hash f861dbc54a9c9768bb31d0911e896044 parsl.dataflow.memoization INFO: Task 3834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3834 try 0 launched on executor multi with executor id 3835 parsl.dataflow.dflow INFO: Standard output for task 3834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/8fc7c475-e057-4d59-93e2-4ffe8c10f65d_isrForCrosstalkSources_855424_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/8fc7c475-e057-4d59-93e2-4ffe8c10f65d_isrForCrosstalkSources_855424_57.stderr parsl.dataflow.dflow DEBUG: Task 3835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3835 has memoization hash 167ceaca2180f7da608b763758342034 parsl.dataflow.memoization INFO: Task 3835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d1f1779d-476d-4329-ab8f-30d03ba56c6a_isrForCrosstalkSources_855421_18.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d1f1779d-476d-4329-ab8f-30d03ba56c6a_isrForCrosstalkSources_855421_18.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba3af1482ce52f70ca124c254d4fa8d with result from task 238 parsl.dataflow.dflow INFO: Parsl task 3835 try 0 launched on executor multi with executor id 3836 parsl.dataflow.dflow INFO: Standard output for task 3835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fddd5fe9-ba87-4aed-81a7-735e201909a3_isrForCrosstalkSources_855430_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fddd5fe9-ba87-4aed-81a7-735e201909a3_isrForCrosstalkSources_855430_56.stderr parsl.dataflow.dflow DEBUG: Task 3836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3836 has memoization hash df7fdd0a18c6027dcdaa38d4e04a0c64 parsl.dataflow.memoization INFO: Task 3836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3836 try 0 launched on executor multi with executor id 3837 parsl.dataflow.dflow INFO: Standard output for task 3836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5aee3072-2901-4329-b832-cec8a30747ea_isrForCrosstalkSources_855436_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5aee3072-2901-4329-b832-cec8a30747ea_isrForCrosstalkSources_855436_56.stderr parsl.dataflow.dflow DEBUG: Task 3837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3837 has memoization hash e10ca46770b0388c532cbcec171d8e39 parsl.dataflow.memoization INFO: Task 3837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3837 try 0 launched on executor multi with executor id 3838 parsl.dataflow.dflow INFO: Standard output for task 3837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f44f06bf-54bb-4591-ac43-d5ab8585b48c_isrForCrosstalkSources_855379_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f44f06bf-54bb-4591-ac43-d5ab8585b48c_isrForCrosstalkSources_855379_40.stderr parsl.dataflow.dflow DEBUG: Task 3838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3838 has memoization hash a362e06a53cc9bde2b3425c8d595aa77 parsl.dataflow.memoization INFO: Task 3838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d631a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3838 try 0 launched on executor multi with executor id 3839 parsl.dataflow.dflow INFO: Standard output for task 3838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2a588564-a765-4c51-a437-05ed67187ecd_isrForCrosstalkSources_855455_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2a588564-a765-4c51-a437-05ed67187ecd_isrForCrosstalkSources_855455_6.stderr parsl.dataflow.dflow DEBUG: Task 3839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3839 has memoization hash 952fbfa41c17a98145cc3f9942b83c8c parsl.dataflow.memoization INFO: Task 3839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/112b75d2-a1f1-4fa7-8868-20f66a4d08ba_isrForCrosstalkSources_855434_40.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/112b75d2-a1f1-4fa7-8868-20f66a4d08ba_isrForCrosstalkSources_855434_40.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20435a79be801ed3146752bf534bec9 with result from task 237 parsl.dataflow.dflow INFO: Parsl task 3839 try 0 launched on executor multi with executor id 3840 parsl.dataflow.dflow INFO: Standard output for task 3839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ba469a5b-2f9d-41d9-962e-bce4c2b2526f_isrForCrosstalkSources_855379_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ba469a5b-2f9d-41d9-962e-bce4c2b2526f_isrForCrosstalkSources_855379_13.stderr parsl.dataflow.dflow DEBUG: Task 3840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3840 has memoization hash ea525f878b3d1657140beec0f95666b8 parsl.dataflow.memoization INFO: Task 3840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f1956a3b-a300-449f-8f9e-c3e073c2e0c2_isrForCrosstalkSources_855432_60.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f1956a3b-a300-449f-8f9e-c3e073c2e0c2_isrForCrosstalkSources_855432_60.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b34a5ff145382a05fb51f2ff25c62e0 with result from task 239 parsl.dataflow.dflow INFO: Parsl task 3840 try 0 launched on executor multi with executor id 3841 parsl.dataflow.dflow INFO: Standard output for task 3840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a00ae263-0f39-4371-9ab1-58d2289cc5c6_isrForCrosstalkSources_855411_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a00ae263-0f39-4371-9ab1-58d2289cc5c6_isrForCrosstalkSources_855411_43.stderr parsl.dataflow.dflow DEBUG: Task 3841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3841 has memoization hash 9f24148e1f35ac59f680a9c033ca6573 parsl.dataflow.memoization INFO: Task 3841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3841 try 0 launched on executor multi with executor id 3842 parsl.dataflow.dflow INFO: Standard output for task 3841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b65b25af-fe5e-42d0-9be5-16c11c42cad8_isrForCrosstalkSources_855389_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b65b25af-fe5e-42d0-9be5-16c11c42cad8_isrForCrosstalkSources_855389_37.stderr parsl.dataflow.dflow DEBUG: Task 3842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3842 has memoization hash aa704e41ea1001ffc2bccd25f56fb608 parsl.dataflow.memoization INFO: Task 3842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d63060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/df839e70-fd29-4534-ba66-202c996f5c1e_isrForCrosstalkSources_855383_20.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/df839e70-fd29-4534-ba66-202c996f5c1e_isrForCrosstalkSources_855383_20.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065453a78a84e737f38557709c8aabc2 with result from task 219 2024-03-09 14:51:41 proc_lsst.multi:170 [INFO] [multi] job_id 16964175 proc_lsst.multi INFO: [multi] job_id 16964175 2024-03-09 14:51:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-09 14:51:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964175 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964175 parsl.executors.status_handling DEBUG: Launched block 149 on executor multi with job ID 16964175 parsl.executors.status_handling INFO: Allocated block ID 150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-150 2024-03-09 14:51:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3842 try 0 launched on executor multi with executor id 3843 parsl.dataflow.dflow INFO: Standard output for task 3842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e96774b4-c4fe-4e81-9e0d-81405906ed48_isrForCrosstalkSources_855420_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e96774b4-c4fe-4e81-9e0d-81405906ed48_isrForCrosstalkSources_855420_51.stderr parsl.dataflow.dflow DEBUG: Task 3843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3843 has memoization hash a83cc91075b1dde6b2f1708a1e26a485 parsl.dataflow.memoization INFO: Task 3843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4fefc08-b460-4945-bfa9-c01d6c68bb1f_isrForCrosstalkSources_855393_58.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4fefc08-b460-4945-bfa9-c01d6c68bb1f_isrForCrosstalkSources_855393_58.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48aa5da189eef77b9cce2742efb32687 with result from task 223 parsl.dataflow.dflow INFO: Parsl task 3843 try 0 launched on executor multi with executor id 3844 parsl.dataflow.dflow INFO: Standard output for task 3843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0320d206-926c-40ff-8f57-17585b21d9a0_isrForCrosstalkSources_855387_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0320d206-926c-40ff-8f57-17585b21d9a0_isrForCrosstalkSources_855387_42.stderr parsl.dataflow.dflow DEBUG: Task 3844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3844 has memoization hash ed3aef0f498472e76fa409d6b304ed95 parsl.dataflow.memoization INFO: Task 3844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3844 try 0 launched on executor multi with executor id 3845 parsl.dataflow.dflow INFO: Standard output for task 3844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f9e3e46d-1a7a-44c2-8c9d-e4cb91153b51_isrForCrosstalkSources_855401_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f9e3e46d-1a7a-44c2-8c9d-e4cb91153b51_isrForCrosstalkSources_855401_17.stderr parsl.dataflow.dflow DEBUG: Task 3845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3845 has memoization hash 4e47531fc217d0f4e9dafed917d73ab0 parsl.dataflow.memoization INFO: Task 3845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d616c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a94cf063-0889-4910-8909-457d2a64bcd8_isrForCrosstalkSources_855449_16.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a94cf063-0889-4910-8909-457d2a64bcd8_isrForCrosstalkSources_855449_16.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e64baac1fc2872204e720985c98b35a2 with result from task 222 parsl.dataflow.dflow INFO: Parsl task 3845 try 0 launched on executor multi with executor id 3846 parsl.dataflow.dflow INFO: Standard output for task 3845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbcf1dbd-f46b-4742-b5d0-b89a7ece2764_isrForCrosstalkSources_855431_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbcf1dbd-f46b-4742-b5d0-b89a7ece2764_isrForCrosstalkSources_855431_6.stderr parsl.dataflow.dflow DEBUG: Task 3846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3846 has memoization hash fbd94534318c8f28655731abb14a24df parsl.dataflow.memoization INFO: Task 3846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e6102aad-580a-4694-b6a2-b11f38dc50a5_isrForCrosstalkSources_855420_29.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e6102aad-580a-4694-b6a2-b11f38dc50a5_isrForCrosstalkSources_855420_29.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e2f61b2b17fadcfd5e673829c53a8bb with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/691b38e4-f919-462c-a475-ad0c0cb9d40d_isrForCrosstalkSources_855379_20.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/691b38e4-f919-462c-a475-ad0c0cb9d40d_isrForCrosstalkSources_855379_20.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac9af901db7a5f2f0dcf789a680f29a with result from task 236 parsl.dataflow.dflow INFO: Parsl task 3846 try 0 launched on executor multi with executor id 3847 parsl.dataflow.dflow INFO: Standard output for task 3846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/368457fa-8ba9-4abb-94cf-2a06a7c20bcd_isrForCrosstalkSources_855451_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/368457fa-8ba9-4abb-94cf-2a06a7c20bcd_isrForCrosstalkSources_855451_41.stderr parsl.dataflow.dflow DEBUG: Task 3847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3847 has memoization hash e689675670068cb8413142b78cebdffc parsl.dataflow.memoization INFO: Task 3847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3847 try 0 launched on executor multi with executor id 3848 parsl.dataflow.dflow INFO: Standard output for task 3847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/21facd6d-4ba2-46ca-9b31-9cdca7a62b5b_isrForCrosstalkSources_855436_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/21facd6d-4ba2-46ca-9b31-9cdca7a62b5b_isrForCrosstalkSources_855436_42.stderr parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/2ea068c1-b746-4086-a578-ba88d8df01b4_isrForCrosstalkSources_855385_2.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/2ea068c1-b746-4086-a578-ba88d8df01b4_isrForCrosstalkSources_855385_2.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b8912bad0ab61ece3c3c8ea13e0bce with result from task 218 parsl.dataflow.dflow DEBUG: Task 3848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3848 has memoization hash d8553b5ddaa0eae0f378b5640de34634 parsl.dataflow.memoization INFO: Task 3848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3848 try 0 launched on executor multi with executor id 3849 parsl.dataflow.dflow INFO: Standard output for task 3848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a1e38164-97d9-43ab-b233-953365124433_isrForCrosstalkSources_855448_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a1e38164-97d9-43ab-b233-953365124433_isrForCrosstalkSources_855448_9.stderr parsl.dataflow.dflow DEBUG: Task 3849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3849 has memoization hash 0681203de59dcf28c75807e58b10aa83 parsl.dataflow.memoization INFO: Task 3849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d60720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3849 try 0 launched on executor multi with executor id 3850 parsl.dataflow.dflow INFO: Standard output for task 3849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/acaa8651-5041-4a5a-b4c6-532be4f40ebd_isrForCrosstalkSources_855384_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/acaa8651-5041-4a5a-b4c6-532be4f40ebd_isrForCrosstalkSources_855384_7.stderr parsl.dataflow.dflow DEBUG: Task 3850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3850 has memoization hash 630e9ad84a78a83b4e8c5f2673a2e179 parsl.dataflow.memoization INFO: Task 3850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ced4ed47-012e-4925-b944-db17aea0bc17_isrForCrosstalkSources_855379_57.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ced4ed47-012e-4925-b944-db17aea0bc17_isrForCrosstalkSources_855379_57.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d4acb67df0d6b379f60648485534a7 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a423c639-7cd3-4539-98ff-49bc26b55b8e_isrForCrosstalkSources_855384_18.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a423c639-7cd3-4539-98ff-49bc26b55b8e_isrForCrosstalkSources_855384_18.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52df9e1db34664e5d5f50e70b1e5698f with result from task 241 parsl.dataflow.dflow INFO: Parsl task 3850 try 0 launched on executor multi with executor id 3851 parsl.dataflow.dflow INFO: Standard output for task 3850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1550e032-8527-45b6-b5a2-19c905163406_isrForCrosstalkSources_855392_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1550e032-8527-45b6-b5a2-19c905163406_isrForCrosstalkSources_855392_34.stderr parsl.dataflow.dflow DEBUG: Task 3851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Task 3851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b0cb3b7c-3f71-47d7-992a-cb74d7914727_isrForCrosstalkSources_855414_61.stdout parsl.dataflow.dflow DEBUG: Task 3851 set to pending state with AppFuture: parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b0cb3b7c-3f71-47d7-992a-cb74d7914727_isrForCrosstalkSources_855414_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Storing app cache entry 4337bccadc0c6f92fa542248f1c8973b with result from task 220 parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3851 has memoization hash d88cf456a4d5406fe02ea20815977137 parsl.dataflow.memoization INFO: Task 3851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e65d006-6493-4183-9478-ca0f24607960_isrForCrosstalkSources_855408_43.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e65d006-6493-4183-9478-ca0f24607960_isrForCrosstalkSources_855408_43.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b08b625ab1ef73d2a184016dbfe1849 with result from task 224 parsl.dataflow.dflow INFO: Parsl task 3851 try 0 launched on executor multi with executor id 3852 parsl.dataflow.dflow INFO: Standard output for task 3851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d924536b-caa3-412b-ade6-e79348ad582f_isrForCrosstalkSources_855422_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d924536b-caa3-412b-ade6-e79348ad582f_isrForCrosstalkSources_855422_24.stderr parsl.dataflow.dflow DEBUG: Task 3852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3852 has memoization hash 3277591ba1b5ca5b65452526bbf8dfd2 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.memoization INFO: Task 3852 had no result in cache parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c182c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/2f5130f2-e7c3-4ce3-b456-d267b1511e61_isrForCrosstalkSources_855435_18.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/2f5130f2-e7c3-4ce3-b456-d267b1511e61_isrForCrosstalkSources_855435_18.stderr parsl.dataflow.dflow INFO: Parsl task 3852 try 0 launched on executor multi with executor id 3853 parsl.dataflow.dflow INFO: Standard output for task 3852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/8d610b44-6a6f-4dc2-ae76-a419a8eef38d_isrForCrosstalkSources_855413_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/8d610b44-6a6f-4dc2-ae76-a419a8eef38d_isrForCrosstalkSources_855413_15.stderr parsl.dataflow.dflow DEBUG: Task 3853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3853 has memoization hash c0af29039568694a0036a3533c30f398 parsl.dataflow.memoization INFO: Task 3853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3853 try 0 launched on executor multi with executor id 3854 parsl.dataflow.dflow INFO: Standard output for task 3853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/bfb54fda-f97c-44c0-880f-d2d9d57abb9c_isrForCrosstalkSources_855447_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/bfb54fda-f97c-44c0-880f-d2d9d57abb9c_isrForCrosstalkSources_855447_22.stderr parsl.dataflow.dflow DEBUG: Task 3854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3854 has memoization hash 93d0b7c7dd662711e009d4c6d5898cc8 parsl.dataflow.memoization INFO: Task 3854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c180e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3854 try 0 launched on executor multi with executor id 3855 parsl.dataflow.dflow INFO: Standard output for task 3854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b8c49923-e362-49f2-8f68-f61af9e89214_isrForCrosstalkSources_855393_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b8c49923-e362-49f2-8f68-f61af9e89214_isrForCrosstalkSources_855393_34.stderr parsl.dataflow.dflow DEBUG: Task 3855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3855 has memoization hash e880f25d3cb371c88b17257111e9cdb1 parsl.dataflow.memoization INFO: Task 3855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3855 try 0 launched on executor multi with executor id 3856 parsl.dataflow.dflow INFO: Standard output for task 3855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9236d008-0735-459d-898e-88723bbdb4cd_isrForCrosstalkSources_855457_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9236d008-0735-459d-898e-88723bbdb4cd_isrForCrosstalkSources_855457_30.stderr parsl.dataflow.dflow DEBUG: Task 3856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3856 has memoization hash 9468e45c5e9c2fe8725af39a676d017d parsl.dataflow.memoization INFO: Task 3856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3856 try 0 launched on executor multi with executor id 3857 parsl.dataflow.dflow INFO: Standard output for task 3856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0797670f-ea2e-42fe-bfa7-c2302e7fc9eb_isrForCrosstalkSources_855435_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0797670f-ea2e-42fe-bfa7-c2302e7fc9eb_isrForCrosstalkSources_855435_49.stderr parsl.dataflow.dflow DEBUG: Task 3857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3857 has memoization hash e14a7eebbcdec4a734d4637e746f05b2 parsl.dataflow.memoization INFO: Task 3857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c184a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3857 try 0 launched on executor multi with executor id 3858 parsl.dataflow.dflow INFO: Standard output for task 3857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb070a53-0b29-4dc5-bd78-89e54b89737a_isrForCrosstalkSources_855386_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb070a53-0b29-4dc5-bd78-89e54b89737a_isrForCrosstalkSources_855386_8.stderr parsl.dataflow.dflow DEBUG: Task 3858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3858 has memoization hash 9189d207bf08622004cba5674e3716bc parsl.dataflow.memoization INFO: Task 3858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c185e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3858 try 0 launched on executor multi with executor id 3859 parsl.dataflow.dflow INFO: Standard output for task 3858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/7deb292d-3cc1-4199-b472-be5eb08e54f7_isrForCrosstalkSources_855444_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/7deb292d-3cc1-4199-b472-be5eb08e54f7_isrForCrosstalkSources_855444_44.stderr parsl.dataflow.dflow DEBUG: Task 3859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3859 has memoization hash e0aa2bfd9488e49f217378b189a9d087 parsl.dataflow.memoization INFO: Task 3859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3859 try 0 launched on executor multi with executor id 3860 parsl.dataflow.dflow INFO: Standard output for task 3859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8aa1e1a8-5863-4618-8724-90cda2593111_isrForCrosstalkSources_855440_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8aa1e1a8-5863-4618-8724-90cda2593111_isrForCrosstalkSources_855440_52.stderr parsl.dataflow.dflow DEBUG: Task 3860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3860 has memoization hash b00a89b9b89e4a4826b4a6f758c188ec parsl.dataflow.memoization INFO: Task 3860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3860 try 0 launched on executor multi with executor id 3861 parsl.dataflow.dflow INFO: Standard output for task 3860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/86637b6a-d7d0-420a-a4c6-48d810b3518a_isrForCrosstalkSources_855421_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/86637b6a-d7d0-420a-a4c6-48d810b3518a_isrForCrosstalkSources_855421_9.stderr parsl.dataflow.dflow DEBUG: Task 3861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3861 has memoization hash 1901aeea47ad7a14311ccf616734aa19 parsl.dataflow.memoization INFO: Task 3861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c187c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3861 try 0 launched on executor multi with executor id 3862 parsl.dataflow.dflow INFO: Standard output for task 3861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/195650d4-6bc3-4692-b0a9-36b2105d2d6e_isrForCrosstalkSources_855437_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/195650d4-6bc3-4692-b0a9-36b2105d2d6e_isrForCrosstalkSources_855437_8.stderr parsl.dataflow.dflow DEBUG: Task 3862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3862 has memoization hash aac385dcf72101f59c0b8b2d24fdc219 parsl.dataflow.memoization INFO: Task 3862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3862 try 0 launched on executor multi with executor id 3863 parsl.dataflow.dflow INFO: Standard output for task 3862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e3da1462-1419-49d7-9deb-f436f47c131d_isrForCrosstalkSources_855456_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e3da1462-1419-49d7-9deb-f436f47c131d_isrForCrosstalkSources_855456_59.stderr parsl.dataflow.dflow DEBUG: Task 3863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3863 has memoization hash 49efa70eb3f128a002de67c99e737b15 parsl.dataflow.memoization INFO: Task 3863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3863 try 0 launched on executor multi with executor id 3864 parsl.dataflow.dflow INFO: Standard output for task 3863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/230f2c7d-05be-4885-ab1f-3085930f9ddf_isrForCrosstalkSources_855451_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/230f2c7d-05be-4885-ab1f-3085930f9ddf_isrForCrosstalkSources_855451_28.stderr parsl.dataflow.dflow DEBUG: Task 3864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3864 has memoization hash e6014bebea579b260b92187b7baf8efb parsl.dataflow.memoization INFO: Task 3864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3864 try 0 launched on executor multi with executor id 3865 parsl.dataflow.dflow INFO: Standard output for task 3864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b738ac9d-c7b0-423d-8a3c-69dead1ad002_isrForCrosstalkSources_855453_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b738ac9d-c7b0-423d-8a3c-69dead1ad002_isrForCrosstalkSources_855453_61.stderr parsl.dataflow.dflow DEBUG: Task 3865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3865 has memoization hash 6a544f97a22aa6b19b2a49ffb4edbc7b parsl.dataflow.memoization INFO: Task 3865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c189a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3865 try 0 launched on executor multi with executor id 3866 parsl.dataflow.dflow INFO: Standard output for task 3865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f027c974-951a-4a82-8e94-617a3f898de8_isrForCrosstalkSources_855384_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f027c974-951a-4a82-8e94-617a3f898de8_isrForCrosstalkSources_855384_41.stderr parsl.dataflow.dflow DEBUG: Task 3866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3866 has memoization hash fb82e0f55e525f64d9bb76e67ccdb0b4 parsl.dataflow.memoization INFO: Task 3866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3866 try 0 launched on executor multi with executor id 3867 parsl.dataflow.dflow INFO: Standard output for task 3866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/def4886c-cd72-4832-9981-7e1746232089_isrForCrosstalkSources_855428_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/def4886c-cd72-4832-9981-7e1746232089_isrForCrosstalkSources_855428_46.stderr parsl.dataflow.dflow DEBUG: Task 3867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3867 has memoization hash 7b71a03d808dc2f81b7a818bceebaa84 parsl.dataflow.memoization INFO: Task 3867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3867 try 0 launched on executor multi with executor id 3868 parsl.dataflow.dflow INFO: Standard output for task 3867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/99cde655-876e-4875-bb67-ebab618c9e81_isrForCrosstalkSources_855459_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/99cde655-876e-4875-bb67-ebab618c9e81_isrForCrosstalkSources_855459_2.stderr parsl.dataflow.dflow DEBUG: Task 3868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3868 has memoization hash da9dec055ea66fadf597a9d504824f4d parsl.dataflow.memoization INFO: Task 3868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3868 try 0 launched on executor multi with executor id 3869 parsl.dataflow.dflow INFO: Standard output for task 3868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d9ada8ad-ce6f-4197-8127-ce8e44f5d56d_isrForCrosstalkSources_855444_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d9ada8ad-ce6f-4197-8127-ce8e44f5d56d_isrForCrosstalkSources_855444_8.stderr parsl.dataflow.dflow DEBUG: Task 3869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3869 has memoization hash 927103a9d5e9782a4bad542d5f7a273e parsl.dataflow.memoization INFO: Task 3869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3869 try 0 launched on executor multi with executor id 3870 parsl.dataflow.dflow INFO: Standard output for task 3869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/49067eda-a1db-4013-80a8-059f115661b8_isrForCrosstalkSources_855382_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/49067eda-a1db-4013-80a8-059f115661b8_isrForCrosstalkSources_855382_50.stderr parsl.dataflow.dflow DEBUG: Task 3870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3870 has memoization hash ae69e59d08a9d812144e8c0d9296d45a parsl.dataflow.memoization INFO: Task 3870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534c3ed8348eda1f390147ee3be2cb8c with result from task 180 parsl.dataflow.dflow INFO: Parsl task 3870 try 0 launched on executor multi with executor id 3871 parsl.dataflow.dflow INFO: Standard output for task 3870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/63ce581c-9188-4bef-b044-c07bf49802da_isrForCrosstalkSources_855445_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/63ce581c-9188-4bef-b044-c07bf49802da_isrForCrosstalkSources_855445_3.stderr parsl.dataflow.dflow DEBUG: Task 3871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3871 has memoization hash 2e8979d7e6eb357db3d0c78f7b48bd02 parsl.dataflow.memoization INFO: Task 3871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3871 try 0 launched on executor multi with executor id 3872 parsl.dataflow.dflow INFO: Standard output for task 3871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b41767e5-7cc7-414a-b5a1-6c92a14557f3_isrForCrosstalkSources_855457_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b41767e5-7cc7-414a-b5a1-6c92a14557f3_isrForCrosstalkSources_855457_51.stderr parsl.dataflow.dflow DEBUG: Task 3872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3872 has memoization hash be0957f412dd5e30dac292aac1c7369c parsl.dataflow.memoization INFO: Task 3872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/9cfc6a3d-a04c-46c9-b61e-88074c43ce1f_isrForCrosstalkSources_855404_31.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/9cfc6a3d-a04c-46c9-b61e-88074c43ce1f_isrForCrosstalkSources_855404_31.stderr parsl.dataflow.dflow INFO: Parsl task 3872 try 0 launched on executor multi with executor id 3873 parsl.dataflow.dflow INFO: Standard output for task 3872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/eb132a83-42cf-43d7-afeb-63d9ede0c37e_isrForCrosstalkSources_855417_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/eb132a83-42cf-43d7-afeb-63d9ede0c37e_isrForCrosstalkSources_855417_34.stderr parsl.dataflow.dflow DEBUG: Task 3873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3873 has memoization hash aa9a43a8751765defbbcaf65c0b6601c parsl.dataflow.memoization INFO: Task 3873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3873 try 0 launched on executor multi with executor id 3874 parsl.dataflow.dflow INFO: Standard output for task 3873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/32cb854d-a9e0-4d2d-bf2c-7e770c3b7902_isrForCrosstalkSources_855422_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/32cb854d-a9e0-4d2d-bf2c-7e770c3b7902_isrForCrosstalkSources_855422_50.stderr parsl.dataflow.dflow DEBUG: Task 3874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3874 has memoization hash 1f5531909e61eca032ccc9d376b17845 parsl.dataflow.memoization INFO: Task 3874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3874 try 0 launched on executor multi with executor id 3875 parsl.dataflow.dflow INFO: Standard output for task 3874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0af23013-2ea9-4499-8343-27996a76bdaf_isrForCrosstalkSources_855456_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0af23013-2ea9-4499-8343-27996a76bdaf_isrForCrosstalkSources_855456_45.stderr parsl.dataflow.dflow DEBUG: Task 3875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3875 has memoization hash 23fdf50a3ef2c491479051a7d5764848 parsl.dataflow.memoization INFO: Task 3875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3875 try 0 launched on executor multi with executor id 3876 parsl.dataflow.dflow INFO: Standard output for task 3875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0c1a7bd8-4354-44d7-b387-ed1c53bfbbbd_isrForCrosstalkSources_855435_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0c1a7bd8-4354-44d7-b387-ed1c53bfbbbd_isrForCrosstalkSources_855435_46.stderr parsl.dataflow.dflow DEBUG: Task 3876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3876 has memoization hash 8876db91f8717c86863d8e59f7af65f1 parsl.dataflow.memoization INFO: Task 3876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3876 try 0 launched on executor multi with executor id 3877 parsl.dataflow.dflow INFO: Standard output for task 3876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cf74abc9-5de9-4ad1-8ce6-37c318440bc6_isrForCrosstalkSources_855435_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cf74abc9-5de9-4ad1-8ce6-37c318440bc6_isrForCrosstalkSources_855435_32.stderr parsl.dataflow.dflow DEBUG: Task 3877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3877 has memoization hash 6afff33df111d5aa5982679185f952a6 parsl.dataflow.memoization INFO: Task 3877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3877 try 0 launched on executor multi with executor id 3878 parsl.dataflow.dflow INFO: Standard output for task 3877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24bf88a6-111b-4b94-be28-c45169c085dc_isrForCrosstalkSources_855379_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24bf88a6-111b-4b94-be28-c45169c085dc_isrForCrosstalkSources_855379_42.stderr parsl.dataflow.dflow DEBUG: Task 3878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3878 has memoization hash f2d006c7f66a9521989ed86df62e6a62 parsl.dataflow.memoization INFO: Task 3878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c191c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3878 try 0 launched on executor multi with executor id 3879 parsl.dataflow.dflow INFO: Standard output for task 3878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/4a473633-5257-4127-a5cd-5f1914ce594b_isrForCrosstalkSources_855442_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/4a473633-5257-4127-a5cd-5f1914ce594b_isrForCrosstalkSources_855442_48.stderr parsl.dataflow.dflow DEBUG: Task 3879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3879 has memoization hash f0ccf1f36157f345954bd02caeffaeab parsl.dataflow.memoization INFO: Task 3879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3879 try 0 launched on executor multi with executor id 3880 parsl.dataflow.dflow INFO: Standard output for task 3879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/abc22dad-f572-4cf8-a18f-56fe02d27aaf_isrForCrosstalkSources_855414_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/abc22dad-f572-4cf8-a18f-56fe02d27aaf_isrForCrosstalkSources_855414_54.stderr parsl.dataflow.dflow DEBUG: Task 3880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3880 has memoization hash 9a7be1b28a8e9c3ed4f3dd30ae25fa3c parsl.dataflow.memoization INFO: Task 3880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c193a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3880 try 0 launched on executor multi with executor id 3881 parsl.dataflow.dflow INFO: Standard output for task 3880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/4f4202a2-4193-45ed-8051-25567e5fb333_isrForCrosstalkSources_855446_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/4f4202a2-4193-45ed-8051-25567e5fb333_isrForCrosstalkSources_855446_24.stderr parsl.dataflow.dflow DEBUG: Task 3881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3881 has memoization hash 9ede5b1c27536e4bb5c5880fe126d1ed parsl.dataflow.memoization INFO: Task 3881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3881 try 0 launched on executor multi with executor id 3882 parsl.dataflow.dflow INFO: Standard output for task 3881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/5224d7b6-7970-4221-a67c-29b9acda5192_isrForCrosstalkSources_855420_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/5224d7b6-7970-4221-a67c-29b9acda5192_isrForCrosstalkSources_855420_34.stderr parsl.dataflow.dflow DEBUG: Task 3882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3882 has memoization hash 68fca4f7fc1a887fba73aeaf6dc1e456 parsl.dataflow.memoization INFO: Task 3882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3882 try 0 launched on executor multi with executor id 3883 parsl.dataflow.dflow INFO: Standard output for task 3882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/55b22387-f72a-4914-a004-f674c13399fb_isrForCrosstalkSources_855436_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/55b22387-f72a-4914-a004-f674c13399fb_isrForCrosstalkSources_855436_43.stderr parsl.dataflow.dflow DEBUG: Task 3883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3883 has memoization hash d4d4e8c1d92e70ec8d3d1846a6fd72dc parsl.dataflow.memoization INFO: Task 3883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c194e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3883 try 0 launched on executor multi with executor id 3884 parsl.dataflow.dflow INFO: Standard output for task 3883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/f86d8be2-be64-4d63-b3c0-135c97c3f75e_isrForCrosstalkSources_855413_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/f86d8be2-be64-4d63-b3c0-135c97c3f75e_isrForCrosstalkSources_855413_4.stderr parsl.dataflow.dflow DEBUG: Task 3884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3884 has memoization hash 984b00ed3805bc462a4ff97c33b4e01f parsl.dataflow.memoization INFO: Task 3884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3884 try 0 launched on executor multi with executor id 3885 parsl.dataflow.dflow INFO: Standard output for task 3884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/0a2f38c1-6ab9-4c14-9d46-ea4ba7738214_isrForCrosstalkSources_855423_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/0a2f38c1-6ab9-4c14-9d46-ea4ba7738214_isrForCrosstalkSources_855423_16.stderr parsl.dataflow.dflow DEBUG: Task 3885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3885 has memoization hash 900467fdcc995ba5fdda89ee1ebd8835 parsl.dataflow.memoization INFO: Task 3885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c196c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3885 try 0 launched on executor multi with executor id 3886 parsl.dataflow.dflow INFO: Standard output for task 3885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/d654b0ad-9b79-40c7-a6e9-5c13091d5c4a_isrForCrosstalkSources_855418_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/d654b0ad-9b79-40c7-a6e9-5c13091d5c4a_isrForCrosstalkSources_855418_28.stderr parsl.dataflow.dflow DEBUG: Task 3886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3886 has memoization hash 4f0eb0e9b0572ddce676666bc887dbaf parsl.dataflow.memoization INFO: Task 3886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83f264537a7bd7ae55822499ad7b2525 with result from task 321 parsl.dataflow.dflow INFO: Parsl task 3886 try 0 launched on executor multi with executor id 3887 parsl.dataflow.dflow INFO: Standard output for task 3886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a5d3597d-110a-4e36-9e39-07d9fdedf0ee_isrForCrosstalkSources_855416_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a5d3597d-110a-4e36-9e39-07d9fdedf0ee_isrForCrosstalkSources_855416_43.stderr parsl.dataflow.dflow DEBUG: Task 3887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3887 has memoization hash a05adf27e7df6cff136d6339264a9d2d parsl.dataflow.memoization INFO: Task 3887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/30a43159-a185-4b27-874f-ebd509cbaec9_isrForCrosstalkSources_855423_44.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/30a43159-a185-4b27-874f-ebd509cbaec9_isrForCrosstalkSources_855423_44.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2507c381a5fa403b033dca964b5a3e45 with result from task 530 parsl.dataflow.dflow INFO: Parsl task 3887 try 0 launched on executor multi with executor id 3888 parsl.dataflow.dflow INFO: Standard output for task 3887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/76935a02-32d9-4b77-889c-d2a7f7a106f7_isrForCrosstalkSources_855424_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/76935a02-32d9-4b77-889c-d2a7f7a106f7_isrForCrosstalkSources_855424_31.stderr parsl.dataflow.dflow DEBUG: Task 3888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3888 has memoization hash dc6173397a8642c1bc8cb1157d12a9fe parsl.dataflow.memoization INFO: Task 3888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c199e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3888 try 0 launched on executor multi with executor id 3889 parsl.dataflow.dflow INFO: Standard output for task 3888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/dd5caa20-2449-46c8-b1e3-d8409cbeac58_isrForCrosstalkSources_855399_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/dd5caa20-2449-46c8-b1e3-d8409cbeac58_isrForCrosstalkSources_855399_57.stderr parsl.dataflow.dflow DEBUG: Task 3889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3889 has memoization hash 01f9adda23399c05398bd2c4eb453552 parsl.dataflow.memoization INFO: Task 3889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c18360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3889 try 0 launched on executor multi with executor id 3890 parsl.dataflow.dflow INFO: Standard output for task 3889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3018ff7f-951d-4219-96a1-9ebca82b5200_isrForCrosstalkSources_855427_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3018ff7f-951d-4219-96a1-9ebca82b5200_isrForCrosstalkSources_855427_22.stderr parsl.dataflow.dflow DEBUG: Task 3890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3890 has memoization hash e1aa3bb61fced47fdb6a0dd52f42b9d6 parsl.dataflow.memoization INFO: Task 3890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3890 try 0 launched on executor multi with executor id 3891 parsl.dataflow.dflow INFO: Standard output for task 3890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/138acd79-482a-4aa1-b62d-988bc6348dc6_isrForCrosstalkSources_855440_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/138acd79-482a-4aa1-b62d-988bc6348dc6_isrForCrosstalkSources_855440_46.stderr parsl.dataflow.dflow DEBUG: Task 3891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3891 has memoization hash eb79ddaa60cae111a2b8e390840d8178 parsl.dataflow.memoization INFO: Task 3891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3891 try 0 launched on executor multi with executor id 3892 parsl.dataflow.dflow INFO: Standard output for task 3891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e29e7b27-ce34-4b53-92e0-130effe98810_isrForCrosstalkSources_855405_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e29e7b27-ce34-4b53-92e0-130effe98810_isrForCrosstalkSources_855405_52.stderr parsl.dataflow.dflow DEBUG: Task 3892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3892 has memoization hash 8748dd091297832bd042367ddad97f3c parsl.dataflow.memoization INFO: Task 3892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c198a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3892 try 0 launched on executor multi with executor id 3893 parsl.dataflow.dflow INFO: Standard output for task 3892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/cfeef5d3-817c-420c-94ae-8d66fb1a5252_isrForCrosstalkSources_855428_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/cfeef5d3-817c-420c-94ae-8d66fb1a5252_isrForCrosstalkSources_855428_49.stderr parsl.dataflow.dflow DEBUG: Task 3893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3893 has memoization hash 818865af0bff762855c6bd9909969cb7 parsl.dataflow.memoization INFO: Task 3893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3893 try 0 launched on executor multi with executor id 3894 parsl.dataflow.dflow INFO: Standard output for task 3893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/01229cae-cb3e-4419-8167-649dbf03422c_isrForCrosstalkSources_855421_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/01229cae-cb3e-4419-8167-649dbf03422c_isrForCrosstalkSources_855421_59.stderr parsl.dataflow.dflow DEBUG: Task 3894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3894 has memoization hash 8c5b5d023a2af1ba925158fe500b9f06 parsl.dataflow.memoization INFO: Task 3894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ca13abe5-735f-4642-9168-8bb796485acb_isrForCrosstalkSources_855398_43.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ca13abe5-735f-4642-9168-8bb796485acb_isrForCrosstalkSources_855398_43.stderr parsl.dataflow.dflow INFO: Parsl task 3894 try 0 launched on executor multi with executor id 3895 parsl.dataflow.dflow INFO: Standard output for task 3894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68f5efea-c1e6-4555-ab5b-ae610ec8b837_isrForCrosstalkSources_855399_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68f5efea-c1e6-4555-ab5b-ae610ec8b837_isrForCrosstalkSources_855399_33.stderr parsl.dataflow.dflow DEBUG: Task 3895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3895 has memoization hash b506450ad87410936ef453f25a970769 parsl.dataflow.memoization INFO: Task 3895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3895 try 0 launched on executor multi with executor id 3896 parsl.dataflow.dflow INFO: Standard output for task 3895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e5463549-345b-4749-9e40-94afe26c287b_isrForCrosstalkSources_855434_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e5463549-345b-4749-9e40-94afe26c287b_isrForCrosstalkSources_855434_2.stderr parsl.dataflow.dflow DEBUG: Task 3896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3896 has memoization hash 9e637d05535f7bcb25c35014bf989334 parsl.dataflow.memoization INFO: Task 3896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3896 try 0 launched on executor multi with executor id 3897 parsl.dataflow.dflow INFO: Standard output for task 3896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2d37c0a8-9fb9-42b1-b0f1-0137924d4d6c_isrForCrosstalkSources_855401_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2d37c0a8-9fb9-42b1-b0f1-0137924d4d6c_isrForCrosstalkSources_855401_13.stderr parsl.dataflow.dflow DEBUG: Task 3897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3897 has memoization hash b70f4a6d502607e3c2acea71c8b7d714 parsl.dataflow.memoization INFO: Task 3897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3897 try 0 launched on executor multi with executor id 3898 parsl.dataflow.dflow INFO: Standard output for task 3897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f0a8532c-9235-4370-96da-c8c4851dc746_isrForCrosstalkSources_855419_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f0a8532c-9235-4370-96da-c8c4851dc746_isrForCrosstalkSources_855419_42.stderr parsl.dataflow.dflow DEBUG: Task 3898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3898 has memoization hash 40c7027ab451adefcb0e04accfe5c737 parsl.dataflow.memoization INFO: Task 3898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3898 try 0 launched on executor multi with executor id 3899 parsl.dataflow.dflow INFO: Standard output for task 3898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/79b20288-bac2-42fc-888f-4bed5fc9cb0b_isrForCrosstalkSources_855458_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/79b20288-bac2-42fc-888f-4bed5fc9cb0b_isrForCrosstalkSources_855458_11.stderr parsl.dataflow.dflow DEBUG: Task 3899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3899 has memoization hash e57872d292d59d87bd3089cb5e85180f parsl.dataflow.memoization INFO: Task 3899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3899 try 0 launched on executor multi with executor id 3900 parsl.dataflow.dflow INFO: Standard output for task 3899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1d967d13-6b18-48ee-ad9d-7a2be81a9b9c_isrForCrosstalkSources_855382_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1d967d13-6b18-48ee-ad9d-7a2be81a9b9c_isrForCrosstalkSources_855382_8.stderr parsl.dataflow.dflow DEBUG: Task 3900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3900 has memoization hash 57908a1ceed663b6372a9f893e95ad7f parsl.dataflow.memoization INFO: Task 3900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3900 try 0 launched on executor multi with executor id 3901 parsl.dataflow.dflow INFO: Standard output for task 3900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a90d2da6-38f1-48c7-bb7d-62679f831cb2_isrForCrosstalkSources_855455_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a90d2da6-38f1-48c7-bb7d-62679f831cb2_isrForCrosstalkSources_855455_12.stderr parsl.dataflow.dflow DEBUG: Task 3901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3901 has memoization hash f05d0f2120062e8b0a706741ee0f9b4f parsl.dataflow.memoization INFO: Task 3901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:41 proc_lsst.multi:170 [INFO] [multi] job_id 16964176 proc_lsst.multi INFO: [multi] job_id 16964176 2024-03-09 14:51:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-09 14:51:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964176 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964176 parsl.executors.status_handling DEBUG: Launched block 150 on executor multi with job ID 16964176 parsl.executors.status_handling INFO: Allocated block ID 151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-151 2024-03-09 14:51:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-09 14:51:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3901 try 0 launched on executor multi with executor id 3902 parsl.dataflow.dflow INFO: Standard output for task 3901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/2a3855bd-ce06-4b14-b23a-dc0e54dd0a6b_isrForCrosstalkSources_855412_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/2a3855bd-ce06-4b14-b23a-dc0e54dd0a6b_isrForCrosstalkSources_855412_4.stderr parsl.dataflow.dflow DEBUG: Task 3902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3902 has memoization hash ce66b35be87a10fb5829e96fe197702e parsl.dataflow.memoization INFO: Task 3902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3902 try 0 launched on executor multi with executor id 3903 parsl.dataflow.dflow INFO: Standard output for task 3902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c4bb778f-5ae4-4684-819b-0b212dab2ec8_isrForCrosstalkSources_855385_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c4bb778f-5ae4-4684-819b-0b212dab2ec8_isrForCrosstalkSources_855385_58.stderr parsl.dataflow.dflow DEBUG: Task 3903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3903 has memoization hash da6ede873488d3476093688fc186dc5a parsl.dataflow.memoization INFO: Task 3903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3903 try 0 launched on executor multi with executor id 3904 parsl.dataflow.dflow INFO: Standard output for task 3903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/002578a0-a7fa-4356-8817-6d5060634888_isrForCrosstalkSources_855408_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/002578a0-a7fa-4356-8817-6d5060634888_isrForCrosstalkSources_855408_4.stderr parsl.dataflow.dflow DEBUG: Task 3904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3904 has memoization hash 906ae49d0c16b99bd9d42e236b49edc3 parsl.dataflow.memoization INFO: Task 3904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5eeefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3904 try 0 launched on executor multi with executor id 3905 parsl.dataflow.dflow INFO: Standard output for task 3904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fbea2a0d-7d59-4e99-98ae-665dff156e3d_isrForCrosstalkSources_855412_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fbea2a0d-7d59-4e99-98ae-665dff156e3d_isrForCrosstalkSources_855412_37.stderr parsl.dataflow.dflow DEBUG: Task 3905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3905 has memoization hash 65a6342e13bc3aba49441b1a6d9b26b8 parsl.dataflow.memoization INFO: Task 3905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 506e42c1f81d525740cd807313d8b2d8 with result from task 521 parsl.dataflow.dflow INFO: Parsl task 3905 try 0 launched on executor multi with executor id 3906 parsl.dataflow.dflow INFO: Standard output for task 3905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/2eb7d098-faf3-4db3-8100-e29a2feef5a6_isrForCrosstalkSources_855440_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/2eb7d098-faf3-4db3-8100-e29a2feef5a6_isrForCrosstalkSources_855440_22.stderr parsl.dataflow.dflow DEBUG: Task 3906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3906 has memoization hash e0147dd4b84b1ec586ac4290234d23cf parsl.dataflow.memoization INFO: Task 3906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3906 try 0 launched on executor multi with executor id 3907 parsl.dataflow.dflow INFO: Standard output for task 3906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/46360f89-66df-48a0-979c-7e7512269e3b_isrForCrosstalkSources_855457_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/46360f89-66df-48a0-979c-7e7512269e3b_isrForCrosstalkSources_855457_26.stderr parsl.dataflow.dflow DEBUG: Task 3907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3907 has memoization hash e1a66197f5903db430ca32a8072a85d2 parsl.dataflow.memoization INFO: Task 3907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eea189e5-fc1b-4cfa-9fc1-d9749f4824d6_isrForCrosstalkSources_855446_13.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eea189e5-fc1b-4cfa-9fc1-d9749f4824d6_isrForCrosstalkSources_855446_13.stderr parsl.dataflow.dflow INFO: Parsl task 3907 try 0 launched on executor multi with executor id 3908 parsl.dataflow.dflow INFO: Standard output for task 3907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/acb8c892-79f5-4071-931a-45503b8fdfa1_isrForCrosstalkSources_855445_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/acb8c892-79f5-4071-931a-45503b8fdfa1_isrForCrosstalkSources_855445_33.stderr parsl.dataflow.dflow DEBUG: Task 3908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3908 has memoization hash 19d2168243c3bbbf8b084da84f2d3ea3 parsl.dataflow.memoization INFO: Task 3908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5d62d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3908 try 0 launched on executor multi with executor id 3909 parsl.dataflow.dflow INFO: Standard output for task 3908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4bc6cc34-d0d3-498d-988b-ae0dab3716dd_isrForCrosstalkSources_855382_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4bc6cc34-d0d3-498d-988b-ae0dab3716dd_isrForCrosstalkSources_855382_20.stderr parsl.dataflow.dflow DEBUG: Task 3909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3909 has memoization hash bab9c9b6a1ddea47976a736da14de9b2 parsl.dataflow.memoization INFO: Task 3909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3909 try 0 launched on executor multi with executor id 3910 parsl.dataflow.dflow INFO: Standard output for task 3909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fffe5ddf-302d-481a-9cab-f4babc8c2249_isrForCrosstalkSources_855439_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fffe5ddf-302d-481a-9cab-f4babc8c2249_isrForCrosstalkSources_855439_39.stderr parsl.dataflow.dflow DEBUG: Task 3910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3910 has memoization hash 27e4f577506544e61f4191176d259c70 parsl.dataflow.memoization INFO: Task 3910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3910 try 0 launched on executor multi with executor id 3911 parsl.dataflow.dflow INFO: Standard output for task 3910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/512b9b0b-1395-42b8-913f-8a00fed14d84_isrForCrosstalkSources_855442_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/512b9b0b-1395-42b8-913f-8a00fed14d84_isrForCrosstalkSources_855442_60.stderr parsl.dataflow.dflow DEBUG: Task 3911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3911 has memoization hash cdc43cfba34c1d824841bb4548b08bc7 parsl.dataflow.memoization INFO: Task 3911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3911 try 0 launched on executor multi with executor id 3912 parsl.dataflow.dflow INFO: Standard output for task 3911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7e68914f-1918-4eff-befe-e6b76fd46b4f_isrForCrosstalkSources_855387_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7e68914f-1918-4eff-befe-e6b76fd46b4f_isrForCrosstalkSources_855387_33.stderr parsl.dataflow.dflow DEBUG: Task 3912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3912 has memoization hash 32320e7dc8e516bf26cfdb72313771e9 parsl.dataflow.memoization INFO: Task 3912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3912 try 0 launched on executor multi with executor id 3913 parsl.dataflow.dflow INFO: Standard output for task 3912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c78fa6ae-5ad2-4e19-896a-0a4a156b726f_isrForCrosstalkSources_855427_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c78fa6ae-5ad2-4e19-896a-0a4a156b726f_isrForCrosstalkSources_855427_38.stderr parsl.dataflow.dflow DEBUG: Task 3913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3913 has memoization hash 2a8ad7becdc5986963b0543abdf54bf7 parsl.dataflow.memoization INFO: Task 3913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3913 try 0 launched on executor multi with executor id 3914 parsl.dataflow.dflow INFO: Standard output for task 3913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a9ad1ebb-315f-4508-9f7c-0a3791db3b06_isrForCrosstalkSources_855393_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a9ad1ebb-315f-4508-9f7c-0a3791db3b06_isrForCrosstalkSources_855393_5.stderr parsl.dataflow.dflow DEBUG: Task 3914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3914 has memoization hash aef50a8f2e29734ce0ae71a934ad1836 parsl.dataflow.memoization INFO: Task 3914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3914 try 0 launched on executor multi with executor id 3915 parsl.dataflow.dflow INFO: Standard output for task 3914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c7847335-0535-4335-95ec-c6fe2b23c99c_isrForCrosstalkSources_855431_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c7847335-0535-4335-95ec-c6fe2b23c99c_isrForCrosstalkSources_855431_21.stderr parsl.dataflow.dflow DEBUG: Task 3915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3915 has memoization hash d1c9e0810b078e953a831a3acf839a95 parsl.dataflow.memoization INFO: Task 3915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3915 try 0 launched on executor multi with executor id 3916 parsl.dataflow.dflow INFO: Standard output for task 3915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/43d716fd-9df4-4810-8bf6-b73a1293a2ef_isrForCrosstalkSources_855458_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/43d716fd-9df4-4810-8bf6-b73a1293a2ef_isrForCrosstalkSources_855458_29.stderr parsl.dataflow.dflow DEBUG: Task 3916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3916 has memoization hash 2e425a361c391de14b7a473539a96154 parsl.dataflow.memoization INFO: Task 3916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3916 try 0 launched on executor multi with executor id 3917 parsl.dataflow.dflow INFO: Standard output for task 3916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/32863892-a6b5-47d7-aef2-2c9b553c24d1_isrForCrosstalkSources_855379_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/32863892-a6b5-47d7-aef2-2c9b553c24d1_isrForCrosstalkSources_855379_39.stderr parsl.dataflow.dflow DEBUG: Task 3917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3917 has memoization hash 1a20a006cca42d4d20a03bf58a1d5388 parsl.dataflow.memoization INFO: Task 3917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3917 try 0 launched on executor multi with executor id 3918 parsl.dataflow.dflow INFO: Standard output for task 3917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6d4e6402-a013-4885-832e-20d240909f61_isrForCrosstalkSources_855445_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6d4e6402-a013-4885-832e-20d240909f61_isrForCrosstalkSources_855445_20.stderr parsl.dataflow.dflow DEBUG: Task 3918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3918 has memoization hash 46cdcc4a11cfa83129d416baac974896 parsl.dataflow.memoization INFO: Task 3918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3918 try 0 launched on executor multi with executor id 3919 parsl.dataflow.dflow INFO: Standard output for task 3918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/463be0c1-97e4-4059-8a20-75bc6bcbe0dc_isrForCrosstalkSources_855417_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/463be0c1-97e4-4059-8a20-75bc6bcbe0dc_isrForCrosstalkSources_855417_11.stderr parsl.dataflow.dflow DEBUG: Task 3919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3919 has memoization hash 1416e4299de1755d702737479e2dc5a9 parsl.dataflow.memoization INFO: Task 3919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3919 try 0 launched on executor multi with executor id 3920 parsl.dataflow.dflow INFO: Standard output for task 3919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/fb267b70-f9b1-4dbc-bdd0-d20bcba6f595_isrForCrosstalkSources_855444_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/fb267b70-f9b1-4dbc-bdd0-d20bcba6f595_isrForCrosstalkSources_855444_49.stderr parsl.dataflow.dflow DEBUG: Task 3920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3920 has memoization hash 18de3aef85dd2e16a0f49f51783621dd parsl.dataflow.memoization INFO: Task 3920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3920 try 0 launched on executor multi with executor id 3921 parsl.dataflow.dflow INFO: Standard output for task 3920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f7366e35-adf0-4597-8ad1-ef7cfd07b6c2_isrForCrosstalkSources_855432_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f7366e35-adf0-4597-8ad1-ef7cfd07b6c2_isrForCrosstalkSources_855432_36.stderr parsl.dataflow.dflow DEBUG: Task 3921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3921 has memoization hash 3f55f510e4eb66367746f8a8cc0a7b8c parsl.dataflow.memoization INFO: Task 3921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3921 try 0 launched on executor multi with executor id 3922 parsl.dataflow.dflow INFO: Standard output for task 3921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5fea71a8-98ed-4bef-8561-83bf2883842b_isrForCrosstalkSources_855402_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5fea71a8-98ed-4bef-8561-83bf2883842b_isrForCrosstalkSources_855402_2.stderr parsl.dataflow.dflow DEBUG: Task 3922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3922 has memoization hash 07a15b8898a8a0056cc35c12d35fca20 parsl.dataflow.memoization INFO: Task 3922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3922 try 0 launched on executor multi with executor id 3923 parsl.dataflow.dflow INFO: Standard output for task 3922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/84684709-caa1-4b20-bca1-736c79a58a9f_isrForCrosstalkSources_855384_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/84684709-caa1-4b20-bca1-736c79a58a9f_isrForCrosstalkSources_855384_54.stderr parsl.dataflow.dflow DEBUG: Task 3923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3923 has memoization hash 9264bad1c1baffb99b406f586582cffa parsl.dataflow.memoization INFO: Task 3923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4a7cb0b430a8ad7c43df9df43e840a7 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e60b8503-494d-4b25-8d0e-0d4f37ee2f23_isrForCrosstalkSources_855412_43.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e60b8503-494d-4b25-8d0e-0d4f37ee2f23_isrForCrosstalkSources_855412_43.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d571a0c4a59ff720fd7d083045976dd with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4863cfa-6646-4d7e-959d-a0d385721c62_isrForCrosstalkSources_855434_18.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4863cfa-6646-4d7e-959d-a0d385721c62_isrForCrosstalkSources_855434_18.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6077f8728d73b33dd9b4f6fc1a17cf16 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e7a890b9-cc29-4b33-b663-d6c093ccae9d_isrForCrosstalkSources_855428_13.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e7a890b9-cc29-4b33-b663-d6c093ccae9d_isrForCrosstalkSources_855428_13.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30864e491b6f937bfa46354efe834a1d with result from task 337 parsl.dataflow.dflow INFO: Parsl task 3923 try 0 launched on executor multi with executor id 3924 parsl.dataflow.dflow INFO: Standard output for task 3923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/3696c9b7-c0e9-4e02-ba55-7777fa2e73a9_isrForCrosstalkSources_855410_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/3696c9b7-c0e9-4e02-ba55-7777fa2e73a9_isrForCrosstalkSources_855410_60.stderr parsl.dataflow.dflow DEBUG: Task 3924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3924 has memoization hash ef82255a8c140a97384361f41e8c1ac6 parsl.dataflow.memoization INFO: Task 3924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3924 try 0 launched on executor multi with executor id 3925 parsl.dataflow.dflow INFO: Standard output for task 3924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/682c8723-8207-4ed7-bec7-f8a7151fecf7_isrForCrosstalkSources_855428_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/682c8723-8207-4ed7-bec7-f8a7151fecf7_isrForCrosstalkSources_855428_60.stderr parsl.dataflow.dflow DEBUG: Task 3925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3925 has memoization hash a97e3def911d3ff3579f0e2f9c4676fb parsl.dataflow.memoization INFO: Task 3925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3925 try 0 launched on executor multi with executor id 3926 parsl.dataflow.dflow INFO: Standard output for task 3925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e3ed6000-e3d5-45e1-b46b-34fd3cffb48f_isrForCrosstalkSources_855432_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e3ed6000-e3d5-45e1-b46b-34fd3cffb48f_isrForCrosstalkSources_855432_46.stderr parsl.dataflow.dflow DEBUG: Task 3926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3926 has memoization hash 9fc2c52a20eeebd6d2e27f893ea540c2 parsl.dataflow.memoization INFO: Task 3926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f0795439-b226-474f-9d9b-be54370e4dba_isrForCrosstalkSources_855429_2.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f0795439-b226-474f-9d9b-be54370e4dba_isrForCrosstalkSources_855429_2.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3237caa1af44e66fbbf044678586e03 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/32cc831c-a1e1-4e12-9f15-9e8dd35617f1_isrForCrosstalkSources_855390_30.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/32cc831c-a1e1-4e12-9f15-9e8dd35617f1_isrForCrosstalkSources_855390_30.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e7bc506cbd462ea07a2b2ebfadb0d93 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/574e3960-6912-48b5-a42d-a82acebe42c2_isrForCrosstalkSources_855433_34.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/574e3960-6912-48b5-a42d-a82acebe42c2_isrForCrosstalkSources_855433_34.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48e4bd002fd164017043fc65a6ec2052 with result from task 344 parsl.dataflow.dflow INFO: Parsl task 3926 try 0 launched on executor multi with executor id 3927 parsl.dataflow.dflow INFO: Standard output for task 3926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/288d04a6-f452-48b3-843e-67ec515a9e30_isrForCrosstalkSources_855389_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/288d04a6-f452-48b3-843e-67ec515a9e30_isrForCrosstalkSources_855389_38.stderr parsl.dataflow.dflow DEBUG: Task 3927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3927 has memoization hash 75c058ffd60e52b721ab66b546303317 parsl.dataflow.memoization INFO: Task 3927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ccb0799b-565d-4b71-9800-f6b394a3acd7_isrForCrosstalkSources_855455_53.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ccb0799b-565d-4b71-9800-f6b394a3acd7_isrForCrosstalkSources_855455_53.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 464f0092d5061f1939676469d4ff31cc with result from task 339 parsl.dataflow.dflow INFO: Parsl task 3927 try 0 launched on executor multi with executor id 3928 parsl.dataflow.dflow INFO: Standard output for task 3927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/5370f7a8-26c5-4695-8375-10377cc9af19_isrForCrosstalkSources_855426_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/5370f7a8-26c5-4695-8375-10377cc9af19_isrForCrosstalkSources_855426_51.stderr parsl.dataflow.dflow DEBUG: Task 3928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3928 has memoization hash db14dd5063933dd363a4f201baaae9c4 parsl.dataflow.memoization INFO: Task 3928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3928 try 0 launched on executor multi with executor id 3929 parsl.dataflow.dflow INFO: Standard output for task 3928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/cfa4f8db-2648-47d6-860a-778aed1ca4f2_isrForCrosstalkSources_855416_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/cfa4f8db-2648-47d6-860a-778aed1ca4f2_isrForCrosstalkSources_855416_11.stderr parsl.dataflow.dflow DEBUG: Task 3929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3929 has memoization hash 153647177b9acfc41e43f6a8ac87ed49 parsl.dataflow.memoization INFO: Task 3929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b80468c8-8547-4f54-9de0-a65e6b020da9_isrForCrosstalkSources_855421_55.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b80468c8-8547-4f54-9de0-a65e6b020da9_isrForCrosstalkSources_855421_55.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8935e9c6fce79eadc1d255379a3262d with result from task 226 parsl.dataflow.dflow INFO: Parsl task 3929 try 0 launched on executor multi with executor id 3930 parsl.dataflow.dflow INFO: Standard output for task 3929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/bf2dc53c-6ed0-4b57-bc03-a19f9b1d08ed_isrForCrosstalkSources_855406_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/bf2dc53c-6ed0-4b57-bc03-a19f9b1d08ed_isrForCrosstalkSources_855406_57.stderr parsl.dataflow.dflow DEBUG: Task 3930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3930 has memoization hash 02f5c2bd849f5f9970f922b91ae741a6 parsl.dataflow.memoization INFO: Task 3930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/bac87108-fa39-4a72-9418-9423aa2e31d8_isrForCrosstalkSources_855452_59.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/bac87108-fa39-4a72-9418-9423aa2e31d8_isrForCrosstalkSources_855452_59.stderr parsl.dataflow.dflow INFO: Parsl task 3930 try 0 launched on executor multi with executor id 3931 parsl.dataflow.dflow INFO: Standard output for task 3930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/90d47d33-8974-44db-9a23-16d52d75bcdd_isrForCrosstalkSources_855428_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/90d47d33-8974-44db-9a23-16d52d75bcdd_isrForCrosstalkSources_855428_62.stderr parsl.dataflow.dflow DEBUG: Task 3931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3931 has memoization hash f518dd73a524ce6d2b7d524093e82d99 parsl.dataflow.memoization INFO: Task 3931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3931 try 0 launched on executor multi with executor id 3932 parsl.dataflow.dflow INFO: Standard output for task 3931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/af878195-5c79-4506-b1be-0512e77d3b8b_isrForCrosstalkSources_855458_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/af878195-5c79-4506-b1be-0512e77d3b8b_isrForCrosstalkSources_855458_57.stderr parsl.dataflow.dflow DEBUG: Task 3932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3932 has memoization hash 169eb70b4fd2c9978a86c0a8b70863ed parsl.dataflow.memoization INFO: Task 3932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3932 try 0 launched on executor multi with executor id 3933 parsl.dataflow.dflow INFO: Standard output for task 3932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a7bcc0a3-4ba9-4791-b3e9-0e54b292a73b_isrForCrosstalkSources_855416_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a7bcc0a3-4ba9-4791-b3e9-0e54b292a73b_isrForCrosstalkSources_855416_16.stderr parsl.dataflow.dflow DEBUG: Task 3933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3933 has memoization hash 9e0f8ea312db0abc315d34ec8c0806cb parsl.dataflow.memoization INFO: Task 3933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3933 try 0 launched on executor multi with executor id 3934 parsl.dataflow.dflow INFO: Standard output for task 3933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c7c6a5e2-7b3c-447f-886e-fc742f2cbfcb_isrForCrosstalkSources_855401_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c7c6a5e2-7b3c-447f-886e-fc742f2cbfcb_isrForCrosstalkSources_855401_43.stderr parsl.dataflow.dflow DEBUG: Task 3934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3934 has memoization hash c429e9b4a1e26e1462005d059646cea5 parsl.dataflow.memoization INFO: Task 3934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3934 try 0 launched on executor multi with executor id 3935 parsl.dataflow.dflow INFO: Standard output for task 3934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a8eb784d-bb22-4b03-899b-64db91c78bb8_isrForCrosstalkSources_855410_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a8eb784d-bb22-4b03-899b-64db91c78bb8_isrForCrosstalkSources_855410_27.stderr parsl.dataflow.dflow DEBUG: Task 3935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3935 has memoization hash 8bd8966b8fb38a45e597889562975d60 parsl.dataflow.memoization INFO: Task 3935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:42 proc_lsst.multi:170 [INFO] [multi] job_id 16964177 proc_lsst.multi INFO: [multi] job_id 16964177 2024-03-09 14:51:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-09 14:51:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964177 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964177 parsl.executors.status_handling DEBUG: Launched block 151 on executor multi with job ID 16964177 parsl.executors.status_handling INFO: Allocated block ID 152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-152 2024-03-09 14:51:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 2024-03-09 14:51:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-09 14:51:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3935 try 0 launched on executor multi with executor id 3936 parsl.dataflow.dflow INFO: Standard output for task 3935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7809eb10-57e8-43d2-b000-5e004ca04553_isrForCrosstalkSources_855390_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7809eb10-57e8-43d2-b000-5e004ca04553_isrForCrosstalkSources_855390_10.stderr parsl.dataflow.dflow DEBUG: Task 3936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3936 has memoization hash ee25ca98005d9426ee247553b18f80c3 parsl.dataflow.memoization INFO: Task 3936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3936 try 0 launched on executor multi with executor id 3937 parsl.dataflow.dflow INFO: Standard output for task 3936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/c5bb0709-27de-4ee5-87ef-c5947f616cb2_isrForCrosstalkSources_855424_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/c5bb0709-27de-4ee5-87ef-c5947f616cb2_isrForCrosstalkSources_855424_23.stderr parsl.dataflow.dflow DEBUG: Task 3937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3937 has memoization hash ed185e8f8de74b2a2170c4e6247d649a parsl.dataflow.memoization INFO: Task 3937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67a108ce3e9d9b14e7e98d64d9ae6bfe with result from task 177 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3937 try 0 launched on executor multi with executor id 3938 parsl.dataflow.dflow INFO: Standard output for task 3937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0278b029-7e91-4a10-b6b3-1d76b61c938a_isrForCrosstalkSources_855411_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0278b029-7e91-4a10-b6b3-1d76b61c938a_isrForCrosstalkSources_855411_27.stderr parsl.dataflow.dflow DEBUG: Task 3938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3938 has memoization hash 61f00b52c097bc5996217db8787be3c0 parsl.dataflow.memoization INFO: Task 3938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/102dca8d-a803-4926-9e37-22c5479fbb44_isrForCrosstalkSources_855455_1.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/102dca8d-a803-4926-9e37-22c5479fbb44_isrForCrosstalkSources_855455_1.stderr parsl.dataflow.dflow INFO: Parsl task 3938 try 0 launched on executor multi with executor id 3939 parsl.dataflow.dflow INFO: Standard output for task 3938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/41f7834c-acd0-48d8-9d11-e57df0aa1d46_isrForCrosstalkSources_855435_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/41f7834c-acd0-48d8-9d11-e57df0aa1d46_isrForCrosstalkSources_855435_20.stderr parsl.dataflow.dflow DEBUG: Task 3939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3939 has memoization hash 4d830a6e092ff73f3950d82f7a1db011 parsl.dataflow.memoization INFO: Task 3939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3939 try 0 launched on executor multi with executor id 3940 parsl.dataflow.dflow INFO: Standard output for task 3939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/788b57e3-8262-401c-a52f-565dbac8e31d_isrForCrosstalkSources_855402_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/788b57e3-8262-401c-a52f-565dbac8e31d_isrForCrosstalkSources_855402_52.stderr parsl.dataflow.dflow DEBUG: Task 3940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3940 has memoization hash 35676ba5548a5c65e9f2584d7cf884a8 parsl.dataflow.memoization INFO: Task 3940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3940 try 0 launched on executor multi with executor id 3941 parsl.dataflow.dflow INFO: Standard output for task 3940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f16b4c6a-4668-4702-bd61-4e41f477abd2_isrForCrosstalkSources_855379_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f16b4c6a-4668-4702-bd61-4e41f477abd2_isrForCrosstalkSources_855379_12.stderr parsl.dataflow.dflow DEBUG: Task 3941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3941 has memoization hash c669460fc86b597e112af199eda6c4f8 parsl.dataflow.memoization INFO: Task 3941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c19f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3941 try 0 launched on executor multi with executor id 3942 parsl.dataflow.dflow INFO: Standard output for task 3941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/af5c812d-3140-4c78-8302-e5a584408037_isrForCrosstalkSources_855407_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/af5c812d-3140-4c78-8302-e5a584408037_isrForCrosstalkSources_855407_17.stderr parsl.dataflow.dflow DEBUG: Task 3942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3942 has memoization hash 3420af97c1a719e6d8fb9537ba486180 parsl.dataflow.memoization INFO: Task 3942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3942 try 0 launched on executor multi with executor id 3943 parsl.dataflow.dflow INFO: Standard output for task 3942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7dd81437-5405-43cd-a533-e65b0920459d_isrForCrosstalkSources_855410_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7dd81437-5405-43cd-a533-e65b0920459d_isrForCrosstalkSources_855410_28.stderr parsl.dataflow.dflow DEBUG: Task 3943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3943 has memoization hash cd02b4f0354d915c04e6cbedfd1b1e76 parsl.dataflow.memoization INFO: Task 3943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3943 try 0 launched on executor multi with executor id 3944 parsl.dataflow.dflow INFO: Standard output for task 3943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/68644483-ade7-4182-b667-eb0ccfcf5804_isrForCrosstalkSources_855436_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/68644483-ade7-4182-b667-eb0ccfcf5804_isrForCrosstalkSources_855436_54.stderr parsl.dataflow.dflow DEBUG: Task 3944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3944 has memoization hash 0a671a8e132d5955ee51f91346d606ab parsl.dataflow.memoization INFO: Task 3944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3944 try 0 launched on executor multi with executor id 3945 parsl.dataflow.dflow INFO: Standard output for task 3944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/fb3ea7e2-084e-4824-a061-bfa61b0b22d1_isrForCrosstalkSources_855429_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/fb3ea7e2-084e-4824-a061-bfa61b0b22d1_isrForCrosstalkSources_855429_23.stderr parsl.dataflow.dflow DEBUG: Task 3945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3945 has memoization hash 5a3b77b862350f35c00fe926cc9049e2 parsl.dataflow.memoization INFO: Task 3945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3945 try 0 launched on executor multi with executor id 3946 parsl.dataflow.dflow INFO: Standard output for task 3945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/5289e483-47bd-415f-a6e6-dbda71e45928_isrForCrosstalkSources_855411_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/5289e483-47bd-415f-a6e6-dbda71e45928_isrForCrosstalkSources_855411_57.stderr parsl.dataflow.dflow DEBUG: Task 3946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3946 has memoization hash cdcb138758358bae543a017c3fb00ad7 parsl.dataflow.memoization INFO: Task 3946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3946 try 0 launched on executor multi with executor id 3947 parsl.dataflow.dflow INFO: Standard output for task 3946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e85ab6f8-53cb-4974-a7ff-9ba2c55d985e_isrForCrosstalkSources_855422_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e85ab6f8-53cb-4974-a7ff-9ba2c55d985e_isrForCrosstalkSources_855422_43.stderr parsl.dataflow.dflow DEBUG: Task 3947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3947 has memoization hash ab34b0411034636ae5aa7ae4a079ae57 parsl.dataflow.memoization INFO: Task 3947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3947 try 0 launched on executor multi with executor id 3948 parsl.dataflow.dflow INFO: Standard output for task 3947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/64459758-5ec8-4135-85d9-0c2d4b14da79_isrForCrosstalkSources_855419_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/64459758-5ec8-4135-85d9-0c2d4b14da79_isrForCrosstalkSources_855419_62.stderr parsl.dataflow.dflow DEBUG: Task 3948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3948 has memoization hash 9ff75c90244b3aca1b24ffb355ad6654 parsl.dataflow.memoization INFO: Task 3948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3364df049f080bbe1aebba076d9cf9fa with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/46fb774e-9eab-4de7-b444-6e4573739549_isrForCrosstalkSources_855417_49.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/46fb774e-9eab-4de7-b444-6e4573739549_isrForCrosstalkSources_855417_49.stderr parsl.dataflow.dflow INFO: Parsl task 3948 try 0 launched on executor multi with executor id 3949 parsl.dataflow.dflow INFO: Standard output for task 3948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f9388783-84cb-41c3-804b-72532829d5eb_isrForCrosstalkSources_855455_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f9388783-84cb-41c3-804b-72532829d5eb_isrForCrosstalkSources_855455_24.stderr parsl.dataflow.dflow DEBUG: Task 3949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3949 has memoization hash 4dfe85b5221ef7a41d7f7401450ccffa parsl.dataflow.memoization INFO: Task 3949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3949 try 0 launched on executor multi with executor id 3950 parsl.dataflow.dflow INFO: Standard output for task 3949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a814d647-4bb8-4abd-98b4-7dbee10e0e38_isrForCrosstalkSources_855448_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a814d647-4bb8-4abd-98b4-7dbee10e0e38_isrForCrosstalkSources_855448_48.stderr parsl.dataflow.dflow DEBUG: Task 3950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3950 has memoization hash 77a24ec98fd0ba1c1325b04e897fcd28 parsl.dataflow.memoization INFO: Task 3950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3950 try 0 launched on executor multi with executor id 3951 parsl.dataflow.dflow INFO: Standard output for task 3950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/172ec7a2-0b44-47e7-8dd8-aa42af8f2512_isrForCrosstalkSources_855459_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/172ec7a2-0b44-47e7-8dd8-aa42af8f2512_isrForCrosstalkSources_855459_48.stderr parsl.dataflow.dflow DEBUG: Task 3951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3951 has memoization hash 86e76907f39b2717e6dba53cceb4b0a5 parsl.dataflow.memoization INFO: Task 3951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3951 try 0 launched on executor multi with executor id 3952 parsl.dataflow.dflow INFO: Standard output for task 3951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f2ad1495-1740-4bf0-8d16-710de03704a4_isrForCrosstalkSources_855384_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f2ad1495-1740-4bf0-8d16-710de03704a4_isrForCrosstalkSources_855384_39.stderr parsl.dataflow.dflow DEBUG: Task 3952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3952 has memoization hash 3ede8f1e4fd8561e4d5bd5a824e37316 parsl.dataflow.memoization INFO: Task 3952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3952 try 0 launched on executor multi with executor id 3953 parsl.dataflow.dflow INFO: Standard output for task 3952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/e9fe238e-263e-4718-acfc-e93316e3ab67_isrForCrosstalkSources_855417_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/e9fe238e-263e-4718-acfc-e93316e3ab67_isrForCrosstalkSources_855417_14.stderr parsl.dataflow.dflow DEBUG: Task 3953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3953 has memoization hash 7c899688bef4722f2ff8678a4be91ca7 parsl.dataflow.memoization INFO: Task 3953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3953 try 0 launched on executor multi with executor id 3954 parsl.dataflow.dflow INFO: Standard output for task 3953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f8d9ac6e-bebc-492a-935e-ec604a0e1ada_isrForCrosstalkSources_855459_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f8d9ac6e-bebc-492a-935e-ec604a0e1ada_isrForCrosstalkSources_855459_6.stderr parsl.dataflow.dflow DEBUG: Task 3954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3954 has memoization hash d03dccba9db98242e99338e607b73773 parsl.dataflow.memoization INFO: Task 3954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2713aa7935b77e7d673e3c5f2ed13581 with result from task 285 parsl.dataflow.dflow INFO: Parsl task 3954 try 0 launched on executor multi with executor id 3955 parsl.dataflow.dflow INFO: Standard output for task 3954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2e0b2885-43bc-4e5f-b848-27eb3fc6f171_isrForCrosstalkSources_855432_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2e0b2885-43bc-4e5f-b848-27eb3fc6f171_isrForCrosstalkSources_855432_50.stderr parsl.dataflow.dflow DEBUG: Task 3955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3955 has memoization hash 98671d16de36b1d55e6837888fab8141 parsl.dataflow.memoization INFO: Task 3955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3955 try 0 launched on executor multi with executor id 3956 parsl.dataflow.dflow INFO: Standard output for task 3955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9dc0b178-7169-4255-b8c2-33486c9b51aa_isrForCrosstalkSources_855426_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9dc0b178-7169-4255-b8c2-33486c9b51aa_isrForCrosstalkSources_855426_38.stderr parsl.dataflow.dflow DEBUG: Task 3956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3956 has memoization hash 15a3b9a67cb60bb629eaaa635c3fb90f parsl.dataflow.memoization INFO: Task 3956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/289d0cef-335c-4b6e-9b9e-cafe867aabc4_isrForCrosstalkSources_855433_38.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/289d0cef-335c-4b6e-9b9e-cafe867aabc4_isrForCrosstalkSources_855433_38.stderr parsl.dataflow.dflow INFO: Parsl task 3956 try 0 launched on executor multi with executor id 3957 parsl.dataflow.dflow INFO: Standard output for task 3956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa4289db-8470-48e5-9438-1d59b77f9d34_isrForCrosstalkSources_855408_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa4289db-8470-48e5-9438-1d59b77f9d34_isrForCrosstalkSources_855408_40.stderr parsl.dataflow.dflow DEBUG: Task 3957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3957 has memoization hash 3730c4ead6fc9119837c5fabf594ebf7 parsl.dataflow.memoization INFO: Task 3957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3957 try 0 launched on executor multi with executor id 3958 parsl.dataflow.dflow INFO: Standard output for task 3957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ce2df744-c55c-45bf-9a49-5735fe0cb65e_isrForCrosstalkSources_855454_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ce2df744-c55c-45bf-9a49-5735fe0cb65e_isrForCrosstalkSources_855454_61.stderr parsl.dataflow.dflow DEBUG: Task 3958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3958 has memoization hash ff5f00299c1e923f149281ff155bc3b3 parsl.dataflow.memoization INFO: Task 3958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3958 try 0 launched on executor multi with executor id 3959 parsl.dataflow.dflow INFO: Standard output for task 3958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bfa080b1-bd6c-4c62-a648-50444f07014e_isrForCrosstalkSources_855458_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bfa080b1-bd6c-4c62-a648-50444f07014e_isrForCrosstalkSources_855458_12.stderr parsl.dataflow.dflow DEBUG: Task 3959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3959 has memoization hash 47abb9d3abbfb62a58058527f3332b11 parsl.dataflow.memoization INFO: Task 3959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3959 try 0 launched on executor multi with executor id 3960 parsl.dataflow.dflow INFO: Standard output for task 3959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/80dbe48d-8d77-45aa-aab7-812798163726_isrForCrosstalkSources_855453_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/80dbe48d-8d77-45aa-aab7-812798163726_isrForCrosstalkSources_855453_55.stderr parsl.dataflow.dflow DEBUG: Task 3960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3960 has memoization hash 42b4a6518a708ffefabcd528fffe9b93 parsl.dataflow.memoization INFO: Task 3960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3960 try 0 launched on executor multi with executor id 3961 parsl.dataflow.dflow INFO: Standard output for task 3960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e524de97-b1b2-4171-aa36-90d820d858d6_isrForCrosstalkSources_855450_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e524de97-b1b2-4171-aa36-90d820d858d6_isrForCrosstalkSources_855450_60.stderr parsl.dataflow.dflow DEBUG: Task 3961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3961 has memoization hash 4023e07a7eda2b191c09eede3576bb80 parsl.dataflow.memoization INFO: Task 3961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc48b3ca2f7fad715b2a839aeba476c5 with result from task 163 parsl.dataflow.dflow INFO: Parsl task 3961 try 0 launched on executor multi with executor id 3962 parsl.dataflow.dflow INFO: Standard output for task 3961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9b1c87d3-bcca-43f9-be5a-802a006ac4a9_isrForCrosstalkSources_855403_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9b1c87d3-bcca-43f9-be5a-802a006ac4a9_isrForCrosstalkSources_855403_57.stderr parsl.dataflow.dflow DEBUG: Task 3962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3962 has memoization hash 9a7b0560dbcfa4bf2d95702ddbc0581e parsl.dataflow.memoization INFO: Task 3962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3962 try 0 launched on executor multi with executor id 3963 parsl.dataflow.dflow INFO: Standard output for task 3962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/82ccee9b-d29e-4e9e-a985-d4001885ef91_isrForCrosstalkSources_855408_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/82ccee9b-d29e-4e9e-a985-d4001885ef91_isrForCrosstalkSources_855408_17.stderr parsl.dataflow.dflow DEBUG: Task 3963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3963 has memoization hash 8561e7147c4c9c6553517d92e786bd27 parsl.dataflow.memoization INFO: Task 3963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3963 try 0 launched on executor multi with executor id 3964 parsl.dataflow.dflow INFO: Standard output for task 3963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/9945782e-a4e6-4185-ae57-d1cc09a7586b_isrForCrosstalkSources_855421_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/9945782e-a4e6-4185-ae57-d1cc09a7586b_isrForCrosstalkSources_855421_47.stderr parsl.dataflow.dflow DEBUG: Task 3964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3964 has memoization hash 251f378a1c86266b4eb45a7ac14c878c parsl.dataflow.memoization INFO: Task 3964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c7ec8907-dbf7-49fa-b0df-6d61a1b23418_isrForCrosstalkSources_855417_53.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c7ec8907-dbf7-49fa-b0df-6d61a1b23418_isrForCrosstalkSources_855417_53.stderr parsl.dataflow.dflow INFO: Parsl task 3964 try 0 launched on executor multi with executor id 3965 parsl.dataflow.dflow INFO: Standard output for task 3964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/166cee37-11b9-4d0e-b4e4-7a2724fe7c47_isrForCrosstalkSources_855398_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/166cee37-11b9-4d0e-b4e4-7a2724fe7c47_isrForCrosstalkSources_855398_21.stderr parsl.dataflow.dflow DEBUG: Task 3965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3965 has memoization hash ac934e66eef5385cfa5951e79514a237 parsl.dataflow.memoization INFO: Task 3965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3965 try 0 launched on executor multi with executor id 3966 parsl.dataflow.dflow INFO: Standard output for task 3965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70e6bc59-2b56-469f-96d5-7b8079b62fb2_isrForCrosstalkSources_855443_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70e6bc59-2b56-469f-96d5-7b8079b62fb2_isrForCrosstalkSources_855443_57.stderr parsl.dataflow.dflow DEBUG: Task 3966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3966 has memoization hash 5539970ccf2653011dd1b74bd25fa6cf parsl.dataflow.memoization INFO: Task 3966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3966 try 0 launched on executor multi with executor id 3967 parsl.dataflow.dflow INFO: Standard output for task 3966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c2ccc096-3b4c-4874-9a07-c288bd5ea0f7_isrForCrosstalkSources_855408_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c2ccc096-3b4c-4874-9a07-c288bd5ea0f7_isrForCrosstalkSources_855408_21.stderr parsl.dataflow.dflow DEBUG: Task 3967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3967 has memoization hash fe8780798b0c1e13482e0bdfd84f3005 parsl.dataflow.memoization INFO: Task 3967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3967 try 0 launched on executor multi with executor id 3968 parsl.dataflow.dflow INFO: Standard output for task 3967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/740bda14-1af6-417d-9acf-e7cd23b8d8d3_isrForCrosstalkSources_855432_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/740bda14-1af6-417d-9acf-e7cd23b8d8d3_isrForCrosstalkSources_855432_38.stderr parsl.dataflow.dflow DEBUG: Task 3968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3968 has memoization hash 29099d9babb91388b180acacc0aaee46 parsl.dataflow.memoization INFO: Task 3968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3968 try 0 launched on executor multi with executor id 3969 parsl.dataflow.dflow INFO: Standard output for task 3968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8b3f2487-e26f-4ec2-930b-d18af10aeeaa_isrForCrosstalkSources_855442_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8b3f2487-e26f-4ec2-930b-d18af10aeeaa_isrForCrosstalkSources_855442_24.stderr parsl.dataflow.dflow DEBUG: Task 3969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3969 has memoization hash 49769807634be64795c5dbeb855a5a23 parsl.dataflow.memoization INFO: Task 3969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3969 try 0 launched on executor multi with executor id 3970 parsl.dataflow.dflow INFO: Standard output for task 3969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d9a86474-5dba-49da-beb2-6aab423a15d6_isrForCrosstalkSources_855407_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d9a86474-5dba-49da-beb2-6aab423a15d6_isrForCrosstalkSources_855407_23.stderr parsl.dataflow.dflow DEBUG: Task 3970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3970 has memoization hash def8e1216416e9615783a27df8da1431 parsl.dataflow.memoization INFO: Task 3970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3970 try 0 launched on executor multi with executor id 3971 parsl.dataflow.dflow INFO: Standard output for task 3970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/3fc607b1-499f-4b66-82f9-8e1dc59709a0_isrForCrosstalkSources_855439_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/3fc607b1-499f-4b66-82f9-8e1dc59709a0_isrForCrosstalkSources_855439_28.stderr parsl.dataflow.dflow DEBUG: Task 3971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3971 has memoization hash 9de25ed3198ec678d63a4b23ac252ee9 parsl.dataflow.memoization INFO: Task 3971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3971 try 0 launched on executor multi with executor id 3972 parsl.dataflow.dflow INFO: Standard output for task 3971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/71c037d1-acd1-4c43-b4f2-8c21e26902ae_isrForCrosstalkSources_855393_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/71c037d1-acd1-4c43-b4f2-8c21e26902ae_isrForCrosstalkSources_855393_41.stderr parsl.dataflow.dflow DEBUG: Task 3972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3972 has memoization hash 805186fc033bba8f5a24c1d4666b4822 parsl.dataflow.memoization INFO: Task 3972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3972 try 0 launched on executor multi with executor id 3973 parsl.dataflow.dflow INFO: Standard output for task 3972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a4bf2cda-7e05-4861-9e9b-aabcf20121b2_isrForCrosstalkSources_855422_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a4bf2cda-7e05-4861-9e9b-aabcf20121b2_isrForCrosstalkSources_855422_51.stderr parsl.dataflow.dflow DEBUG: Task 3973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3973 has memoization hash 09fc10bef0b34c8581e726db5f46db46 parsl.dataflow.memoization INFO: Task 3973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3973 try 0 launched on executor multi with executor id 3974 parsl.dataflow.dflow INFO: Standard output for task 3973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/96978879-0f53-43cd-b2a8-ebe50541465d_isrForCrosstalkSources_855387_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/96978879-0f53-43cd-b2a8-ebe50541465d_isrForCrosstalkSources_855387_34.stderr parsl.dataflow.dflow DEBUG: Task 3974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3974 has memoization hash b128ee80558e9819a06d08931b309535 parsl.dataflow.memoization INFO: Task 3974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3974 try 0 launched on executor multi with executor id 3975 parsl.dataflow.dflow INFO: Standard output for task 3974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ab4de581-290c-4b93-9830-fade76cb882d_isrForCrosstalkSources_855414_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ab4de581-290c-4b93-9830-fade76cb882d_isrForCrosstalkSources_855414_10.stderr parsl.dataflow.dflow DEBUG: Task 3975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3975 has memoization hash 99a837bddf62243a8e45db64c90ff1eb parsl.dataflow.memoization INFO: Task 3975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:43 proc_lsst.multi:170 [INFO] [multi] job_id 16964178 proc_lsst.multi INFO: [multi] job_id 16964178 2024-03-09 14:51:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-09 14:51:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964178 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964178 parsl.executors.status_handling DEBUG: Launched block 152 on executor multi with job ID 16964178 parsl.executors.status_handling INFO: Allocated block ID 153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-153 2024-03-09 14:51:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3975 try 0 launched on executor multi with executor id 3976 parsl.dataflow.dflow INFO: Standard output for task 3975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4cb62395-ab44-4e85-8e83-100b7604f0b9_isrForCrosstalkSources_855392_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4cb62395-ab44-4e85-8e83-100b7604f0b9_isrForCrosstalkSources_855392_4.stderr parsl.dataflow.dflow DEBUG: Task 3976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3976 has memoization hash 90b3a09db1dc2b48e05ce677b6c1c63f parsl.dataflow.memoization INFO: Task 3976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3976 try 0 launched on executor multi with executor id 3977 parsl.dataflow.dflow INFO: Standard output for task 3976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e729e89a-ab47-426f-a879-d857fe595146_isrForCrosstalkSources_855458_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e729e89a-ab47-426f-a879-d857fe595146_isrForCrosstalkSources_855458_17.stderr parsl.dataflow.dflow DEBUG: Task 3977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3977 has memoization hash 1c3b0ec620f02383effcf1c158046ff8 parsl.dataflow.memoization INFO: Task 3977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3977 try 0 launched on executor multi with executor id 3978 parsl.dataflow.dflow INFO: Standard output for task 3977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5391bd99-0992-489c-8dca-c6c92e48a00f_isrForCrosstalkSources_855419_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5391bd99-0992-489c-8dca-c6c92e48a00f_isrForCrosstalkSources_855419_6.stderr parsl.dataflow.dflow DEBUG: Task 3978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3978 has memoization hash eb4b45561618e05577d6cccbb5727a62 parsl.dataflow.memoization INFO: Task 3978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3978 try 0 launched on executor multi with executor id 3979 parsl.dataflow.dflow INFO: Standard output for task 3978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/076b6598-1f64-481f-8a27-af285d49a416_isrForCrosstalkSources_855392_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/076b6598-1f64-481f-8a27-af285d49a416_isrForCrosstalkSources_855392_26.stderr parsl.dataflow.dflow DEBUG: Task 3979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3979 has memoization hash a49717835f3a5fa1e30565382b3c5537 parsl.dataflow.memoization INFO: Task 3979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3979 try 0 launched on executor multi with executor id 3980 parsl.dataflow.dflow INFO: Standard output for task 3979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c792a03-eac7-4db0-8b01-5fcfb5ce778a_isrForCrosstalkSources_855435_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c792a03-eac7-4db0-8b01-5fcfb5ce778a_isrForCrosstalkSources_855435_19.stderr parsl.dataflow.dflow DEBUG: Task 3980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3980 has memoization hash b6a6ed9969d505a1192607a1f2439f61 parsl.dataflow.memoization INFO: Task 3980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3980 try 0 launched on executor multi with executor id 3981 parsl.dataflow.dflow INFO: Standard output for task 3980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8f81c688-6b99-4cb1-8c53-d91f5165ff6e_isrForCrosstalkSources_855400_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8f81c688-6b99-4cb1-8c53-d91f5165ff6e_isrForCrosstalkSources_855400_9.stderr parsl.dataflow.dflow DEBUG: Task 3981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3981 has memoization hash 2170dd57aa0d6c334bc6530daf1cd3d8 parsl.dataflow.memoization INFO: Task 3981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6912b1a9d43ef85be2d46469b7d37ac7 with result from task 166 parsl.dataflow.dflow INFO: Parsl task 3981 try 0 launched on executor multi with executor id 3982 parsl.dataflow.dflow INFO: Standard output for task 3981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/791c04f5-8d98-4666-bb47-a74adc1ab1d8_isrForCrosstalkSources_855439_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/791c04f5-8d98-4666-bb47-a74adc1ab1d8_isrForCrosstalkSources_855439_3.stderr parsl.dataflow.dflow DEBUG: Task 3982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3982 has memoization hash 5e41bde3fe733b2598b5ce9466ca84f5 parsl.dataflow.memoization INFO: Task 3982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3982 try 0 launched on executor multi with executor id 3983 parsl.dataflow.dflow INFO: Standard output for task 3982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6efab700-da07-4750-b504-8d8d5987aa8b_isrForCrosstalkSources_855395_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6efab700-da07-4750-b504-8d8d5987aa8b_isrForCrosstalkSources_855395_18.stderr parsl.dataflow.dflow DEBUG: Task 3983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3983 has memoization hash 3d2a69a3dd76aecf5b4ffd90cb5e801d parsl.dataflow.memoization INFO: Task 3983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ad99dc1d-6480-4821-9bc4-3dc27e64c2e9_isrForCrosstalkSources_855447_1.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ad99dc1d-6480-4821-9bc4-3dc27e64c2e9_isrForCrosstalkSources_855447_1.stderr parsl.dataflow.dflow INFO: Parsl task 3983 try 0 launched on executor multi with executor id 3984 parsl.dataflow.dflow INFO: Standard output for task 3983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/89379901-8f88-41dc-807f-e54899c35632_isrForCrosstalkSources_855458_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/89379901-8f88-41dc-807f-e54899c35632_isrForCrosstalkSources_855458_34.stderr parsl.dataflow.dflow DEBUG: Task 3984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3984 has memoization hash 4fbe2ddb9121fbb5a838bac8f78028ca parsl.dataflow.memoization INFO: Task 3984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3984 try 0 launched on executor multi with executor id 3985 parsl.dataflow.dflow INFO: Standard output for task 3984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/56c6eeb3-d9e4-4777-a8df-f03f79652968_isrForCrosstalkSources_855430_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/56c6eeb3-d9e4-4777-a8df-f03f79652968_isrForCrosstalkSources_855430_10.stderr parsl.dataflow.dflow DEBUG: Task 3985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3985 has memoization hash cab35d818ed127d520bc65cb6e2d9811 parsl.dataflow.memoization INFO: Task 3985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3985 try 0 launched on executor multi with executor id 3986 parsl.dataflow.dflow INFO: Standard output for task 3985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/97658274-4fc6-4ca2-97a1-a89f49d40002_isrForCrosstalkSources_855418_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/97658274-4fc6-4ca2-97a1-a89f49d40002_isrForCrosstalkSources_855418_21.stderr parsl.dataflow.dflow DEBUG: Task 3986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3986 has memoization hash 4cdaa4dc3ab130c70af4c30fa78ea74b parsl.dataflow.memoization INFO: Task 3986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3986 try 0 launched on executor multi with executor id 3987 parsl.dataflow.dflow INFO: Standard output for task 3986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b1813780-e950-419a-812a-e2d6c378d83c_isrForCrosstalkSources_855423_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b1813780-e950-419a-812a-e2d6c378d83c_isrForCrosstalkSources_855423_26.stderr parsl.dataflow.dflow DEBUG: Task 3987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3987 has memoization hash 505c861711ce97b4ec871b8938e3b07d parsl.dataflow.memoization INFO: Task 3987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3987 try 0 launched on executor multi with executor id 3988 parsl.dataflow.dflow INFO: Standard output for task 3987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b56d1927-06df-4dc5-bad0-dc2fb1c876f1_isrForCrosstalkSources_855438_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b56d1927-06df-4dc5-bad0-dc2fb1c876f1_isrForCrosstalkSources_855438_45.stderr parsl.dataflow.dflow DEBUG: Task 3988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3988 has memoization hash 9727494c08b6bd7d877c8f9cb8ac44e5 parsl.dataflow.memoization INFO: Task 3988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3988 try 0 launched on executor multi with executor id 3989 parsl.dataflow.dflow INFO: Standard output for task 3988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/aedbb355-0b54-4ecd-a2c6-848845d0eeba_isrForCrosstalkSources_855415_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/aedbb355-0b54-4ecd-a2c6-848845d0eeba_isrForCrosstalkSources_855415_17.stderr parsl.dataflow.dflow DEBUG: Task 3989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3989 has memoization hash 7297608d04f9516f0ffdd499eef7ae6f parsl.dataflow.memoization INFO: Task 3989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3989 try 0 launched on executor multi with executor id 3990 parsl.dataflow.dflow INFO: Standard output for task 3989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/81f20587-de80-4a88-b5a0-a3b14d7d9101_isrForCrosstalkSources_855410_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/81f20587-de80-4a88-b5a0-a3b14d7d9101_isrForCrosstalkSources_855410_45.stderr parsl.dataflow.dflow DEBUG: Task 3990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3990 has memoization hash 371c749253c15200da39a57415c2d1ea parsl.dataflow.memoization INFO: Task 3990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28a92f70593430d815e99bbeb7030999 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/77e17cde-47b9-4825-bf7c-af3a0063c465_isrForCrosstalkSources_855411_52.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/77e17cde-47b9-4825-bf7c-af3a0063c465_isrForCrosstalkSources_855411_52.stderr parsl.dataflow.dflow INFO: Parsl task 3990 try 0 launched on executor multi with executor id 3991 parsl.dataflow.dflow INFO: Standard output for task 3990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/517c74c4-aa07-4dd9-8a0d-d93f7b4af689_isrForCrosstalkSources_855386_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/517c74c4-aa07-4dd9-8a0d-d93f7b4af689_isrForCrosstalkSources_855386_5.stderr parsl.dataflow.dflow DEBUG: Task 3991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3991 has memoization hash 0d14f20e1801729bfb3e5c749b7b0a7f parsl.dataflow.memoization INFO: Task 3991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3991 try 0 launched on executor multi with executor id 3992 parsl.dataflow.dflow INFO: Standard output for task 3991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/76fadd4b-0e29-46ae-852b-279f8748afda_isrForCrosstalkSources_855408_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/76fadd4b-0e29-46ae-852b-279f8748afda_isrForCrosstalkSources_855408_31.stderr parsl.dataflow.dflow DEBUG: Task 3992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3992 has memoization hash dc6094d77770b3697707046100d64cd5 parsl.dataflow.memoization INFO: Task 3992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3992 try 0 launched on executor multi with executor id 3993 parsl.dataflow.dflow INFO: Standard output for task 3992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/47dcad13-5bfd-4a0d-b8d2-204b2d4c94ee_isrForCrosstalkSources_855400_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/47dcad13-5bfd-4a0d-b8d2-204b2d4c94ee_isrForCrosstalkSources_855400_29.stderr parsl.dataflow.dflow DEBUG: Task 3993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3993 has memoization hash 0ede12d30acb1917e9f65a8f20c14325 parsl.dataflow.memoization INFO: Task 3993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3993 try 0 launched on executor multi with executor id 3994 parsl.dataflow.dflow INFO: Standard output for task 3993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1668ffa8-dbba-449d-865e-e95ce7814081_isrForCrosstalkSources_855450_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1668ffa8-dbba-449d-865e-e95ce7814081_isrForCrosstalkSources_855450_52.stderr parsl.dataflow.dflow DEBUG: Task 3994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3994 has memoization hash 46784337868b998096478a70f62d496b parsl.dataflow.memoization INFO: Task 3994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62da8ba7ca5ee5c1470c4d6cca44fc48 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c44e0032-3ed4-48bc-93f3-fd2f6c893ded_isrForCrosstalkSources_855387_7.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c44e0032-3ed4-48bc-93f3-fd2f6c893ded_isrForCrosstalkSources_855387_7.stderr parsl.dataflow.dflow INFO: Parsl task 3994 try 0 launched on executor multi with executor id 3995 parsl.dataflow.dflow INFO: Standard output for task 3994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee114502-24bb-4025-87a7-e0cd13566410_isrForCrosstalkSources_855443_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee114502-24bb-4025-87a7-e0cd13566410_isrForCrosstalkSources_855443_19.stderr parsl.dataflow.dflow DEBUG: Task 3995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3995 has memoization hash 3337740f1afba9b32c99b163ee677d07 parsl.dataflow.memoization INFO: Task 3995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3995 try 0 launched on executor multi with executor id 3996 parsl.dataflow.dflow INFO: Standard output for task 3995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fbe00247-6794-4f96-9f78-cb39e16be885_isrForCrosstalkSources_855458_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fbe00247-6794-4f96-9f78-cb39e16be885_isrForCrosstalkSources_855458_28.stderr parsl.dataflow.dflow DEBUG: Task 3996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3996 has memoization hash 59f3326a13413b0ba620f91b0823065b parsl.dataflow.memoization INFO: Task 3996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3996 try 0 launched on executor multi with executor id 3997 parsl.dataflow.dflow INFO: Standard output for task 3996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/57970e2b-5960-47c2-a88b-c11bb769a74c_isrForCrosstalkSources_855408_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/57970e2b-5960-47c2-a88b-c11bb769a74c_isrForCrosstalkSources_855408_53.stderr parsl.dataflow.dflow DEBUG: Task 3997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3997 has memoization hash 3054501da97d685350de31c1512c5ea2 parsl.dataflow.memoization INFO: Task 3997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3997 try 0 launched on executor multi with executor id 3998 parsl.dataflow.dflow INFO: Standard output for task 3997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6ea8a8b8-3371-481b-af4b-3bfc920fa484_isrForCrosstalkSources_855452_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6ea8a8b8-3371-481b-af4b-3bfc920fa484_isrForCrosstalkSources_855452_57.stderr parsl.dataflow.dflow DEBUG: Task 3998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3998 has memoization hash 3d969036b297bfbfadbc68bcd9029183 parsl.dataflow.memoization INFO: Task 3998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3998 try 0 launched on executor multi with executor id 3999 parsl.dataflow.dflow INFO: Standard output for task 3998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7fa07982-0e76-4e19-ac8d-9a0376f87498_isrForCrosstalkSources_855390_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7fa07982-0e76-4e19-ac8d-9a0376f87498_isrForCrosstalkSources_855390_48.stderr parsl.dataflow.dflow DEBUG: Task 3999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3999 has memoization hash c08ca8736a733a04fecd1268636b29ed parsl.dataflow.memoization INFO: Task 3999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3999 try 0 launched on executor multi with executor id 4000 parsl.dataflow.dflow INFO: Standard output for task 3999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/759fa897-4544-40b3-9a9d-85814b265fcc_isrForCrosstalkSources_855455_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/759fa897-4544-40b3-9a9d-85814b265fcc_isrForCrosstalkSources_855455_5.stderr parsl.dataflow.dflow DEBUG: Task 4000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4000 has memoization hash 70a255a254e825739fc6871cc2ca3dec parsl.dataflow.memoization INFO: Task 4000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4000 try 0 launched on executor multi with executor id 4001 parsl.dataflow.dflow INFO: Standard output for task 4000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d368e4b6-77f0-4ac0-b2f2-3d57d54b8f5f_isrForCrosstalkSources_855448_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d368e4b6-77f0-4ac0-b2f2-3d57d54b8f5f_isrForCrosstalkSources_855448_4.stderr parsl.dataflow.dflow DEBUG: Task 4001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4001 has memoization hash d371d6291560ca4fd5c1510d079e80b7 parsl.dataflow.memoization INFO: Task 4001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4001 try 0 launched on executor multi with executor id 4002 parsl.dataflow.dflow INFO: Standard output for task 4001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85e75c3c-ed7d-4128-8970-cb9a4651dc36_isrForCrosstalkSources_855413_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85e75c3c-ed7d-4128-8970-cb9a4651dc36_isrForCrosstalkSources_855413_24.stderr parsl.dataflow.dflow DEBUG: Task 4002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4002 has memoization hash 7ef08c6630c19b77e266fb2ae543f346 parsl.dataflow.memoization INFO: Task 4002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4002 try 0 launched on executor multi with executor id 4003 parsl.dataflow.dflow INFO: Standard output for task 4002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6536cba7-e5f8-4c99-8d9e-2730fb2000b7_isrForCrosstalkSources_855431_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6536cba7-e5f8-4c99-8d9e-2730fb2000b7_isrForCrosstalkSources_855431_49.stderr parsl.dataflow.dflow DEBUG: Task 4003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4003 has memoization hash 6cc373bf92e40c76423c33364ffad94d parsl.dataflow.memoization INFO: Task 4003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4003 try 0 launched on executor multi with executor id 4004 parsl.dataflow.dflow INFO: Standard output for task 4003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918a470-adad-4cd3-bd3c-fc7950ad7ec7_isrForCrosstalkSources_855404_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918a470-adad-4cd3-bd3c-fc7950ad7ec7_isrForCrosstalkSources_855404_10.stderr parsl.dataflow.dflow DEBUG: Task 4004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4004 has memoization hash 455973d8a35e8d0022041c3ba38cc5d7 parsl.dataflow.memoization INFO: Task 4004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4004 try 0 launched on executor multi with executor id 4005 parsl.dataflow.dflow INFO: Standard output for task 4004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/4afa071f-5295-4690-ac67-ea005706b10c_isrForCrosstalkSources_855447_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/4afa071f-5295-4690-ac67-ea005706b10c_isrForCrosstalkSources_855447_20.stderr parsl.dataflow.dflow DEBUG: Task 4005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4005 has memoization hash 814232a1a11c84f460958a26ca054629 parsl.dataflow.memoization INFO: Task 4005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4005 try 0 launched on executor multi with executor id 4006 parsl.dataflow.dflow INFO: Standard output for task 4005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/25f38ffe-2c82-4fcd-81b9-d97765cf6745_isrForCrosstalkSources_855455_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/25f38ffe-2c82-4fcd-81b9-d97765cf6745_isrForCrosstalkSources_855455_58.stderr parsl.dataflow.dflow DEBUG: Task 4006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4006 has memoization hash 845f15a40a5262f37370b2d6e96288e5 parsl.dataflow.memoization INFO: Task 4006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4006 try 0 launched on executor multi with executor id 4007 parsl.dataflow.dflow INFO: Standard output for task 4006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c3bc1176-1481-4e57-9c74-8722ead872cd_isrForCrosstalkSources_855385_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c3bc1176-1481-4e57-9c74-8722ead872cd_isrForCrosstalkSources_855385_10.stderr parsl.dataflow.dflow DEBUG: Task 4007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4007 has memoization hash e78ab654725b2e2bee189476de5e5823 parsl.dataflow.memoization INFO: Task 4007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4007 try 0 launched on executor multi with executor id 4008 parsl.dataflow.dflow INFO: Standard output for task 4007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e35b501d-6e54-4881-963b-12c99d41a179_isrForCrosstalkSources_855439_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e35b501d-6e54-4881-963b-12c99d41a179_isrForCrosstalkSources_855439_9.stderr parsl.dataflow.dflow DEBUG: Task 4008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4008 has memoization hash 489c12e23402f44ce6111bf805510cca parsl.dataflow.memoization INFO: Task 4008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4008 try 0 launched on executor multi with executor id 4009 parsl.dataflow.dflow INFO: Standard output for task 4008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1d0a8981-5492-40ab-99a0-84bec853605e_isrForCrosstalkSources_855441_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1d0a8981-5492-40ab-99a0-84bec853605e_isrForCrosstalkSources_855441_28.stderr parsl.dataflow.dflow DEBUG: Task 4009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4009 has memoization hash 1dd4d7aa6b1b0f7bc6a36a016256c465 parsl.dataflow.memoization INFO: Task 4009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4009 try 0 launched on executor multi with executor id 4010 parsl.dataflow.dflow INFO: Standard output for task 4009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/36b9ce7f-487a-44ae-9773-dd525c7ec042_isrForCrosstalkSources_855430_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/36b9ce7f-487a-44ae-9773-dd525c7ec042_isrForCrosstalkSources_855430_16.stderr parsl.dataflow.dflow DEBUG: Task 4010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4010 has memoization hash 3849ee5974f96ecdb186431651ec0eba parsl.dataflow.memoization INFO: Task 4010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4010 try 0 launched on executor multi with executor id 4011 parsl.dataflow.dflow INFO: Standard output for task 4010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6607d03f-7c79-4d3e-8a7a-441a6b8dfbb9_isrForCrosstalkSources_855392_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6607d03f-7c79-4d3e-8a7a-441a6b8dfbb9_isrForCrosstalkSources_855392_13.stderr parsl.dataflow.dflow DEBUG: Task 4011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4011 has memoization hash 207be426b078f85d24d1296fed089a42 parsl.dataflow.memoization INFO: Task 4011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4011 try 0 launched on executor multi with executor id 4012 parsl.dataflow.dflow INFO: Standard output for task 4011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e0aa0099-1dda-4afa-976f-25a9ebb38dc9_isrForCrosstalkSources_855422_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e0aa0099-1dda-4afa-976f-25a9ebb38dc9_isrForCrosstalkSources_855422_23.stderr parsl.dataflow.dflow DEBUG: Task 4012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4012 has memoization hash 3c364ce1ca3e12eb6392adb9cf97e1b2 parsl.dataflow.memoization INFO: Task 4012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4012 try 0 launched on executor multi with executor id 4013 parsl.dataflow.dflow INFO: Standard output for task 4012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9cc0025c-405a-4aeb-8d70-2da3c3e40e5a_isrForCrosstalkSources_855434_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9cc0025c-405a-4aeb-8d70-2da3c3e40e5a_isrForCrosstalkSources_855434_61.stderr parsl.dataflow.dflow DEBUG: Task 4013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4013 has memoization hash 840dbadce3cd9b8b1d44408c6b564ba4 parsl.dataflow.memoization INFO: Task 4013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4013 try 0 launched on executor multi with executor id 4014 parsl.dataflow.dflow INFO: Standard output for task 4013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/53385777-37ab-49b6-b581-2fea9ded981c_isrForCrosstalkSources_855404_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/53385777-37ab-49b6-b581-2fea9ded981c_isrForCrosstalkSources_855404_41.stderr parsl.dataflow.dflow DEBUG: Task 4014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4014 has memoization hash 9aec777a3f60b9d7d293c349e8bc2196 parsl.dataflow.memoization INFO: Task 4014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4014 try 0 launched on executor multi with executor id 4015 parsl.dataflow.dflow INFO: Standard output for task 4014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/83ebaeea-3a91-4f0a-91c4-5139fcafed75_isrForCrosstalkSources_855397_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/83ebaeea-3a91-4f0a-91c4-5139fcafed75_isrForCrosstalkSources_855397_46.stderr parsl.dataflow.dflow DEBUG: Task 4015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4015 has memoization hash 52752e7ee0158cd810257df749e02064 parsl.dataflow.memoization INFO: Task 4015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4015 try 0 launched on executor multi with executor id 4016 parsl.dataflow.dflow INFO: Standard output for task 4015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/05cfa153-36dd-4904-9570-3acb4d701dd3_isrForCrosstalkSources_855413_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/05cfa153-36dd-4904-9570-3acb4d701dd3_isrForCrosstalkSources_855413_35.stderr parsl.dataflow.dflow DEBUG: Task 4016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4016 has memoization hash 264e49444ae25af0fc3f5cbea935e872 parsl.dataflow.memoization INFO: Task 4016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4016 try 0 launched on executor multi with executor id 4017 parsl.dataflow.dflow INFO: Standard output for task 4016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ef40cbe9-28d7-43c0-acca-861a510ec7c1_isrForCrosstalkSources_855459_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ef40cbe9-28d7-43c0-acca-861a510ec7c1_isrForCrosstalkSources_855459_22.stderr parsl.dataflow.dflow DEBUG: Task 4017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4017 has memoization hash e4d1af51e8e2efe4ee2525d05636d828 parsl.dataflow.memoization INFO: Task 4017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4017 try 0 launched on executor multi with executor id 4018 parsl.dataflow.dflow INFO: Standard output for task 4017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/45200f85-6923-4ab7-aaea-95f8f3519044_isrForCrosstalkSources_855429_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/45200f85-6923-4ab7-aaea-95f8f3519044_isrForCrosstalkSources_855429_34.stderr parsl.dataflow.dflow DEBUG: Task 4018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4018 has memoization hash 9b6a6c05f3b4e524a484041266879915 parsl.dataflow.memoization INFO: Task 4018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4018 try 0 launched on executor multi with executor id 4019 parsl.dataflow.dflow INFO: Standard output for task 4018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/55332c2d-537a-4892-ac4c-1711c06e2178_isrForCrosstalkSources_855445_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/55332c2d-537a-4892-ac4c-1711c06e2178_isrForCrosstalkSources_855445_16.stderr parsl.dataflow.dflow DEBUG: Task 4019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4019 has memoization hash eaf6ef1f96eff1e900c722a5b5a8b1a8 parsl.dataflow.memoization INFO: Task 4019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4019 try 0 launched on executor multi with executor id 4020 parsl.dataflow.dflow INFO: Standard output for task 4019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/12156752-df13-43c5-8980-fbcf4de647f6_isrForCrosstalkSources_855410_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/12156752-df13-43c5-8980-fbcf4de647f6_isrForCrosstalkSources_855410_39.stderr parsl.dataflow.dflow DEBUG: Task 4020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4020 has memoization hash 50e02ff0554d61c71706613c4bd20ed9 parsl.dataflow.memoization INFO: Task 4020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4020 try 0 launched on executor multi with executor id 4021 parsl.dataflow.dflow INFO: Standard output for task 4020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0780edad-64a7-402b-b6e7-bed3e064e4c7_isrForCrosstalkSources_855420_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0780edad-64a7-402b-b6e7-bed3e064e4c7_isrForCrosstalkSources_855420_39.stderr parsl.dataflow.dflow DEBUG: Task 4021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4021 has memoization hash b6e8af2a78d13fe7ce162cf879cc5591 parsl.dataflow.memoization INFO: Task 4021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4021 try 0 launched on executor multi with executor id 4022 parsl.dataflow.dflow INFO: Standard output for task 4021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/4bda4d30-3e98-4628-acfc-4d52cf4a34d2_isrForCrosstalkSources_855397_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/4bda4d30-3e98-4628-acfc-4d52cf4a34d2_isrForCrosstalkSources_855397_59.stderr parsl.dataflow.dflow DEBUG: Task 4022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4022 has memoization hash 46088bbcf597ba9cae3ded39b41c14c9 parsl.dataflow.memoization INFO: Task 4022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4022 try 0 launched on executor multi with executor id 4023 parsl.dataflow.dflow INFO: Standard output for task 4022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8d208a-3c01-44fc-abff-b213ff883e44_isrForCrosstalkSources_855404_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8d208a-3c01-44fc-abff-b213ff883e44_isrForCrosstalkSources_855404_60.stderr parsl.dataflow.dflow DEBUG: Task 4023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4023 has memoization hash 78c3bf2238724ba5f62afa755bff46f3 parsl.dataflow.memoization INFO: Task 4023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4023 try 0 launched on executor multi with executor id 4024 parsl.dataflow.dflow INFO: Standard output for task 4023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/68a54f7e-0d75-4e83-839a-61c9b30ff1f9_isrForCrosstalkSources_855392_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/68a54f7e-0d75-4e83-839a-61c9b30ff1f9_isrForCrosstalkSources_855392_1.stderr parsl.dataflow.dflow DEBUG: Task 4024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4024 has memoization hash 97f306ecbc0da7aa49ca034aa52c24cf parsl.dataflow.memoization INFO: Task 4024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66abb658593a5341bea37b59b43b231c with result from task 326 2024-03-09 14:51:43 proc_lsst.multi:170 [INFO] [multi] job_id 16964179 proc_lsst.multi INFO: [multi] job_id 16964179 2024-03-09 14:51:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-09 14:51:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964179 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964179 parsl.executors.status_handling DEBUG: Launched block 153 on executor multi with job ID 16964179 parsl.executors.status_handling INFO: Allocated block ID 154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-154 2024-03-09 14:51:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-09 14:51:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4024 try 0 launched on executor multi with executor id 4025 parsl.dataflow.dflow INFO: Standard output for task 4024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70b0af10-085b-48e7-9888-19d77e079edc_isrForCrosstalkSources_855443_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70b0af10-085b-48e7-9888-19d77e079edc_isrForCrosstalkSources_855443_37.stderr parsl.dataflow.dflow DEBUG: Task 4025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4025 has memoization hash 2107bc0cb85acb0e2ac0a9af7e8e2d6a parsl.dataflow.memoization INFO: Task 4025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0c9bfa7d-3230-412c-ae98-3ffdeb977eb8_isrForCrosstalkSources_855444_54.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0c9bfa7d-3230-412c-ae98-3ffdeb977eb8_isrForCrosstalkSources_855444_54.stderr parsl.dataflow.dflow INFO: Parsl task 4025 try 0 launched on executor multi with executor id 4026 parsl.dataflow.dflow INFO: Standard output for task 4025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/231b3e97-7c13-4278-a41a-29c6e9fe526e_isrForCrosstalkSources_855448_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/231b3e97-7c13-4278-a41a-29c6e9fe526e_isrForCrosstalkSources_855448_20.stderr parsl.dataflow.dflow DEBUG: Task 4026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4026 has memoization hash 7f9f576d0ea42398cab6c08022c09a21 parsl.dataflow.memoization INFO: Task 4026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4026 try 0 launched on executor multi with executor id 4027 parsl.dataflow.dflow INFO: Standard output for task 4026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/88e93d8a-955a-437c-93f6-0596b373ca46_isrForCrosstalkSources_855447_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/88e93d8a-955a-437c-93f6-0596b373ca46_isrForCrosstalkSources_855447_35.stderr parsl.dataflow.dflow DEBUG: Task 4027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4027 has memoization hash ffb8d64a9e8bfdc161c9e2dd894afdb6 parsl.dataflow.memoization INFO: Task 4027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4027 try 0 launched on executor multi with executor id 4028 parsl.dataflow.dflow INFO: Standard output for task 4027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eadc066-99d3-463f-802f-e217c2513453_isrForCrosstalkSources_855434_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eadc066-99d3-463f-802f-e217c2513453_isrForCrosstalkSources_855434_32.stderr parsl.dataflow.dflow DEBUG: Task 4028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4028 has memoization hash 5c6d1d1c0db33aa769fb9eccd7551993 parsl.dataflow.memoization INFO: Task 4028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4028 try 0 launched on executor multi with executor id 4029 parsl.dataflow.dflow INFO: Standard output for task 4028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/cb79bdfe-3abe-40f3-9b26-0cee6d6fde9c_isrForCrosstalkSources_855383_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/cb79bdfe-3abe-40f3-9b26-0cee6d6fde9c_isrForCrosstalkSources_855383_33.stderr parsl.dataflow.dflow DEBUG: Task 4029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4029 has memoization hash 80acb71d63a5273a7cd9957c4f2991a0 parsl.dataflow.memoization INFO: Task 4029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4029 try 0 launched on executor multi with executor id 4030 parsl.dataflow.dflow INFO: Standard output for task 4029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/431b91d8-da30-4e33-9e6b-0ae6b79174c4_isrForCrosstalkSources_855384_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/431b91d8-da30-4e33-9e6b-0ae6b79174c4_isrForCrosstalkSources_855384_25.stderr parsl.dataflow.dflow DEBUG: Task 4030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4030 has memoization hash 448ee219aab8bad2780e100b43eeb2c8 parsl.dataflow.memoization INFO: Task 4030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4030 try 0 launched on executor multi with executor id 4031 parsl.dataflow.dflow INFO: Standard output for task 4030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73c542aa-0542-4e52-ad58-d5c54d9af3c6_isrForCrosstalkSources_855444_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73c542aa-0542-4e52-ad58-d5c54d9af3c6_isrForCrosstalkSources_855444_27.stderr parsl.dataflow.dflow DEBUG: Task 4031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4031 has memoization hash 9d59ee4dd848deb61a049f2cdb612826 parsl.dataflow.memoization INFO: Task 4031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4031 try 0 launched on executor multi with executor id 4032 parsl.dataflow.dflow INFO: Standard output for task 4031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ba2cc7-f2ae-481d-9a7b-32c70a702556_isrForCrosstalkSources_855396_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ba2cc7-f2ae-481d-9a7b-32c70a702556_isrForCrosstalkSources_855396_26.stderr parsl.dataflow.dflow DEBUG: Task 4032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4032 has memoization hash 8077113e031aaf1e3530981143308cb7 parsl.dataflow.memoization INFO: Task 4032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4032 try 0 launched on executor multi with executor id 4033 parsl.dataflow.dflow INFO: Standard output for task 4032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9ce74088-1fef-438d-8708-0eee8edac9ea_isrForCrosstalkSources_855399_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9ce74088-1fef-438d-8708-0eee8edac9ea_isrForCrosstalkSources_855399_11.stderr parsl.dataflow.dflow DEBUG: Task 4033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4033 has memoization hash c2594ca3832ef7c8cf17f298a757b3d4 parsl.dataflow.memoization INFO: Task 4033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4033 try 0 launched on executor multi with executor id 4034 parsl.dataflow.dflow INFO: Standard output for task 4033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/60036fcf-0b99-4870-bc54-18afd82022ee_isrForCrosstalkSources_855392_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/60036fcf-0b99-4870-bc54-18afd82022ee_isrForCrosstalkSources_855392_29.stderr parsl.dataflow.dflow DEBUG: Task 4034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4034 has memoization hash 210f4becc088b64e97ba8085d564aa67 parsl.dataflow.memoization INFO: Task 4034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4034 try 0 launched on executor multi with executor id 4035 parsl.dataflow.dflow INFO: Standard output for task 4034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/14f2cad6-bff2-4adb-a3af-43daec8d0fc0_isrForCrosstalkSources_855439_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/14f2cad6-bff2-4adb-a3af-43daec8d0fc0_isrForCrosstalkSources_855439_56.stderr parsl.dataflow.dflow DEBUG: Task 4035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4035 has memoization hash d78580b6a9d8c5cd79e004d78e683f63 parsl.dataflow.memoization INFO: Task 4035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4035 try 0 launched on executor multi with executor id 4036 parsl.dataflow.dflow INFO: Standard output for task 4035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/41f8ffb7-53fe-4e44-b678-cc172325746f_isrForCrosstalkSources_855404_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/41f8ffb7-53fe-4e44-b678-cc172325746f_isrForCrosstalkSources_855404_57.stderr parsl.dataflow.dflow DEBUG: Task 4036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4036 has memoization hash 8788a9051b9ee849290feca804fecd57 parsl.dataflow.memoization INFO: Task 4036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4036 try 0 launched on executor multi with executor id 4037 parsl.dataflow.dflow INFO: Standard output for task 4036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8be988df-2a29-4a24-9ee6-8df0e2131565_isrForCrosstalkSources_855410_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8be988df-2a29-4a24-9ee6-8df0e2131565_isrForCrosstalkSources_855410_61.stderr parsl.dataflow.dflow DEBUG: Task 4037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4037 has memoization hash 8a51ce63942042b45574c0c10b04f1da parsl.dataflow.memoization INFO: Task 4037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4037 try 0 launched on executor multi with executor id 4038 parsl.dataflow.dflow INFO: Standard output for task 4037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f3ef4a97-dcc7-4079-bc62-92e8090ce2ad_isrForCrosstalkSources_855395_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f3ef4a97-dcc7-4079-bc62-92e8090ce2ad_isrForCrosstalkSources_855395_42.stderr parsl.dataflow.dflow DEBUG: Task 4038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4038 has memoization hash 95022c5980f4e54096166e06d982674f parsl.dataflow.memoization INFO: Task 4038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4038 try 0 launched on executor multi with executor id 4039 parsl.dataflow.dflow INFO: Standard output for task 4038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0bb39b89-0cf6-462a-a707-df9e40953561_isrForCrosstalkSources_855411_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0bb39b89-0cf6-462a-a707-df9e40953561_isrForCrosstalkSources_855411_38.stderr parsl.dataflow.dflow DEBUG: Task 4039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4039 has memoization hash 3a7006b4d8d6ad6c5572b92a8fcec9de parsl.dataflow.memoization INFO: Task 4039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb7637020bbadd6c179bd3a0f2ea3b1 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/169f296c-b55c-47ff-b5b1-3a08b9774840_isrForCrosstalkSources_855406_55.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/169f296c-b55c-47ff-b5b1-3a08b9774840_isrForCrosstalkSources_855406_55.stderr parsl.dataflow.dflow INFO: Parsl task 4039 try 0 launched on executor multi with executor id 4040 parsl.dataflow.dflow INFO: Standard output for task 4039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/12f053dc-30e9-4352-8447-d8f0db5fcc22_isrForCrosstalkSources_855437_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/12f053dc-30e9-4352-8447-d8f0db5fcc22_isrForCrosstalkSources_855437_16.stderr parsl.dataflow.dflow DEBUG: Task 4040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4040 has memoization hash f3f4a029d8ff1dec6f9fe4322a76c1bc parsl.dataflow.memoization INFO: Task 4040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4040 try 0 launched on executor multi with executor id 4041 parsl.dataflow.dflow INFO: Standard output for task 4040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/be6a3853-a0f5-4f4c-81e6-a613bb62ec3f_isrForCrosstalkSources_855436_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/be6a3853-a0f5-4f4c-81e6-a613bb62ec3f_isrForCrosstalkSources_855436_46.stderr parsl.dataflow.dflow DEBUG: Task 4041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4041 has memoization hash 70775d894b1b87528f3c0edbb6468ca6 parsl.dataflow.memoization INFO: Task 4041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4041 try 0 launched on executor multi with executor id 4042 parsl.dataflow.dflow INFO: Standard output for task 4041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a3d9fa00-69d8-4094-ad44-c53d7097e208_isrForCrosstalkSources_855420_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a3d9fa00-69d8-4094-ad44-c53d7097e208_isrForCrosstalkSources_855420_43.stderr parsl.dataflow.dflow DEBUG: Task 4042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4042 has memoization hash b665d4a00175af2488a6699e54d41073 parsl.dataflow.memoization INFO: Task 4042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4042 try 0 launched on executor multi with executor id 4043 parsl.dataflow.dflow INFO: Standard output for task 4042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c00b1945-3fc5-485c-90b0-4efd9ac523c8_isrForCrosstalkSources_855429_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c00b1945-3fc5-485c-90b0-4efd9ac523c8_isrForCrosstalkSources_855429_37.stderr parsl.dataflow.dflow DEBUG: Task 4043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4043 has memoization hash 7ad00a669067cba7e4bd35df0dadcd72 parsl.dataflow.memoization INFO: Task 4043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4043 try 0 launched on executor multi with executor id 4044 parsl.dataflow.dflow INFO: Standard output for task 4043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d53ef0ef-ae96-4d8d-ae9a-386c9ef17a3d_isrForCrosstalkSources_855431_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d53ef0ef-ae96-4d8d-ae9a-386c9ef17a3d_isrForCrosstalkSources_855431_26.stderr parsl.dataflow.dflow DEBUG: Task 4044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4044 has memoization hash ffbc2be992361213ce67d13dea53d13d parsl.dataflow.memoization INFO: Task 4044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4044 try 0 launched on executor multi with executor id 4045 parsl.dataflow.dflow INFO: Standard output for task 4044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eed02406-e04d-4051-9e5c-79d785b5e41d_isrForCrosstalkSources_855423_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eed02406-e04d-4051-9e5c-79d785b5e41d_isrForCrosstalkSources_855423_60.stderr parsl.dataflow.dflow DEBUG: Task 4045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4045 has memoization hash f9e32b6a32d2de9a9d79c956ecdaea39 parsl.dataflow.memoization INFO: Task 4045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4045 try 0 launched on executor multi with executor id 4046 parsl.dataflow.dflow INFO: Standard output for task 4045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/5b5e0f72-a910-422a-9adb-dfcbd51a5582_isrForCrosstalkSources_855390_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/5b5e0f72-a910-422a-9adb-dfcbd51a5582_isrForCrosstalkSources_855390_58.stderr parsl.dataflow.dflow DEBUG: Task 4046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4046 has memoization hash f4e56dd6ce1f38b85361497fb42baad4 parsl.dataflow.memoization INFO: Task 4046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4046 try 0 launched on executor multi with executor id 4047 parsl.dataflow.dflow INFO: Standard output for task 4046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1015263-12ca-4d96-84e9-67cfd9e67a14_isrForCrosstalkSources_855395_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1015263-12ca-4d96-84e9-67cfd9e67a14_isrForCrosstalkSources_855395_29.stderr parsl.dataflow.dflow DEBUG: Task 4047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4047 has memoization hash c95252adcfb6391d7a8995dd258923cd parsl.dataflow.memoization INFO: Task 4047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4047 try 0 launched on executor multi with executor id 4048 parsl.dataflow.dflow INFO: Standard output for task 4047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1739cb96-23c3-4007-9694-e327a68966b3_isrForCrosstalkSources_855385_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1739cb96-23c3-4007-9694-e327a68966b3_isrForCrosstalkSources_855385_5.stderr parsl.dataflow.dflow DEBUG: Task 4048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4048 has memoization hash 7f97da3eef8d91a5d9277ce1280bde32 parsl.dataflow.memoization INFO: Task 4048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4048 try 0 launched on executor multi with executor id 4049 parsl.dataflow.dflow INFO: Standard output for task 4048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b49768f6-ed97-4c85-9dc3-a186565602a2_isrForCrosstalkSources_855459_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b49768f6-ed97-4c85-9dc3-a186565602a2_isrForCrosstalkSources_855459_58.stderr parsl.dataflow.dflow DEBUG: Task 4049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4049 has memoization hash 664c339ef65513716a14d0d412482508 parsl.dataflow.memoization INFO: Task 4049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe7fdb0d7be2c5246c5e3b7c1886287 with result from task 556 parsl.dataflow.dflow INFO: Parsl task 4049 try 0 launched on executor multi with executor id 4050 parsl.dataflow.dflow INFO: Standard output for task 4049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6943df4f-a1f7-4f84-b92a-19aadb1e6abd_isrForCrosstalkSources_855379_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6943df4f-a1f7-4f84-b92a-19aadb1e6abd_isrForCrosstalkSources_855379_43.stderr parsl.dataflow.dflow DEBUG: Task 4050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4050 has memoization hash cff31f1e834fe8b037086a233537ad4a parsl.dataflow.memoization INFO: Task 4050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a5bbd32-66f0-411b-a141-a947a63d04dc_isrForCrosstalkSources_855449_19.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a5bbd32-66f0-411b-a141-a947a63d04dc_isrForCrosstalkSources_855449_19.stderr parsl.dataflow.dflow INFO: Parsl task 4050 try 0 launched on executor multi with executor id 4051 parsl.dataflow.dflow INFO: Standard output for task 4050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/def584f9-868d-4895-b674-cfcd0c9d2222_isrForCrosstalkSources_855399_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/def584f9-868d-4895-b674-cfcd0c9d2222_isrForCrosstalkSources_855399_35.stderr parsl.dataflow.dflow DEBUG: Task 4051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4051 has memoization hash b29c26230ff6cf40e616b41758aa0e69 parsl.dataflow.memoization INFO: Task 4051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4051 try 0 launched on executor multi with executor id 4052 parsl.dataflow.dflow INFO: Standard output for task 4051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/da78e0f6-e453-42ba-bee4-fe31f3c150fc_isrForCrosstalkSources_855389_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/da78e0f6-e453-42ba-bee4-fe31f3c150fc_isrForCrosstalkSources_855389_3.stderr parsl.dataflow.dflow DEBUG: Task 4052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4052 has memoization hash 3e4f1bf7d8252e937e9902a2a6de4982 parsl.dataflow.memoization INFO: Task 4052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4052 try 0 launched on executor multi with executor id 4053 parsl.dataflow.dflow INFO: Standard output for task 4052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3eccadee-5586-41f4-817b-a5011d33d756_isrForCrosstalkSources_855447_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3eccadee-5586-41f4-817b-a5011d33d756_isrForCrosstalkSources_855447_26.stderr parsl.dataflow.dflow DEBUG: Task 4053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4053 has memoization hash 92d12a6883d335d0aa2b6a00a4e254bd parsl.dataflow.memoization INFO: Task 4053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4053 try 0 launched on executor multi with executor id 4054 parsl.dataflow.dflow INFO: Standard output for task 4053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f19bdcfc-b364-4b5f-afff-2c2369f98bd9_isrForCrosstalkSources_855445_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f19bdcfc-b364-4b5f-afff-2c2369f98bd9_isrForCrosstalkSources_855445_54.stderr parsl.dataflow.dflow DEBUG: Task 4054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4054 has memoization hash 73edd4c21aa100726fab1aaf053e2038 parsl.dataflow.memoization INFO: Task 4054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4054 try 0 launched on executor multi with executor id 4055 parsl.dataflow.dflow INFO: Standard output for task 4054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/5043290e-7317-4b88-bbe6-0788ac930ab9_isrForCrosstalkSources_855386_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/5043290e-7317-4b88-bbe6-0788ac930ab9_isrForCrosstalkSources_855386_12.stderr parsl.dataflow.dflow DEBUG: Task 4055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4055 has memoization hash 9b028e5e69f4b1473c81c0a6e937c989 parsl.dataflow.memoization INFO: Task 4055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4055 try 0 launched on executor multi with executor id 4056 parsl.dataflow.dflow INFO: Standard output for task 4055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b80c4d22-fe84-41a2-8830-1571a5fe59d4_isrForCrosstalkSources_855414_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b80c4d22-fe84-41a2-8830-1571a5fe59d4_isrForCrosstalkSources_855414_36.stderr parsl.dataflow.dflow DEBUG: Task 4056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4056 has memoization hash dafe2f87afc7b43f444a80f2c51d1e13 parsl.dataflow.memoization INFO: Task 4056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4056 try 0 launched on executor multi with executor id 4057 parsl.dataflow.dflow INFO: Standard output for task 4056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c63ee3cc-25f2-44cc-8120-35b0374fc093_isrForCrosstalkSources_855384_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c63ee3cc-25f2-44cc-8120-35b0374fc093_isrForCrosstalkSources_855384_13.stderr parsl.dataflow.dflow DEBUG: Task 4057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4057 has memoization hash fc0aecc5801c51d9b0160d434c67afa2 parsl.dataflow.memoization INFO: Task 4057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4057 try 0 launched on executor multi with executor id 4058 parsl.dataflow.dflow INFO: Standard output for task 4057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f43dfe17-38d6-4e55-a0a3-9fececf0f44f_isrForCrosstalkSources_855407_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f43dfe17-38d6-4e55-a0a3-9fececf0f44f_isrForCrosstalkSources_855407_4.stderr parsl.dataflow.dflow DEBUG: Task 4058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4058 has memoization hash c98bb9cdb8ec6ce6068bfcc5d80f201d parsl.dataflow.memoization INFO: Task 4058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4058 try 0 launched on executor multi with executor id 4059 parsl.dataflow.dflow INFO: Standard output for task 4058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/624eb7eb-e2e4-4837-8cef-0e2bb4332568_isrForCrosstalkSources_855408_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/624eb7eb-e2e4-4837-8cef-0e2bb4332568_isrForCrosstalkSources_855408_26.stderr parsl.dataflow.dflow DEBUG: Task 4059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4059 has memoization hash 0f586cb3087dcfa1fe3d629dbc3feff4 parsl.dataflow.memoization INFO: Task 4059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4059 try 0 launched on executor multi with executor id 4060 parsl.dataflow.dflow INFO: Standard output for task 4059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/251396bd-9553-4ed8-88fb-995f5c59f071_isrForCrosstalkSources_855442_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/251396bd-9553-4ed8-88fb-995f5c59f071_isrForCrosstalkSources_855442_50.stderr parsl.dataflow.dflow DEBUG: Task 4060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4060 has memoization hash d267fb98bd4ddd94ee66ba3641c953c9 parsl.dataflow.memoization INFO: Task 4060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4060 try 0 launched on executor multi with executor id 4061 parsl.dataflow.dflow INFO: Standard output for task 4060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f667d1f9-7fda-4957-8996-71c5a39f9c48_isrForCrosstalkSources_855380_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f667d1f9-7fda-4957-8996-71c5a39f9c48_isrForCrosstalkSources_855380_51.stderr parsl.dataflow.dflow DEBUG: Task 4061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4061 has memoization hash dbc8843cd5e445bb63b8c96bb5625d08 parsl.dataflow.memoization INFO: Task 4061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:44 proc_lsst.multi:170 [INFO] [multi] job_id 16964180 proc_lsst.multi INFO: [multi] job_id 16964180 2024-03-09 14:51:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-09 14:51:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964180 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964180 parsl.executors.status_handling DEBUG: Launched block 154 on executor multi with job ID 16964180 parsl.executors.status_handling INFO: Allocated block ID 155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-155 2024-03-09 14:51:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4061 try 0 launched on executor multi with executor id 4062 parsl.dataflow.dflow INFO: Standard output for task 4061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f86a8568-2ac7-4735-8feb-a67c01abad58_isrForCrosstalkSources_855426_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f86a8568-2ac7-4735-8feb-a67c01abad58_isrForCrosstalkSources_855426_50.stderr parsl.dataflow.dflow DEBUG: Task 4062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4062 has memoization hash c3c08f942c5d162502d7c95d3a2de4ff parsl.dataflow.memoization INFO: Task 4062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a1ffe0ccbc68ceadbe723297ea8e1d with result from task 169 parsl.dataflow.dflow INFO: Parsl task 4062 try 0 launched on executor multi with executor id 4063 parsl.dataflow.dflow INFO: Standard output for task 4062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c27b1780-ec4b-4e3a-8a55-7915af2af541_isrForCrosstalkSources_855429_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c27b1780-ec4b-4e3a-8a55-7915af2af541_isrForCrosstalkSources_855429_48.stderr parsl.dataflow.dflow DEBUG: Task 4063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4063 has memoization hash a629dd8ff0aec0376bb4e311d894647c parsl.dataflow.memoization INFO: Task 4063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/89452317-4c20-425c-9a34-ccdac4c6f150_isrForCrosstalkSources_855434_44.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/89452317-4c20-425c-9a34-ccdac4c6f150_isrForCrosstalkSources_855434_44.stderr parsl.dataflow.dflow INFO: Parsl task 4063 try 0 launched on executor multi with executor id 4064 parsl.dataflow.dflow INFO: Standard output for task 4063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c873f042-4a45-4997-8d7b-c0831f5b9a5c_isrForCrosstalkSources_855442_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c873f042-4a45-4997-8d7b-c0831f5b9a5c_isrForCrosstalkSources_855442_23.stderr parsl.dataflow.dflow DEBUG: Task 4064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4064 has memoization hash 4aaa715e513c12ff66fd4bf3d825cc1d parsl.dataflow.memoization INFO: Task 4064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4064 try 0 launched on executor multi with executor id 4065 parsl.dataflow.dflow INFO: Standard output for task 4064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e8ecc3a2-1384-43c9-a6cb-2172c30648ca_isrForCrosstalkSources_855421_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e8ecc3a2-1384-43c9-a6cb-2172c30648ca_isrForCrosstalkSources_855421_10.stderr parsl.dataflow.dflow DEBUG: Task 4065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4065 has memoization hash 4f5da4b8f99f87823bfa92a9f0d33016 parsl.dataflow.memoization INFO: Task 4065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4065 try 0 launched on executor multi with executor id 4066 parsl.dataflow.dflow INFO: Standard output for task 4065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a776f574-d089-42ca-a3cb-e0cb955450d3_isrForCrosstalkSources_855434_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a776f574-d089-42ca-a3cb-e0cb955450d3_isrForCrosstalkSources_855434_62.stderr parsl.dataflow.dflow DEBUG: Task 4066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4066 has memoization hash ad4a8b4125d7eec3c3207ddb06a81187 parsl.dataflow.memoization INFO: Task 4066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4066 try 0 launched on executor multi with executor id 4067 parsl.dataflow.dflow INFO: Standard output for task 4066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f7d574d0-43ab-4678-8a0b-66e0c21bb352_isrForCrosstalkSources_855391_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f7d574d0-43ab-4678-8a0b-66e0c21bb352_isrForCrosstalkSources_855391_44.stderr parsl.dataflow.dflow DEBUG: Task 4067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4067 has memoization hash cd285e67568b958f8043fd637ca0934d parsl.dataflow.memoization INFO: Task 4067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4067 try 0 launched on executor multi with executor id 4068 parsl.dataflow.dflow INFO: Standard output for task 4067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/88ef6291-d910-430f-a865-f5c712ffceb7_isrForCrosstalkSources_855398_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/88ef6291-d910-430f-a865-f5c712ffceb7_isrForCrosstalkSources_855398_55.stderr parsl.dataflow.dflow DEBUG: Task 4068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4068 has memoization hash 0ce7ec6cbc328f93a583eabafbb21ed0 parsl.dataflow.memoization INFO: Task 4068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4068 try 0 launched on executor multi with executor id 4069 parsl.dataflow.dflow INFO: Standard output for task 4068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67ecdf70-c74a-4acc-80ad-8711c3f490da_isrForCrosstalkSources_855382_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67ecdf70-c74a-4acc-80ad-8711c3f490da_isrForCrosstalkSources_855382_19.stderr parsl.dataflow.dflow DEBUG: Task 4069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4069 has memoization hash 2013a38afe315b0e175e6e9de3c47565 parsl.dataflow.memoization INFO: Task 4069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4069 try 0 launched on executor multi with executor id 4070 parsl.dataflow.dflow INFO: Standard output for task 4069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/90ae8827-d561-46c4-9f90-40e176883118_isrForCrosstalkSources_855440_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/90ae8827-d561-46c4-9f90-40e176883118_isrForCrosstalkSources_855440_20.stderr parsl.dataflow.dflow DEBUG: Task 4070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4070 has memoization hash 4d7038a7ebfc2c3ce876c1ce7e2506b9 parsl.dataflow.memoization INFO: Task 4070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4070 try 0 launched on executor multi with executor id 4071 parsl.dataflow.dflow INFO: Standard output for task 4070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b8982267-92a5-4651-a460-ad9278fab4b9_isrForCrosstalkSources_855384_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b8982267-92a5-4651-a460-ad9278fab4b9_isrForCrosstalkSources_855384_61.stderr parsl.dataflow.dflow DEBUG: Task 4071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4071 has memoization hash 9a3ab87220fa77b4c86be2f6d64077f9 parsl.dataflow.memoization INFO: Task 4071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4071 try 0 launched on executor multi with executor id 4072 parsl.dataflow.dflow INFO: Standard output for task 4071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e7b4080b-5b7d-4539-906d-4c762b653d70_isrForCrosstalkSources_855445_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e7b4080b-5b7d-4539-906d-4c762b653d70_isrForCrosstalkSources_855445_55.stderr parsl.dataflow.dflow DEBUG: Task 4072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4072 has memoization hash 67cacd64a69b15e1e780d19a654800fb parsl.dataflow.memoization INFO: Task 4072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a02d492fedeef5f0fc915a27e8bcc72e with result from task 555 parsl.dataflow.dflow INFO: Parsl task 4072 try 0 launched on executor multi with executor id 4073 parsl.dataflow.dflow INFO: Standard output for task 4072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/28efe684-2215-4b5c-bbb1-156541818ca3_isrForCrosstalkSources_855435_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/28efe684-2215-4b5c-bbb1-156541818ca3_isrForCrosstalkSources_855435_17.stderr parsl.dataflow.dflow DEBUG: Task 4073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4073 has memoization hash 78cd1af0d8fc635292112f8b078b3b0c parsl.dataflow.memoization INFO: Task 4073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4073 try 0 launched on executor multi with executor id 4074 parsl.dataflow.dflow INFO: Standard output for task 4073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/547dec9d-29ef-41c9-ab31-7b21e92c5933_isrForCrosstalkSources_855384_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/547dec9d-29ef-41c9-ab31-7b21e92c5933_isrForCrosstalkSources_855384_29.stderr parsl.dataflow.dflow DEBUG: Task 4074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4074 has memoization hash 01be05dd176847e719a4a900bc52edf9 parsl.dataflow.memoization INFO: Task 4074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1bf0182f-cb35-4b86-82ff-2d67aa0ab18e_isrForCrosstalkSources_855381_18.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1bf0182f-cb35-4b86-82ff-2d67aa0ab18e_isrForCrosstalkSources_855381_18.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de3a5d9a0721d2436ebf8ecc4d8819b5 with result from task 348 parsl.dataflow.dflow INFO: Parsl task 4074 try 0 launched on executor multi with executor id 4075 parsl.dataflow.dflow INFO: Standard output for task 4074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f68bef45-6c86-40c6-a4a9-bc07e16a6126_isrForCrosstalkSources_855407_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f68bef45-6c86-40c6-a4a9-bc07e16a6126_isrForCrosstalkSources_855407_33.stderr parsl.dataflow.dflow DEBUG: Task 4075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4075 has memoization hash 7a511acf4b34f774e70c8c085463ee22 parsl.dataflow.memoization INFO: Task 4075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4075 try 0 launched on executor multi with executor id 4076 parsl.dataflow.dflow INFO: Standard output for task 4075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5bed13c9-6bd7-4759-bdb0-38cd36362cb7_isrForCrosstalkSources_855455_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5bed13c9-6bd7-4759-bdb0-38cd36362cb7_isrForCrosstalkSources_855455_22.stderr parsl.dataflow.dflow DEBUG: Task 4076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4076 has memoization hash af955014a0b5a37604c60a017b063018 parsl.dataflow.memoization INFO: Task 4076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/06fb6fd3-77c8-4245-b22d-7082dfaaf675_isrForCrosstalkSources_855411_51.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/06fb6fd3-77c8-4245-b22d-7082dfaaf675_isrForCrosstalkSources_855411_51.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5e4a73ec795330d267ee6ae458f0ba0 with result from task 346 parsl.dataflow.dflow INFO: Parsl task 4076 try 0 launched on executor multi with executor id 4077 parsl.dataflow.dflow INFO: Standard output for task 4076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a2301f6e-4bb2-4ea0-a4bb-a6c542937272_isrForCrosstalkSources_855442_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a2301f6e-4bb2-4ea0-a4bb-a6c542937272_isrForCrosstalkSources_855442_28.stderr parsl.dataflow.dflow DEBUG: Task 4077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4077 has memoization hash 1f2aa5364d31e673e2842e53a51c5265 parsl.dataflow.memoization INFO: Task 4077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2c325436-9b4d-493b-9a95-b9048512beba_isrForCrosstalkSources_855393_56.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2c325436-9b4d-493b-9a95-b9048512beba_isrForCrosstalkSources_855393_56.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f50138552c28c9f921ccfc4e70d30483 with result from task 351 parsl.dataflow.dflow INFO: Parsl task 4077 try 0 launched on executor multi with executor id 4078 parsl.dataflow.dflow INFO: Standard output for task 4077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e33099c8-ad64-4886-84b8-8bc86df0233d_isrForCrosstalkSources_855438_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e33099c8-ad64-4886-84b8-8bc86df0233d_isrForCrosstalkSources_855438_41.stderr parsl.dataflow.dflow DEBUG: Task 4078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4078 has memoization hash bfeae2c6982fcde2adc91e6f4fb3af7b parsl.dataflow.memoization INFO: Task 4078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4078 try 0 launched on executor multi with executor id 4079 parsl.dataflow.dflow INFO: Standard output for task 4078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b319fd85-a851-463f-99d5-98b1a6d9b596_isrForCrosstalkSources_855407_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b319fd85-a851-463f-99d5-98b1a6d9b596_isrForCrosstalkSources_855407_13.stderr parsl.dataflow.dflow DEBUG: Task 4079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4079 has memoization hash 17d1cf7d27d9637bbf7c19cb6fc35f9a parsl.dataflow.memoization INFO: Task 4079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1ce20eeb-3b09-4a80-b85a-104360ec34a8_isrForCrosstalkSources_855423_25.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1ce20eeb-3b09-4a80-b85a-104360ec34a8_isrForCrosstalkSources_855423_25.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1e5fd629baf861d33923f6f99d9552 with result from task 349 parsl.dataflow.dflow INFO: Parsl task 4079 try 0 launched on executor multi with executor id 4080 parsl.dataflow.dflow INFO: Standard output for task 4079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/eba5b252-fab5-40ad-8de2-5b6d8919e224_isrForCrosstalkSources_855418_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/eba5b252-fab5-40ad-8de2-5b6d8919e224_isrForCrosstalkSources_855418_9.stderr parsl.dataflow.dflow DEBUG: Task 4080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4080 has memoization hash 710c9611a19528ed11dc7dd1b71968f7 parsl.dataflow.memoization INFO: Task 4080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4080 try 0 launched on executor multi with executor id 4081 parsl.dataflow.dflow INFO: Standard output for task 4080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b6183798-e658-4507-b86b-1b692c91bf69_isrForCrosstalkSources_855411_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b6183798-e658-4507-b86b-1b692c91bf69_isrForCrosstalkSources_855411_25.stderr parsl.dataflow.dflow DEBUG: Task 4081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4081 has memoization hash c31474b149d8a445dd3de5b8768c6c5e parsl.dataflow.memoization INFO: Task 4081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4081 try 0 launched on executor multi with executor id 4082 parsl.dataflow.dflow INFO: Standard output for task 4081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/20e03570-c069-4da3-9937-0d247399fac1_isrForCrosstalkSources_855415_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/20e03570-c069-4da3-9937-0d247399fac1_isrForCrosstalkSources_855415_20.stderr parsl.dataflow.dflow DEBUG: Task 4082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4082 has memoization hash ecd853b88ddbb48a3f011b59c3187c74 parsl.dataflow.memoization INFO: Task 4082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4082 try 0 launched on executor multi with executor id 4083 parsl.dataflow.dflow INFO: Standard output for task 4082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5993ac30-8b18-415c-8e1e-1633454edcec_isrForCrosstalkSources_855455_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5993ac30-8b18-415c-8e1e-1633454edcec_isrForCrosstalkSources_855455_3.stderr parsl.dataflow.dflow DEBUG: Task 4083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4083 has memoization hash 6dc317aaa251a2827f37a2bb1695950f parsl.dataflow.memoization INFO: Task 4083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/299ecb10-15b2-4e70-964a-4bed5dc61525_isrForCrosstalkSources_855426_43.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/299ecb10-15b2-4e70-964a-4bed5dc61525_isrForCrosstalkSources_855426_43.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50bb850727873d92d1a7f58dc435651c with result from task 347 parsl.dataflow.dflow INFO: Parsl task 4083 try 0 launched on executor multi with executor id 4084 parsl.dataflow.dflow INFO: Standard output for task 4083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b50ee4b3-780a-4eac-94cb-8c482a1a7256_isrForCrosstalkSources_855410_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b50ee4b3-780a-4eac-94cb-8c482a1a7256_isrForCrosstalkSources_855410_14.stderr parsl.dataflow.dflow DEBUG: Task 4084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4084 has memoization hash 3fef3d919ac94a969fc6a17739391cf0 parsl.dataflow.memoization INFO: Task 4084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4084 try 0 launched on executor multi with executor id 4085 parsl.dataflow.dflow INFO: Standard output for task 4084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/85d95276-8375-4f2d-bd26-46e85a59ee65_isrForCrosstalkSources_855387_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/85d95276-8375-4f2d-bd26-46e85a59ee65_isrForCrosstalkSources_855387_22.stderr parsl.dataflow.dflow DEBUG: Task 4085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4085 has memoization hash 242edfd70a1d4ea0c2d3fc421a44018b parsl.dataflow.memoization INFO: Task 4085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4085 try 0 launched on executor multi with executor id 4086 parsl.dataflow.dflow INFO: Standard output for task 4085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/11b7eb84-148a-48ff-aa47-6659a7e31ff9_isrForCrosstalkSources_855398_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/11b7eb84-148a-48ff-aa47-6659a7e31ff9_isrForCrosstalkSources_855398_11.stderr parsl.dataflow.dflow DEBUG: Task 4086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4086 has memoization hash 5e84fecfc724338c9864c6fc83a1c900 parsl.dataflow.memoization INFO: Task 4086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4086 try 0 launched on executor multi with executor id 4087 parsl.dataflow.dflow INFO: Standard output for task 4086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e4b48e2d-22d6-4800-9c4e-27ae72bfa7c1_isrForCrosstalkSources_855392_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e4b48e2d-22d6-4800-9c4e-27ae72bfa7c1_isrForCrosstalkSources_855392_32.stderr parsl.dataflow.dflow DEBUG: Task 4087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4087 has memoization hash 5946d50809f48c1dc019d12bd1aa718e parsl.dataflow.memoization INFO: Task 4087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4087 try 0 launched on executor multi with executor id 4088 parsl.dataflow.dflow INFO: Standard output for task 4087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/45c2e4ca-0de0-4fd3-9149-f5b784f66962_isrForCrosstalkSources_855426_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/45c2e4ca-0de0-4fd3-9149-f5b784f66962_isrForCrosstalkSources_855426_8.stderr parsl.dataflow.dflow DEBUG: Task 4088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4088 has memoization hash fd8152a393c1ed89f7bf3ec3fc324994 parsl.dataflow.memoization INFO: Task 4088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4088 try 0 launched on executor multi with executor id 4089 parsl.dataflow.dflow INFO: Standard output for task 4088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3fe68613-b225-4547-be19-8d928a75bdc3_isrForCrosstalkSources_855443_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3fe68613-b225-4547-be19-8d928a75bdc3_isrForCrosstalkSources_855443_23.stderr parsl.dataflow.dflow DEBUG: Task 4089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4089 has memoization hash b51f7977a775099fbd13d43daec2ed14 parsl.dataflow.memoization INFO: Task 4089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4089 try 0 launched on executor multi with executor id 4090 parsl.dataflow.dflow INFO: Standard output for task 4089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/706b59fd-c491-4ebd-b6e3-900fdf7ff20a_isrForCrosstalkSources_855455_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/706b59fd-c491-4ebd-b6e3-900fdf7ff20a_isrForCrosstalkSources_855455_40.stderr parsl.dataflow.dflow DEBUG: Task 4090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4090 has memoization hash 3ed6eeca1c07f14cbebda5ef84095760 parsl.dataflow.memoization INFO: Task 4090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4090 try 0 launched on executor multi with executor id 4091 parsl.dataflow.dflow INFO: Standard output for task 4090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/dc7e84e0-1546-45f6-858b-eef5a4c83835_isrForCrosstalkSources_855428_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/dc7e84e0-1546-45f6-858b-eef5a4c83835_isrForCrosstalkSources_855428_10.stderr parsl.dataflow.dflow DEBUG: Task 4091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4091 has memoization hash 976e9432389ee9f5f05cab2acf2164d0 parsl.dataflow.memoization INFO: Task 4091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4091 try 0 launched on executor multi with executor id 4092 parsl.dataflow.dflow INFO: Standard output for task 4091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5d1becd9-4ea7-4561-a0cd-4b4b14423d0f_isrForCrosstalkSources_855382_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5d1becd9-4ea7-4561-a0cd-4b4b14423d0f_isrForCrosstalkSources_855382_47.stderr parsl.dataflow.dflow DEBUG: Task 4092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4092 has memoization hash 2fc2cfd2a1d06bdb7faca0a19cd5775d parsl.dataflow.memoization INFO: Task 4092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1418cdd2-8cad-44ce-a65c-0e430102e229_isrForCrosstalkSources_855454_45.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1418cdd2-8cad-44ce-a65c-0e430102e229_isrForCrosstalkSources_855454_45.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ffe1a7d9a0e13a3ebf6cfcfb7e96f3d with result from task 345 2024-03-09 14:51:44 proc_lsst.multi:170 [INFO] [multi] job_id 16964181 proc_lsst.multi INFO: [multi] job_id 16964181 2024-03-09 14:51:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-09 14:51:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964181 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964181 parsl.executors.status_handling DEBUG: Launched block 155 on executor multi with job ID 16964181 parsl.executors.status_handling INFO: Allocated block ID 156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-156 2024-03-09 14:51:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-09 14:51:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4092 try 0 launched on executor multi with executor id 4093 parsl.dataflow.dflow INFO: Standard output for task 4092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/9c2274c9-4c12-4fd6-b32d-c1c8f194e966_isrForCrosstalkSources_855388_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/9c2274c9-4c12-4fd6-b32d-c1c8f194e966_isrForCrosstalkSources_855388_45.stderr parsl.dataflow.dflow DEBUG: Task 4093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4093 has memoization hash 628cef5f9ebd1dceef0262367bfe32ad parsl.dataflow.memoization INFO: Task 4093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e3b90e00-db19-47c1-b4d5-252bd2b14a56_isrForCrosstalkSources_855419_15.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e3b90e00-db19-47c1-b4d5-252bd2b14a56_isrForCrosstalkSources_855419_15.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9669f7e48e1a76c23199b852528bf6e1 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/25a8a36a-6870-405a-8501-8fb9c89085d8_isrForCrosstalkSources_855444_3.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/25a8a36a-6870-405a-8501-8fb9c89085d8_isrForCrosstalkSources_855444_3.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e94c3a92c9157f10da8c3af2f451832 with result from task 352 parsl.dataflow.dflow INFO: Parsl task 4093 try 0 launched on executor multi with executor id 4094 parsl.dataflow.dflow INFO: Standard output for task 4093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5a9ea3ce-a178-4212-b564-4c0a51a3d79c_isrForCrosstalkSources_855405_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5a9ea3ce-a178-4212-b564-4c0a51a3d79c_isrForCrosstalkSources_855405_28.stderr parsl.dataflow.dflow DEBUG: Task 4094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4094 has memoization hash 74a4cdb53dc77f2cdfe5c08a3ac0a038 parsl.dataflow.memoization INFO: Task 4094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/49e0f6e4-2402-4939-aceb-4d26ef8bbeda_isrForCrosstalkSources_855444_32.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/49e0f6e4-2402-4939-aceb-4d26ef8bbeda_isrForCrosstalkSources_855444_32.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5da861e86f602ab3f7a03be834688ca with result from task 172 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6f2640e1-9c2d-4fa0-8d96-2afb8d0c8743_isrForCrosstalkSources_855445_31.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6f2640e1-9c2d-4fa0-8d96-2afb8d0c8743_isrForCrosstalkSources_855445_31.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 544c97ed733d5a098ff4e85370daf40f with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53553d90-274e-4115-9d38-6d47566a7f6a_isrForCrosstalkSources_855389_47.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53553d90-274e-4115-9d38-6d47566a7f6a_isrForCrosstalkSources_855389_47.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4525e0122ff6ee7fabce7a16553cdc1d with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4a556679-8a81-41ec-971e-2e7080e5f88b_isrForCrosstalkSources_855412_14.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4a556679-8a81-41ec-971e-2e7080e5f88b_isrForCrosstalkSources_855412_14.stderr parsl.dataflow.dflow INFO: Parsl task 4094 try 0 launched on executor multi with executor id 4095 parsl.dataflow.dflow INFO: Standard output for task 4094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/419287fe-bc22-4e99-a5a4-5e14edcff36f_isrForCrosstalkSources_855416_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/419287fe-bc22-4e99-a5a4-5e14edcff36f_isrForCrosstalkSources_855416_51.stderr parsl.dataflow.dflow DEBUG: Task 4095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4095 has memoization hash 5daa816b67903216d9787d285df8acf8 parsl.dataflow.memoization INFO: Task 4095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c7ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5ede7e49345bb5afc8682733eea174 with result from task 310 parsl.dataflow.dflow INFO: Parsl task 4095 try 0 launched on executor multi with executor id 4096 parsl.dataflow.dflow INFO: Standard output for task 4095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b843cc33-9fac-4c08-aafb-f292d59a07db_isrForCrosstalkSources_855452_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b843cc33-9fac-4c08-aafb-f292d59a07db_isrForCrosstalkSources_855452_22.stderr parsl.dataflow.dflow DEBUG: Task 4096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4096 has memoization hash 8b1cd599b5333c97071e074c317b0a42 parsl.dataflow.memoization INFO: Task 4096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b1c57994-424a-4836-8f69-12504cc8c604_isrForCrosstalkSources_855379_17.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b1c57994-424a-4836-8f69-12504cc8c604_isrForCrosstalkSources_855379_17.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c76b7a502b2e2f88ada48d86b7245cb2 with result from task 309 parsl.dataflow.dflow INFO: Parsl task 4096 try 0 launched on executor multi with executor id 4097 parsl.dataflow.dflow INFO: Standard output for task 4096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/209b97e8-bb03-440a-8fa3-6f2208b0c7f4_isrForCrosstalkSources_855426_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/209b97e8-bb03-440a-8fa3-6f2208b0c7f4_isrForCrosstalkSources_855426_7.stderr parsl.dataflow.dflow DEBUG: Task 4097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4097 has memoization hash 5eaae01d95cd1971be0c7afab7351f79 parsl.dataflow.memoization INFO: Task 4097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b79689f0-c4e3-4ea2-99c7-752ecc739288_isrForCrosstalkSources_855433_22.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b79689f0-c4e3-4ea2-99c7-752ecc739288_isrForCrosstalkSources_855433_22.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95f75fb9a97c90dfcc00485088facd15 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3223be1e-cdbd-484b-8b3c-98ca3e0e4582_isrForCrosstalkSources_855395_26.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3223be1e-cdbd-484b-8b3c-98ca3e0e4582_isrForCrosstalkSources_855395_26.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fa57f16e685b53e7076993f4cd5eeb5 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2694c71d-bc8d-4ebc-b11b-75dddba2920b_isrForCrosstalkSources_855444_18.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/2694c71d-bc8d-4ebc-b11b-75dddba2920b_isrForCrosstalkSources_855444_18.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18e05027c83b9a9e7301f8d1362b5ac6 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/702e16f8-52b0-43f1-8bbe-60be4478685c_isrForCrosstalkSources_855389_2.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/702e16f8-52b0-43f1-8bbe-60be4478685c_isrForCrosstalkSources_855389_2.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d373f80873937134308a600eae12ad9e with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1c7242b1-7008-4dc9-92a4-a9e4c3d61962_isrForCrosstalkSources_855408_16.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1c7242b1-7008-4dc9-92a4-a9e4c3d61962_isrForCrosstalkSources_855408_16.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8851fdc5710565b15a3fa299d07ef13 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8bb63a92-abaf-420d-ab13-24510e397729_isrForCrosstalkSources_855404_7.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8bb63a92-abaf-420d-ab13-24510e397729_isrForCrosstalkSources_855404_7.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d7be050fd7b3f434dc9af0f262694d with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/cecb02c7-a7ae-460c-a202-c144aeca5753_isrForCrosstalkSources_855381_16.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/cecb02c7-a7ae-460c-a202-c144aeca5753_isrForCrosstalkSources_855381_16.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 424f0c8e7377c2017c7e46a22ff1c4f7 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/43cbe0ca-3b78-45e4-b785-704d4ad1d878_isrForCrosstalkSources_855434_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/43cbe0ca-3b78-45e4-b785-704d4ad1d878_isrForCrosstalkSources_855434_19.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b2bf4ea75c82581374b2052995d04f2 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/76636563-678b-4e7b-88c0-b6f7dc6b99ee_isrForCrosstalkSources_855459_21.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/76636563-678b-4e7b-88c0-b6f7dc6b99ee_isrForCrosstalkSources_855459_21.stderr parsl.dataflow.dflow INFO: Parsl task 4097 try 0 launched on executor multi with executor id 4098 parsl.dataflow.dflow INFO: Standard output for task 4097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/ced8bc50-5c26-4152-bbac-71bfcfa95d8d_isrForCrosstalkSources_855402_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/ced8bc50-5c26-4152-bbac-71bfcfa95d8d_isrForCrosstalkSources_855402_50.stderr parsl.dataflow.dflow DEBUG: Task 4098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4098 has memoization hash 3d36b79e2fe4a92e93f78f3b94a35c45 parsl.dataflow.memoization INFO: Task 4098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4098 try 0 launched on executor multi with executor id 4099 parsl.dataflow.dflow INFO: Standard output for task 4098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f2f75584-d56e-4b37-bade-13d869e08e6e_isrForCrosstalkSources_855424_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f2f75584-d56e-4b37-bade-13d869e08e6e_isrForCrosstalkSources_855424_51.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0471056c62bf0906164ef830ab7557a with result from task 246 parsl.dataflow.dflow DEBUG: Task 4099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4099 has memoization hash db4ca1379e3b3f3cb72549ba442b3906 parsl.dataflow.memoization INFO: Task 4099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4099 try 0 launched on executor multi with executor id 4100 parsl.dataflow.dflow INFO: Standard output for task 4099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/415c60a6-f85b-41e6-8fc3-4bbba030d925_isrForCrosstalkSources_855425_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/415c60a6-f85b-41e6-8fc3-4bbba030d925_isrForCrosstalkSources_855425_39.stderr parsl.dataflow.dflow DEBUG: Task 4100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4100 has memoization hash 43d318bf481a652302e73ef7abc3e871 parsl.dataflow.memoization INFO: Task 4100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/eacba2aa-dfba-4fd3-93ee-10c240450754_isrForCrosstalkSources_855457_1.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/eacba2aa-dfba-4fd3-93ee-10c240450754_isrForCrosstalkSources_855457_1.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6208d7408e8e0a64e81e9f4511e0287c with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b42578a2-70c8-4e10-863e-c9da0a1f958f_isrForCrosstalkSources_855457_24.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b42578a2-70c8-4e10-863e-c9da0a1f958f_isrForCrosstalkSources_855457_24.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec0712062872770a19939e37b2db94a1 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/85eb9b59-7bfa-414a-9bf7-ba3340e57d94_isrForCrosstalkSources_855438_5.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/85eb9b59-7bfa-414a-9bf7-ba3340e57d94_isrForCrosstalkSources_855438_5.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f19e30309087fcc6f214bb85715d5a0 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/0f4c324b-44e5-4f5a-aa2c-235ff3dd8cdd_isrForCrosstalkSources_855399_7.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/0f4c324b-44e5-4f5a-aa2c-235ff3dd8cdd_isrForCrosstalkSources_855399_7.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73020f9c8aafee408067544b6dddb936 with result from task 243 parsl.dataflow.dflow INFO: Parsl task 4100 try 0 launched on executor multi with executor id 4101 parsl.dataflow.dflow INFO: Standard output for task 4100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad694f31-0982-4200-88ab-fda15a45583c_isrForCrosstalkSources_855429_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad694f31-0982-4200-88ab-fda15a45583c_isrForCrosstalkSources_855429_56.stderr parsl.dataflow.dflow DEBUG: Task 4101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4101 has memoization hash cfc2c605464e3e4f273c9d3c7fd3669d parsl.dataflow.memoization INFO: Task 4101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c36d5678-9ebf-4038-a835-fc2f04618420_isrForCrosstalkSources_855438_10.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c36d5678-9ebf-4038-a835-fc2f04618420_isrForCrosstalkSources_855438_10.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 998c39b7cb700194b71a2ea1a99bc9b4 with result from task 249 parsl.dataflow.dflow INFO: Parsl task 4101 try 0 launched on executor multi with executor id 4102 parsl.dataflow.dflow INFO: Standard output for task 4101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed2412aa-8382-46f9-8ded-6fb37558b064_isrForCrosstalkSources_855419_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed2412aa-8382-46f9-8ded-6fb37558b064_isrForCrosstalkSources_855419_27.stderr parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow DEBUG: Task 4102 will be sent to executor multi parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3f50c789-238c-4490-ba83-ef154fbfe1ab_isrForCrosstalkSources_855408_38.stdout parsl.dataflow.dflow INFO: Task 4102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3f50c789-238c-4490-ba83-ef154fbfe1ab_isrForCrosstalkSources_855408_38.stderr parsl.dataflow.dflow DEBUG: Task 4102 set to pending state with AppFuture: parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df2c75458afe5633e6757f16cdcaeb4f with result from task 247 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.memoization DEBUG: Task 4102 has memoization hash 1f7c71dec9846e3bed307b4afef7bce0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization INFO: Task 4102 had no result in cache parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d813064-593d-4087-8187-3ff0371901eb_isrForCrosstalkSources_855380_59.stdout parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d813064-593d-4087-8187-3ff0371901eb_isrForCrosstalkSources_855380_59.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad87428e4c385bf8ef6b62a13705e3a7 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/73827ce2-5851-47bb-b44c-b9042299481b_isrForCrosstalkSources_855395_39.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/73827ce2-5851-47bb-b44c-b9042299481b_isrForCrosstalkSources_855395_39.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fff53cdf04bedf4712ee34a413c9d32 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/ed3594e3-fdfe-4470-a30c-3d7bd236e265_isrForCrosstalkSources_855422_44.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/ed3594e3-fdfe-4470-a30c-3d7bd236e265_isrForCrosstalkSources_855422_44.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e33ff3b776d8a5b8f65d4d5ca520af with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/054c6952-15e1-4eed-a77f-342495918eaf_isrForCrosstalkSources_855415_8.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/054c6952-15e1-4eed-a77f-342495918eaf_isrForCrosstalkSources_855415_8.stderr parsl.dataflow.dflow INFO: Parsl task 4102 try 0 launched on executor multi with executor id 4103 parsl.dataflow.dflow INFO: Standard output for task 4102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e492f7e7-0bbe-4223-961f-039a7d25d717_isrForCrosstalkSources_855394_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e492f7e7-0bbe-4223-961f-039a7d25d717_isrForCrosstalkSources_855394_61.stderr parsl.dataflow.dflow DEBUG: Task 4103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4103 has memoization hash 1f91df7a5533f29b726f6d30430f1c96 parsl.dataflow.memoization INFO: Task 4103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4103 try 0 launched on executor multi with executor id 4104 parsl.dataflow.dflow INFO: Standard output for task 4103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/74e4ccba-d10d-4377-a242-1ca898904c9d_isrForCrosstalkSources_855396_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/74e4ccba-d10d-4377-a242-1ca898904c9d_isrForCrosstalkSources_855396_5.stderr parsl.dataflow.dflow DEBUG: Task 4104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4104 has memoization hash bc7b464ebb05a7a4ddae81e8adb16b10 parsl.dataflow.memoization INFO: Task 4104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4104 try 0 launched on executor multi with executor id 4105 parsl.dataflow.dflow INFO: Standard output for task 4104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cc3b8fb1-0b0e-47e6-84da-5d31a109291e_isrForCrosstalkSources_855447_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cc3b8fb1-0b0e-47e6-84da-5d31a109291e_isrForCrosstalkSources_855447_15.stderr parsl.dataflow.dflow DEBUG: Task 4105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4105 has memoization hash 2a7f819d2969742116334d554a80441c parsl.dataflow.memoization INFO: Task 4105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4105 try 0 launched on executor multi with executor id 4106 parsl.dataflow.dflow INFO: Standard output for task 4105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/ec895573-e668-4b48-9aed-ce4879ca8e23_isrForCrosstalkSources_855425_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/ec895573-e668-4b48-9aed-ce4879ca8e23_isrForCrosstalkSources_855425_46.stderr parsl.dataflow.dflow DEBUG: Task 4106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4106 has memoization hash 5d532a0a511a5450a393f8d93f892413 parsl.dataflow.memoization INFO: Task 4106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4106 try 0 launched on executor multi with executor id 4107 parsl.dataflow.dflow INFO: Standard output for task 4106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2b04521c-b51a-448e-b74e-8ec48fc8c437_isrForCrosstalkSources_855392_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2b04521c-b51a-448e-b74e-8ec48fc8c437_isrForCrosstalkSources_855392_18.stderr parsl.dataflow.dflow DEBUG: Task 4107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4107 has memoization hash 38f218537d39e2ec23c65736c3a303c9 parsl.dataflow.memoization INFO: Task 4107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4107 try 0 launched on executor multi with executor id 4108 parsl.dataflow.dflow INFO: Standard output for task 4107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052c61d6-bbcd-4c00-8470-b5b0f1ff88c0_isrForCrosstalkSources_855451_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052c61d6-bbcd-4c00-8470-b5b0f1ff88c0_isrForCrosstalkSources_855451_45.stderr parsl.dataflow.dflow DEBUG: Task 4108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4108 has memoization hash 24a71d327c7def9e6b2556b485ee506c parsl.dataflow.memoization INFO: Task 4108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4108 try 0 launched on executor multi with executor id 4109 parsl.dataflow.dflow INFO: Standard output for task 4108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/60f6d9f8-b2e7-42c1-9692-a004364752a2_isrForCrosstalkSources_855437_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/60f6d9f8-b2e7-42c1-9692-a004364752a2_isrForCrosstalkSources_855437_14.stderr parsl.dataflow.dflow DEBUG: Task 4109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4109 has memoization hash dadfc1d6b5aff808412fbd117055081e parsl.dataflow.memoization INFO: Task 4109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4109 try 0 launched on executor multi with executor id 4110 parsl.dataflow.dflow INFO: Standard output for task 4109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a72b6744-cbd3-4bd8-8dd1-a5ca31b409e5_isrForCrosstalkSources_855403_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a72b6744-cbd3-4bd8-8dd1-a5ca31b409e5_isrForCrosstalkSources_855403_40.stderr parsl.dataflow.dflow DEBUG: Task 4110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4110 has memoization hash 12c4a7abcbf621b773d2b5afe3cde88c parsl.dataflow.memoization INFO: Task 4110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4110 try 0 launched on executor multi with executor id 4111 parsl.dataflow.dflow INFO: Standard output for task 4110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5c3c9d9a-2806-43ba-9f48-b9a9503d4aed_isrForCrosstalkSources_855444_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5c3c9d9a-2806-43ba-9f48-b9a9503d4aed_isrForCrosstalkSources_855444_36.stderr parsl.dataflow.dflow DEBUG: Task 4111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4111 has memoization hash 62e387979963de31f04e68216b98e7f3 parsl.dataflow.memoization INFO: Task 4111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4111 try 0 launched on executor multi with executor id 4112 parsl.dataflow.dflow INFO: Standard output for task 4111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/598ce0c6-70a7-4206-beb3-3f66a1b3d88f_isrForCrosstalkSources_855402_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/598ce0c6-70a7-4206-beb3-3f66a1b3d88f_isrForCrosstalkSources_855402_44.stderr parsl.dataflow.dflow DEBUG: Task 4112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4112 has memoization hash b60ade5a50a20d507c8263dc0159bcd8 parsl.dataflow.memoization INFO: Task 4112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4112 try 0 launched on executor multi with executor id 4113 parsl.dataflow.dflow INFO: Standard output for task 4112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e7b33a49-3281-4ec2-b0e3-4b9b5334f290_isrForCrosstalkSources_855443_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e7b33a49-3281-4ec2-b0e3-4b9b5334f290_isrForCrosstalkSources_855443_50.stderr parsl.dataflow.dflow DEBUG: Task 4113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4113 has memoization hash f4fb752bcd949fc6500ceab915ed2e1a parsl.dataflow.memoization INFO: Task 4113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4113 try 0 launched on executor multi with executor id 4114 parsl.dataflow.dflow INFO: Standard output for task 4113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/852416bc-87c3-4c33-99d2-bac582a98731_isrForCrosstalkSources_855394_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/852416bc-87c3-4c33-99d2-bac582a98731_isrForCrosstalkSources_855394_28.stderr parsl.dataflow.dflow DEBUG: Task 4114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4114 has memoization hash 39ca224cfbf42aac166b9e252cbeff3f parsl.dataflow.memoization INFO: Task 4114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4114 try 0 launched on executor multi with executor id 4115 parsl.dataflow.dflow INFO: Standard output for task 4114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/53ddf3cb-5cae-48cf-a923-9005c3f02260_isrForCrosstalkSources_855427_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/53ddf3cb-5cae-48cf-a923-9005c3f02260_isrForCrosstalkSources_855427_42.stderr parsl.dataflow.dflow DEBUG: Task 4115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4115 has memoization hash 29ffc8936e5063bc5881486f2bbf100c parsl.dataflow.memoization INFO: Task 4115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4115 try 0 launched on executor multi with executor id 4116 parsl.dataflow.dflow INFO: Standard output for task 4115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5db95259-5d6a-4ca9-b1bd-0f7be1018392_isrForCrosstalkSources_855430_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5db95259-5d6a-4ca9-b1bd-0f7be1018392_isrForCrosstalkSources_855430_14.stderr parsl.dataflow.dflow DEBUG: Task 4116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4116 has memoization hash 77e5a2d66003ddccdfa8c5e2fd19d2a4 parsl.dataflow.memoization INFO: Task 4116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4116 try 0 launched on executor multi with executor id 4117 parsl.dataflow.dflow INFO: Standard output for task 4116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14d92a46-d586-45b0-b325-35d018e33b70_isrForCrosstalkSources_855433_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14d92a46-d586-45b0-b325-35d018e33b70_isrForCrosstalkSources_855433_51.stderr parsl.dataflow.dflow DEBUG: Task 4117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4117 has memoization hash fcfe7bc37104518d2be6b0375ba00ee1 parsl.dataflow.memoization INFO: Task 4117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4117 try 0 launched on executor multi with executor id 4118 parsl.dataflow.dflow INFO: Standard output for task 4117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b9317006-da27-49c5-a8a8-82b2867c9aa3_isrForCrosstalkSources_855398_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b9317006-da27-49c5-a8a8-82b2867c9aa3_isrForCrosstalkSources_855398_18.stderr parsl.dataflow.dflow DEBUG: Task 4118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4118 has memoization hash 8beb584da233a316425fd4909b54991d parsl.dataflow.memoization INFO: Task 4118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4118 try 0 launched on executor multi with executor id 4119 parsl.dataflow.dflow INFO: Standard output for task 4118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/75effb3a-1910-45f8-819c-f48cb8ec3925_isrForCrosstalkSources_855441_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/75effb3a-1910-45f8-819c-f48cb8ec3925_isrForCrosstalkSources_855441_58.stderr parsl.dataflow.dflow DEBUG: Task 4119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4119 has memoization hash 202050e2c177afd388b0b9f6be3baf58 parsl.dataflow.memoization INFO: Task 4119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4119 try 0 launched on executor multi with executor id 4120 parsl.dataflow.dflow INFO: Standard output for task 4119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/353576fd-e977-4151-8f6a-590ba893de6d_isrForCrosstalkSources_855429_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/353576fd-e977-4151-8f6a-590ba893de6d_isrForCrosstalkSources_855429_24.stderr parsl.dataflow.dflow DEBUG: Task 4120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4120 has memoization hash 9aae12af0132d22b8382f817df3d25af parsl.dataflow.memoization INFO: Task 4120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:45 proc_lsst.multi:170 [INFO] [multi] job_id 16964182 proc_lsst.multi INFO: [multi] job_id 16964182 2024-03-09 14:51:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-09 14:51:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964182 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964182 parsl.executors.status_handling DEBUG: Launched block 156 on executor multi with job ID 16964182 parsl.executors.status_handling INFO: Allocated block ID 157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-157 2024-03-09 14:51:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4120 try 0 launched on executor multi with executor id 4121 parsl.dataflow.dflow INFO: Standard output for task 4120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/20799d7f-c212-46da-8845-1e680d01b231_isrForCrosstalkSources_855429_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/20799d7f-c212-46da-8845-1e680d01b231_isrForCrosstalkSources_855429_49.stderr parsl.dataflow.dflow DEBUG: Task 4121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4121 has memoization hash deaf9664a18e65ebfbdd2ca6eb9c353f parsl.dataflow.memoization INFO: Task 4121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4121 try 0 launched on executor multi with executor id 4122 parsl.dataflow.dflow INFO: Standard output for task 4121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2ad2f5c2-2f6b-4339-bfde-9f4387644f43_isrForCrosstalkSources_855397_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2ad2f5c2-2f6b-4339-bfde-9f4387644f43_isrForCrosstalkSources_855397_1.stderr parsl.dataflow.dflow DEBUG: Task 4122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4122 has memoization hash 611804a3950217edaa7eea70cfb17aae parsl.dataflow.memoization INFO: Task 4122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4122 try 0 launched on executor multi with executor id 4123 parsl.dataflow.dflow INFO: Standard output for task 4122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3f99e8ff-7d44-4635-affe-5ad3cc01375e_isrForCrosstalkSources_855402_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3f99e8ff-7d44-4635-affe-5ad3cc01375e_isrForCrosstalkSources_855402_36.stderr parsl.dataflow.dflow DEBUG: Task 4123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4123 has memoization hash 90c497de1c8b2e1469a55a03ed23ee6d parsl.dataflow.memoization INFO: Task 4123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4123 try 0 launched on executor multi with executor id 4124 parsl.dataflow.dflow INFO: Standard output for task 4123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/20c68603-d8c2-4d79-ad0a-31b70c377d50_isrForCrosstalkSources_855405_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/20c68603-d8c2-4d79-ad0a-31b70c377d50_isrForCrosstalkSources_855405_47.stderr parsl.dataflow.dflow DEBUG: Task 4124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4124 has memoization hash 48bc0ed720e3e102e834397ac2aa922e parsl.dataflow.memoization INFO: Task 4124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4124 try 0 launched on executor multi with executor id 4125 parsl.dataflow.dflow INFO: Standard output for task 4124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f698743d-16c5-46a6-8964-82992d2db3ba_isrForCrosstalkSources_855382_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f698743d-16c5-46a6-8964-82992d2db3ba_isrForCrosstalkSources_855382_43.stderr parsl.dataflow.dflow DEBUG: Task 4125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4125 has memoization hash e1bfbba2bb9a4070a4971a2229520f7b parsl.dataflow.memoization INFO: Task 4125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4125 try 0 launched on executor multi with executor id 4126 parsl.dataflow.dflow INFO: Standard output for task 4125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/25eb4267-0a05-4043-8d69-cac24210672b_isrForCrosstalkSources_855388_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/25eb4267-0a05-4043-8d69-cac24210672b_isrForCrosstalkSources_855388_7.stderr parsl.dataflow.dflow DEBUG: Task 4126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4126 has memoization hash f83d0a0000ebcaae5f8d933ebea3bcfb parsl.dataflow.memoization INFO: Task 4126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4126 try 0 launched on executor multi with executor id 4127 parsl.dataflow.dflow INFO: Standard output for task 4126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/64448712-e368-4676-85dc-992029533586_isrForCrosstalkSources_855458_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/64448712-e368-4676-85dc-992029533586_isrForCrosstalkSources_855458_16.stderr parsl.dataflow.dflow DEBUG: Task 4127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4127 has memoization hash ec701c04306ef97dc8d1fe899d5be7bc parsl.dataflow.memoization INFO: Task 4127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55f12d500b447eacb0ac77cf214e8398 with result from task 157 parsl.dataflow.dflow INFO: Parsl task 4127 try 0 launched on executor multi with executor id 4128 parsl.dataflow.dflow INFO: Standard output for task 4127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d629a469-b3b8-4250-93e7-39d6d91fd69b_isrForCrosstalkSources_855435_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d629a469-b3b8-4250-93e7-39d6d91fd69b_isrForCrosstalkSources_855435_9.stderr parsl.dataflow.dflow DEBUG: Task 4128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Task 4128 has memoization hash 11ec59672626f30ab9662fb2b43d804c parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/76c6b48b-02e7-4333-b0dd-99f18d70ea3a_isrForCrosstalkSources_855381_47.stdout parsl.dataflow.memoization INFO: Task 4128 had no result in cache parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/76c6b48b-02e7-4333-b0dd-99f18d70ea3a_isrForCrosstalkSources_855381_47.stderr parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4128 try 0 launched on executor multi with executor id 4129 parsl.dataflow.dflow INFO: Standard output for task 4128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c18772c5-7925-4d52-bbb0-21e2282e7e90_isrForCrosstalkSources_855411_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c18772c5-7925-4d52-bbb0-21e2282e7e90_isrForCrosstalkSources_855411_47.stderr parsl.dataflow.dflow DEBUG: Task 4129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4129 has memoization hash 383e347c920624a98873707672b1f0c6 parsl.dataflow.memoization INFO: Task 4129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4129 try 0 launched on executor multi with executor id 4130 parsl.dataflow.dflow INFO: Standard output for task 4129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2c2cafa-2235-413a-af0b-3f9e523fdf4b_isrForCrosstalkSources_855382_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2c2cafa-2235-413a-af0b-3f9e523fdf4b_isrForCrosstalkSources_855382_32.stderr parsl.dataflow.dflow DEBUG: Task 4130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4130 has memoization hash fa4603055f121c1de66634d7123ef08a parsl.dataflow.memoization INFO: Task 4130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4130 try 0 launched on executor multi with executor id 4131 parsl.dataflow.dflow INFO: Standard output for task 4130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/9c444596-8be8-4514-954f-dc36a352c09b_isrForCrosstalkSources_855454_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/9c444596-8be8-4514-954f-dc36a352c09b_isrForCrosstalkSources_855454_30.stderr parsl.dataflow.dflow DEBUG: Task 4131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4131 has memoization hash 889ccd3c80ab379be015a7d698a99105 parsl.dataflow.memoization INFO: Task 4131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4131 try 0 launched on executor multi with executor id 4132 parsl.dataflow.dflow INFO: Standard output for task 4131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d20a267f-09d8-42a1-b383-ff841b6bd42a_isrForCrosstalkSources_855457_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d20a267f-09d8-42a1-b383-ff841b6bd42a_isrForCrosstalkSources_855457_38.stderr parsl.dataflow.dflow DEBUG: Task 4132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4132 has memoization hash 8c7bb71b194818d229510a2c2332ba6b parsl.dataflow.memoization INFO: Task 4132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4132 try 0 launched on executor multi with executor id 4133 parsl.dataflow.dflow INFO: Standard output for task 4132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/eee5526f-bf37-43d5-819a-834359de4aa3_isrForCrosstalkSources_855432_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/eee5526f-bf37-43d5-819a-834359de4aa3_isrForCrosstalkSources_855432_10.stderr parsl.dataflow.dflow DEBUG: Task 4133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4133 has memoization hash 5b1a2073d4213a7c8cb382c9633f15c5 parsl.dataflow.memoization INFO: Task 4133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4133 try 0 launched on executor multi with executor id 4134 parsl.dataflow.dflow INFO: Standard output for task 4133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8d227e92-3232-452d-9aee-71b4072a3ffd_isrForCrosstalkSources_855450_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8d227e92-3232-452d-9aee-71b4072a3ffd_isrForCrosstalkSources_855450_9.stderr parsl.dataflow.dflow DEBUG: Task 4134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4134 has memoization hash fced5358a3eeb0d8994fff4ca29a713b parsl.dataflow.memoization INFO: Task 4134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4134 try 0 launched on executor multi with executor id 4135 parsl.dataflow.dflow INFO: Standard output for task 4134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6843b2d3-51f0-4ce4-9151-f966eec9025b_isrForCrosstalkSources_855400_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6843b2d3-51f0-4ce4-9151-f966eec9025b_isrForCrosstalkSources_855400_5.stderr parsl.dataflow.dflow DEBUG: Task 4135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4135 has memoization hash a198a00bee7f2d218fda8f197e406eed parsl.dataflow.memoization INFO: Task 4135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0ee6d0648571c120cc969e524e828c with result from task 314 parsl.dataflow.dflow INFO: Parsl task 4135 try 0 launched on executor multi with executor id 4136 parsl.dataflow.dflow INFO: Standard output for task 4135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/517c4b3b-87cc-42c0-b970-d86164a9d381_isrForCrosstalkSources_855424_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/517c4b3b-87cc-42c0-b970-d86164a9d381_isrForCrosstalkSources_855424_52.stderr parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow DEBUG: Task 4136 will be sent to executor multi parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/b776f1ac-0eb7-4ea2-ba3f-49c05e7a12ab_isrForCrosstalkSources_855445_28.stdout parsl.dataflow.dflow INFO: Task 4136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/b776f1ac-0eb7-4ea2-ba3f-49c05e7a12ab_isrForCrosstalkSources_855445_28.stderr parsl.dataflow.dflow DEBUG: Task 4136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4136 has memoization hash 173fcd1d4550c2daad66e0a20967f8da parsl.dataflow.memoization INFO: Task 4136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4136 try 0 launched on executor multi with executor id 4137 parsl.dataflow.dflow INFO: Standard output for task 4136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/70061ece-7f89-484a-b6ec-79ddc9a5a72a_isrForCrosstalkSources_855381_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/70061ece-7f89-484a-b6ec-79ddc9a5a72a_isrForCrosstalkSources_855381_10.stderr parsl.dataflow.dflow DEBUG: Task 4137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4137 has memoization hash c1280a3ec8ac33876057cfa4a057f826 parsl.dataflow.memoization INFO: Task 4137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4137 try 0 launched on executor multi with executor id 4138 parsl.dataflow.dflow INFO: Standard output for task 4137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cc192868-2694-4336-84ea-0fbcf59fcfbd_isrForCrosstalkSources_855415_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cc192868-2694-4336-84ea-0fbcf59fcfbd_isrForCrosstalkSources_855415_4.stderr parsl.dataflow.dflow DEBUG: Task 4138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4138 has memoization hash b71ff2cdefa637c001e4245ba36fe42a parsl.dataflow.memoization INFO: Task 4138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2da912c5fdc256b7499ec21ae553974f with result from task 566 parsl.dataflow.dflow INFO: Parsl task 4138 try 0 launched on executor multi with executor id 4139 parsl.dataflow.dflow INFO: Standard output for task 4138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccdbee24-8d33-421c-9e05-7763798cd891_isrForCrosstalkSources_855433_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccdbee24-8d33-421c-9e05-7763798cd891_isrForCrosstalkSources_855433_49.stderr parsl.dataflow.dflow DEBUG: Task 4139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4139 has memoization hash 89b23fd776f3b9a07729d6654122b666 parsl.dataflow.memoization INFO: Task 4139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4139 try 0 launched on executor multi with executor id 4140 parsl.dataflow.dflow INFO: Standard output for task 4139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/96c4d253-85b3-415d-bc04-4e14c262de9b_isrForCrosstalkSources_855457_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/96c4d253-85b3-415d-bc04-4e14c262de9b_isrForCrosstalkSources_855457_12.stderr parsl.dataflow.dflow DEBUG: Task 4140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4140 has memoization hash cd0d7dc1d3d88aa929ed5ee39d3dcf5a parsl.dataflow.memoization INFO: Task 4140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/078c7896-e2c2-4479-8231-ebd16d6c6a40_isrForCrosstalkSources_855406_34.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/078c7896-e2c2-4479-8231-ebd16d6c6a40_isrForCrosstalkSources_855406_34.stderr parsl.dataflow.dflow INFO: Parsl task 4140 try 0 launched on executor multi with executor id 4141 parsl.dataflow.dflow INFO: Standard output for task 4140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/90d8a7d0-6457-40f0-8bb7-6d7a7a8b6210_isrForCrosstalkSources_855400_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/90d8a7d0-6457-40f0-8bb7-6d7a7a8b6210_isrForCrosstalkSources_855400_45.stderr parsl.dataflow.dflow DEBUG: Task 4141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4141 has memoization hash 73aa250660285b231a0e5fe1f49d5b67 parsl.dataflow.memoization INFO: Task 4141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4141 try 0 launched on executor multi with executor id 4142 parsl.dataflow.dflow INFO: Standard output for task 4141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/60fa02d1-7fdf-4120-9542-4eca5566609f_isrForCrosstalkSources_855421_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/60fa02d1-7fdf-4120-9542-4eca5566609f_isrForCrosstalkSources_855421_40.stderr parsl.dataflow.dflow DEBUG: Task 4142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4142 has memoization hash 171aab325b6205d514f0af1bc3ac495b parsl.dataflow.memoization INFO: Task 4142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:45 proc_lsst.multi:170 [INFO] [multi] job_id 16964183 proc_lsst.multi INFO: [multi] job_id 16964183 2024-03-09 14:51:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-09 14:51:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964183 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964183 parsl.executors.status_handling DEBUG: Launched block 157 on executor multi with job ID 16964183 parsl.executors.status_handling INFO: Allocated block ID 158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-158 2024-03-09 14:51:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-09 14:51:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4142 try 0 launched on executor multi with executor id 4143 parsl.dataflow.dflow INFO: Standard output for task 4142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a51b7543-d2ce-45ab-b0c6-124392513c36_isrForCrosstalkSources_855397_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a51b7543-d2ce-45ab-b0c6-124392513c36_isrForCrosstalkSources_855397_37.stderr parsl.dataflow.dflow DEBUG: Task 4143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4143 has memoization hash b305b6859e46febf2f7b4528937737d3 parsl.dataflow.memoization INFO: Task 4143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4143 try 0 launched on executor multi with executor id 4144 parsl.dataflow.dflow INFO: Standard output for task 4143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3e3183e4-70ee-4e01-9bcc-be6410adf9a1_isrForCrosstalkSources_855411_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3e3183e4-70ee-4e01-9bcc-be6410adf9a1_isrForCrosstalkSources_855411_60.stderr parsl.dataflow.dflow DEBUG: Task 4144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4144 has memoization hash a0120721f488e93f8d998d5cc8f527b8 parsl.dataflow.memoization INFO: Task 4144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4144 try 0 launched on executor multi with executor id 4145 parsl.dataflow.dflow INFO: Standard output for task 4144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d4a8232d-2c9b-4913-b895-a1e7d07e20ee_isrForCrosstalkSources_855384_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d4a8232d-2c9b-4913-b895-a1e7d07e20ee_isrForCrosstalkSources_855384_5.stderr parsl.dataflow.dflow DEBUG: Task 4145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4145 has memoization hash 82a67901b9d70ca186605d7a235b93e3 parsl.dataflow.memoization INFO: Task 4145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4145 try 0 launched on executor multi with executor id 4146 parsl.dataflow.dflow INFO: Standard output for task 4145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1de72c2a-faee-4395-a754-2c1223d3cd15_isrForCrosstalkSources_855381_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1de72c2a-faee-4395-a754-2c1223d3cd15_isrForCrosstalkSources_855381_37.stderr parsl.dataflow.dflow DEBUG: Task 4146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4146 has memoization hash cadec2a6d357de2060c629103aff49f5 parsl.dataflow.memoization INFO: Task 4146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4146 try 0 launched on executor multi with executor id 4147 parsl.dataflow.dflow INFO: Standard output for task 4146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1d5ed7f7-ec75-4e70-ab09-a83d491c763f_isrForCrosstalkSources_855439_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1d5ed7f7-ec75-4e70-ab09-a83d491c763f_isrForCrosstalkSources_855439_50.stderr parsl.dataflow.dflow DEBUG: Task 4147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4147 has memoization hash 01732ec1fc47b36b09d46193049192dc parsl.dataflow.memoization INFO: Task 4147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4147 try 0 launched on executor multi with executor id 4148 parsl.dataflow.dflow INFO: Standard output for task 4147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/118d71b1-6a56-425b-a403-bdc80f4056f4_isrForCrosstalkSources_855414_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/118d71b1-6a56-425b-a403-bdc80f4056f4_isrForCrosstalkSources_855414_22.stderr parsl.dataflow.dflow DEBUG: Task 4148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4148 has memoization hash bbb6e2b8fc31e19d1b6fb1261a908faf parsl.dataflow.memoization INFO: Task 4148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4148 try 0 launched on executor multi with executor id 4149 parsl.dataflow.dflow INFO: Standard output for task 4148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/41297312-710a-48d0-8dbf-a6042b172345_isrForCrosstalkSources_855440_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/41297312-710a-48d0-8dbf-a6042b172345_isrForCrosstalkSources_855440_24.stderr parsl.dataflow.dflow DEBUG: Task 4149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4149 has memoization hash d43c104265d99e1cd0225c05efb8b48f parsl.dataflow.memoization INFO: Task 4149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4149 try 0 launched on executor multi with executor id 4150 parsl.dataflow.dflow INFO: Standard output for task 4149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/77ef7869-4cf8-44f3-907b-b2152900c221_isrForCrosstalkSources_855447_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/77ef7869-4cf8-44f3-907b-b2152900c221_isrForCrosstalkSources_855447_13.stderr parsl.dataflow.dflow DEBUG: Task 4150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4150 has memoization hash 8918071289b3616d803d97414369eedb parsl.dataflow.memoization INFO: Task 4150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4150 try 0 launched on executor multi with executor id 4151 parsl.dataflow.dflow INFO: Standard output for task 4150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c4863a77-8f04-4bf8-9276-63c01e2809ff_isrForCrosstalkSources_855443_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c4863a77-8f04-4bf8-9276-63c01e2809ff_isrForCrosstalkSources_855443_56.stderr parsl.dataflow.dflow DEBUG: Task 4151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4151 has memoization hash 562a2e645ab923ee64fcad1ac6b0bbf3 parsl.dataflow.memoization INFO: Task 4151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4151 try 0 launched on executor multi with executor id 4152 parsl.dataflow.dflow INFO: Standard output for task 4151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3080f4d8-1cce-4f31-9fc7-a59ddd877062_isrForCrosstalkSources_855454_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3080f4d8-1cce-4f31-9fc7-a59ddd877062_isrForCrosstalkSources_855454_4.stderr parsl.dataflow.dflow DEBUG: Task 4152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4152 has memoization hash 605ddd9b358d4aa647a52f9bad81131f parsl.dataflow.memoization INFO: Task 4152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4152 try 0 launched on executor multi with executor id 4153 parsl.dataflow.dflow INFO: Standard output for task 4152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/e7110059-3296-4f5f-9bf8-002c6c76eec6_isrForCrosstalkSources_855448_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/e7110059-3296-4f5f-9bf8-002c6c76eec6_isrForCrosstalkSources_855448_5.stderr parsl.dataflow.dflow DEBUG: Task 4153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4153 has memoization hash 683c2757186c1c5461d4eb8b4882024e parsl.dataflow.memoization INFO: Task 4153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ead4cbccd89422ac73616a3b645674 with result from task 171 parsl.dataflow.dflow INFO: Parsl task 4153 try 0 launched on executor multi with executor id 4154 parsl.dataflow.dflow INFO: Standard output for task 4153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/45e34636-ec02-40de-b586-d632af0a6300_isrForCrosstalkSources_855383_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/45e34636-ec02-40de-b586-d632af0a6300_isrForCrosstalkSources_855383_3.stderr parsl.dataflow.dflow DEBUG: Task 4154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4154 has memoization hash 1493e0e1b7b717b84f807c9a7e8aa7be parsl.dataflow.memoization INFO: Task 4154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/41b25252-2e75-411c-b7a7-f055ed8dfb62_isrForCrosstalkSources_855385_50.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/41b25252-2e75-411c-b7a7-f055ed8dfb62_isrForCrosstalkSources_855385_50.stderr parsl.dataflow.dflow INFO: Parsl task 4154 try 0 launched on executor multi with executor id 4155 parsl.dataflow.dflow INFO: Standard output for task 4154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f0f3166d-808a-445c-ba01-b8273d30c6e5_isrForCrosstalkSources_855425_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f0f3166d-808a-445c-ba01-b8273d30c6e5_isrForCrosstalkSources_855425_9.stderr parsl.dataflow.dflow DEBUG: Task 4155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4155 has memoization hash a1c5cbd71cdc7a1087903cfa4266da08 parsl.dataflow.memoization INFO: Task 4155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4155 try 0 launched on executor multi with executor id 4156 parsl.dataflow.dflow INFO: Standard output for task 4155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec1642b2-0fdd-456e-aec2-a9e79033be57_isrForCrosstalkSources_855413_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec1642b2-0fdd-456e-aec2-a9e79033be57_isrForCrosstalkSources_855413_58.stderr parsl.dataflow.dflow DEBUG: Task 4156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4156 has memoization hash efc7a8eb5a2c85bcf42a7f43d20daf24 parsl.dataflow.memoization INFO: Task 4156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4156 try 0 launched on executor multi with executor id 4157 parsl.dataflow.dflow INFO: Standard output for task 4156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c26332c2-7f3a-45f2-870e-b954f9c5ba46_isrForCrosstalkSources_855397_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c26332c2-7f3a-45f2-870e-b954f9c5ba46_isrForCrosstalkSources_855397_4.stderr parsl.dataflow.dflow DEBUG: Task 4157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4157 has memoization hash 501f44d41fb78b404aa93c3ce7c1e64b parsl.dataflow.memoization INFO: Task 4157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4157 try 0 launched on executor multi with executor id 4158 parsl.dataflow.dflow INFO: Standard output for task 4157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/ed47a31d-69f0-4cec-b149-57e1b66f026a_isrForCrosstalkSources_855423_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/ed47a31d-69f0-4cec-b149-57e1b66f026a_isrForCrosstalkSources_855423_35.stderr parsl.dataflow.dflow DEBUG: Task 4158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4158 has memoization hash a346aff27f695ab4c97774180170413d parsl.dataflow.memoization INFO: Task 4158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4158 try 0 launched on executor multi with executor id 4159 parsl.dataflow.dflow INFO: Standard output for task 4158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/2e1445f8-2549-42a5-bdea-7aae5e8f54ce_isrForCrosstalkSources_855438_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/2e1445f8-2549-42a5-bdea-7aae5e8f54ce_isrForCrosstalkSources_855438_31.stderr parsl.dataflow.dflow DEBUG: Task 4159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4159 has memoization hash cd3214b4269bdbc65ba81ac0086a458e parsl.dataflow.memoization INFO: Task 4159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4159 try 0 launched on executor multi with executor id 4160 parsl.dataflow.dflow INFO: Standard output for task 4159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6ac82cc4-d32b-4aa4-bdf3-38cf01e394ce_isrForCrosstalkSources_855400_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6ac82cc4-d32b-4aa4-bdf3-38cf01e394ce_isrForCrosstalkSources_855400_46.stderr parsl.dataflow.dflow DEBUG: Task 4160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4160 has memoization hash 9380fcd9f4d247ced864aa8e4f2935b4 parsl.dataflow.memoization INFO: Task 4160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:46 proc_lsst.multi:170 [INFO] [multi] job_id 16964184 proc_lsst.multi INFO: [multi] job_id 16964184 2024-03-09 14:51:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-09 14:51:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964184 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964184 parsl.executors.status_handling DEBUG: Launched block 158 on executor multi with job ID 16964184 parsl.executors.status_handling INFO: Allocated block ID 159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-159 2024-03-09 14:51:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4160 try 0 launched on executor multi with executor id 4161 parsl.dataflow.dflow INFO: Standard output for task 4160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7d8c7c2e-8f09-454d-8847-76993cec3f84_isrForCrosstalkSources_855397_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7d8c7c2e-8f09-454d-8847-76993cec3f84_isrForCrosstalkSources_855397_21.stderr parsl.dataflow.dflow DEBUG: Task 4161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4161 has memoization hash d7ee370e7f09816934ea41e1cb01cf1d parsl.dataflow.memoization INFO: Task 4161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4161 try 0 launched on executor multi with executor id 4162 parsl.dataflow.dflow INFO: Standard output for task 4161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/4a1c8034-db71-4eee-b090-3aca2c26728d_isrForCrosstalkSources_855417_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/4a1c8034-db71-4eee-b090-3aca2c26728d_isrForCrosstalkSources_855417_43.stderr parsl.dataflow.dflow DEBUG: Task 4162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4162 has memoization hash 7848470ae4e9860b26feca3fc8c559b3 parsl.dataflow.memoization INFO: Task 4162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4162 try 0 launched on executor multi with executor id 4163 parsl.dataflow.dflow INFO: Standard output for task 4162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a914f258-0a3d-470e-beb2-5876ad95b178_isrForCrosstalkSources_855396_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a914f258-0a3d-470e-beb2-5876ad95b178_isrForCrosstalkSources_855396_1.stderr parsl.dataflow.dflow DEBUG: Task 4163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4163 has memoization hash ef0832b62e69f90a723907a4bd06477e parsl.dataflow.memoization INFO: Task 4163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5b2e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4163 try 0 launched on executor multi with executor id 4164 parsl.dataflow.dflow INFO: Standard output for task 4163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a220c9d-e97e-425e-aee5-2d1e72df5708_isrForCrosstalkSources_855434_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a220c9d-e97e-425e-aee5-2d1e72df5708_isrForCrosstalkSources_855434_12.stderr parsl.dataflow.dflow DEBUG: Task 4164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4164 has memoization hash 565c116a4d0a530410aee1c2d2f0749c parsl.dataflow.memoization INFO: Task 4164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b73f23fd04044f4721d7b4032e23dcdd with result from task 311 parsl.dataflow.dflow INFO: Parsl task 4164 try 0 launched on executor multi with executor id 4165 parsl.dataflow.dflow INFO: Standard output for task 4164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/02d641e1-cf3d-40b4-8fe7-edb00436e476_isrForCrosstalkSources_855410_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/02d641e1-cf3d-40b4-8fe7-edb00436e476_isrForCrosstalkSources_855410_6.stderr parsl.dataflow.dflow DEBUG: Task 4165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4165 has memoization hash 0ed09d885399c271251e06c8b4826615 parsl.dataflow.memoization INFO: Task 4165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/de7f6216-cf0d-47ff-96ab-a0baa21e47b3_isrForCrosstalkSources_855435_31.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/de7f6216-cf0d-47ff-96ab-a0baa21e47b3_isrForCrosstalkSources_855435_31.stderr parsl.dataflow.dflow INFO: Parsl task 4165 try 0 launched on executor multi with executor id 4166 parsl.dataflow.dflow INFO: Standard output for task 4165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/658de0ff-e747-4978-a443-d63110f81e42_isrForCrosstalkSources_855409_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/658de0ff-e747-4978-a443-d63110f81e42_isrForCrosstalkSources_855409_13.stderr parsl.dataflow.dflow DEBUG: Task 4166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4166 has memoization hash ed2164d7fa16c67c8d4c9dbb92ce49cc parsl.dataflow.memoization INFO: Task 4166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4166 try 0 launched on executor multi with executor id 4167 parsl.dataflow.dflow INFO: Standard output for task 4166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a431501a-a3e9-4383-905a-0e2c7f634de1_isrForCrosstalkSources_855443_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a431501a-a3e9-4383-905a-0e2c7f634de1_isrForCrosstalkSources_855443_40.stderr parsl.dataflow.dflow DEBUG: Task 4167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4167 has memoization hash 33cc921b05ce02bac1ec908592cb57c3 parsl.dataflow.memoization INFO: Task 4167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4167 try 0 launched on executor multi with executor id 4168 parsl.dataflow.dflow INFO: Standard output for task 4167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/84020382-f54a-4bdf-8616-a5e0b9e30b0e_isrForCrosstalkSources_855446_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/84020382-f54a-4bdf-8616-a5e0b9e30b0e_isrForCrosstalkSources_855446_41.stderr parsl.dataflow.dflow DEBUG: Task 4168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4168 has memoization hash 00281097ac14442dc3f94a6deb2bee5c parsl.dataflow.memoization INFO: Task 4168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4168 try 0 launched on executor multi with executor id 4169 parsl.dataflow.dflow INFO: Standard output for task 4168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccb2b8a7-7cda-4374-94f2-713b4409ce54_isrForCrosstalkSources_855403_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccb2b8a7-7cda-4374-94f2-713b4409ce54_isrForCrosstalkSources_855403_26.stderr parsl.dataflow.dflow DEBUG: Task 4169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4169 has memoization hash 922e174a5f7616f270e7ae03c431c14b parsl.dataflow.memoization INFO: Task 4169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4169 try 0 launched on executor multi with executor id 4170 parsl.dataflow.dflow INFO: Standard output for task 4169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c8e14736-f2b0-4a07-9768-c3f2cb8c99c7_isrForCrosstalkSources_855457_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c8e14736-f2b0-4a07-9768-c3f2cb8c99c7_isrForCrosstalkSources_855457_11.stderr parsl.dataflow.dflow DEBUG: Task 4170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4170 has memoization hash 86d636619292385e3d6dd5f087c4af23 parsl.dataflow.memoization INFO: Task 4170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a8ab7281a00da5b585015741b57b39c with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3074b678-b8e6-4a6f-b774-e1e18324394c_isrForCrosstalkSources_855383_54.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3074b678-b8e6-4a6f-b774-e1e18324394c_isrForCrosstalkSources_855383_54.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e5689742f82c287954959ef0925ecb1 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/58840948-8d1a-40b5-b2c9-9c174d4c875b_isrForCrosstalkSources_855379_26.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/58840948-8d1a-40b5-b2c9-9c174d4c875b_isrForCrosstalkSources_855379_26.stderr parsl.dataflow.dflow INFO: Parsl task 4170 try 0 launched on executor multi with executor id 4171 parsl.dataflow.dflow INFO: Standard output for task 4170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0fa490fb-573a-4b3a-82a1-16ee1d2d9060_isrForCrosstalkSources_855418_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0fa490fb-573a-4b3a-82a1-16ee1d2d9060_isrForCrosstalkSources_855418_44.stderr parsl.dataflow.dflow DEBUG: Task 4171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4171 has memoization hash 6025d3675caa30b1a831b62bf069de24 parsl.dataflow.memoization INFO: Task 4171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4171 try 0 launched on executor multi with executor id 4172 parsl.dataflow.dflow INFO: Standard output for task 4171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ad55fdb8-cec8-4dc1-a842-423e572e7bff_isrForCrosstalkSources_855410_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ad55fdb8-cec8-4dc1-a842-423e572e7bff_isrForCrosstalkSources_855410_56.stderr parsl.dataflow.dflow DEBUG: Task 4172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4172 has memoization hash cf85fc0fbe103b7c13fdf7e5ea49df31 parsl.dataflow.memoization INFO: Task 4172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4172 try 0 launched on executor multi with executor id 4173 parsl.dataflow.dflow INFO: Standard output for task 4172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/d6122729-3df5-4e31-be7f-9c360016f879_isrForCrosstalkSources_855434_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/d6122729-3df5-4e31-be7f-9c360016f879_isrForCrosstalkSources_855434_55.stderr parsl.dataflow.dflow DEBUG: Task 4173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4173 has memoization hash 8d7e64a13ec97ecc972ddb20101156c4 parsl.dataflow.memoization INFO: Task 4173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4173 try 0 launched on executor multi with executor id 4174 parsl.dataflow.dflow INFO: Standard output for task 4173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8be216d1-b579-442e-bef4-a65be791c8aa_isrForCrosstalkSources_855384_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8be216d1-b579-442e-bef4-a65be791c8aa_isrForCrosstalkSources_855384_31.stderr parsl.dataflow.dflow DEBUG: Task 4174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4174 has memoization hash 07ba00d8b2cb589029a238d992298a80 parsl.dataflow.memoization INFO: Task 4174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4174 try 0 launched on executor multi with executor id 4175 parsl.dataflow.dflow INFO: Standard output for task 4174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8a9a6031-92ea-4f85-ad17-cabbf82a6df5_isrForCrosstalkSources_855414_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8a9a6031-92ea-4f85-ad17-cabbf82a6df5_isrForCrosstalkSources_855414_27.stderr parsl.dataflow.dflow DEBUG: Task 4175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4175 has memoization hash 6de4f5337fb9212fff289acf9e11a561 parsl.dataflow.memoization INFO: Task 4175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4175 try 0 launched on executor multi with executor id 4176 parsl.dataflow.dflow INFO: Standard output for task 4175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/12a23c94-291a-4d73-85bb-a623d1cc7eac_isrForCrosstalkSources_855445_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/12a23c94-291a-4d73-85bb-a623d1cc7eac_isrForCrosstalkSources_855445_57.stderr parsl.dataflow.dflow DEBUG: Task 4176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4176 has memoization hash 254c28ce7ab6df2dbd7a6ace1117256e parsl.dataflow.memoization INFO: Task 4176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4176 try 0 launched on executor multi with executor id 4177 parsl.dataflow.dflow INFO: Standard output for task 4176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/0d02385a-ad6d-4709-82a9-1b95b653b203_isrForCrosstalkSources_855422_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/0d02385a-ad6d-4709-82a9-1b95b653b203_isrForCrosstalkSources_855422_45.stderr parsl.dataflow.dflow DEBUG: Task 4177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4177 has memoization hash d1a7211939bf6d78f9faef0d6522dc22 parsl.dataflow.memoization INFO: Task 4177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8e5af84220fa521ab1d0d7af25d840 with result from task 182 parsl.dataflow.dflow INFO: Parsl task 4177 try 0 launched on executor multi with executor id 4178 parsl.dataflow.dflow INFO: Standard output for task 4177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf54b091-92b1-4059-b987-c077ed566754_isrForCrosstalkSources_855401_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf54b091-92b1-4059-b987-c077ed566754_isrForCrosstalkSources_855401_56.stderr parsl.dataflow.dflow DEBUG: Task 4178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4178 has memoization hash d825addf9eaf9113e41920179e7cfa5d parsl.dataflow.memoization INFO: Task 4178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9d6d25bc-25ea-47ce-902f-a2440481c620_isrForCrosstalkSources_855428_38.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9d6d25bc-25ea-47ce-902f-a2440481c620_isrForCrosstalkSources_855428_38.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eea19bbe3ace23c38560872a3ae1181 with result from task 567 parsl.dataflow.dflow INFO: Parsl task 4178 try 0 launched on executor multi with executor id 4179 parsl.dataflow.dflow INFO: Standard output for task 4178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f987570c-0fc7-42cb-9f87-1a4789ad53bd_isrForCrosstalkSources_855412_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f987570c-0fc7-42cb-9f87-1a4789ad53bd_isrForCrosstalkSources_855412_29.stderr parsl.dataflow.dflow DEBUG: Task 4179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4179 has memoization hash 5a3641ea302e4411a55a3e64db6a3a5c parsl.dataflow.memoization INFO: Task 4179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c4c1a9e-ad38-4339-bce7-9a9c9bff7437_isrForCrosstalkSources_855400_56.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c4c1a9e-ad38-4339-bce7-9a9c9bff7437_isrForCrosstalkSources_855400_56.stderr parsl.dataflow.dflow INFO: Parsl task 4179 try 0 launched on executor multi with executor id 4180 parsl.dataflow.dflow INFO: Standard output for task 4179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/90b658c9-6884-47a6-8237-9d09847bad06_isrForCrosstalkSources_855412_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/90b658c9-6884-47a6-8237-9d09847bad06_isrForCrosstalkSources_855412_11.stderr parsl.dataflow.dflow DEBUG: Task 4180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4180 has memoization hash bfcb2104c26e26a57abcfd31190b094b parsl.dataflow.memoization INFO: Task 4180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4180 try 0 launched on executor multi with executor id 4181 parsl.dataflow.dflow INFO: Standard output for task 4180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/b7ad6cfd-5d12-479b-8c00-221924ecad5a_isrForCrosstalkSources_855434_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/b7ad6cfd-5d12-479b-8c00-221924ecad5a_isrForCrosstalkSources_855434_5.stderr parsl.dataflow.dflow DEBUG: Task 4181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4181 has memoization hash 44bcf52ff91433f22a7b4e3877f2019d parsl.dataflow.memoization INFO: Task 4181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4181 try 0 launched on executor multi with executor id 4182 parsl.dataflow.dflow INFO: Standard output for task 4181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f1648c5e-8999-43bc-b06e-5310b4a14c43_isrForCrosstalkSources_855379_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f1648c5e-8999-43bc-b06e-5310b4a14c43_isrForCrosstalkSources_855379_37.stderr parsl.dataflow.dflow DEBUG: Task 4182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4182 has memoization hash 0212f337b9c0b110f44e0983ebec2068 parsl.dataflow.memoization INFO: Task 4182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4182 try 0 launched on executor multi with executor id 4183 parsl.dataflow.dflow INFO: Standard output for task 4182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0350d338-c59e-46be-ac66-3e464fd8b46e_isrForCrosstalkSources_855443_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0350d338-c59e-46be-ac66-3e464fd8b46e_isrForCrosstalkSources_855443_21.stderr parsl.dataflow.dflow DEBUG: Task 4183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4183 has memoization hash 881f69890bc3bc3313c9ea8f3b33f90a parsl.dataflow.memoization INFO: Task 4183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4183 try 0 launched on executor multi with executor id 4184 parsl.dataflow.dflow INFO: Standard output for task 4183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/dbe6283f-5e5b-4bbf-86e7-153733950c48_isrForCrosstalkSources_855398_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/dbe6283f-5e5b-4bbf-86e7-153733950c48_isrForCrosstalkSources_855398_50.stderr parsl.dataflow.dflow DEBUG: Task 4184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4184 has memoization hash 8e4dc5cd2001be2d2dd116a884bae561 parsl.dataflow.memoization INFO: Task 4184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4184 try 0 launched on executor multi with executor id 4185 parsl.dataflow.dflow INFO: Standard output for task 4184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9b7eba53-f39c-40ad-9b05-afc9c1fb64e9_isrForCrosstalkSources_855422_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9b7eba53-f39c-40ad-9b05-afc9c1fb64e9_isrForCrosstalkSources_855422_47.stderr parsl.dataflow.dflow DEBUG: Task 4185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4185 has memoization hash 3dca36f77790c80ca60be2d851db1a56 parsl.dataflow.memoization INFO: Task 4185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4185 try 0 launched on executor multi with executor id 4186 parsl.dataflow.dflow INFO: Standard output for task 4185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/be43ed3c-dfc3-456c-80c3-25234e0ebef0_isrForCrosstalkSources_855448_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/be43ed3c-dfc3-456c-80c3-25234e0ebef0_isrForCrosstalkSources_855448_15.stderr parsl.dataflow.dflow DEBUG: Task 4186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4186 has memoization hash 7d3cf36735f8c2ada85d590ad182fbc9 parsl.dataflow.memoization INFO: Task 4186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4186 try 0 launched on executor multi with executor id 4187 parsl.dataflow.dflow INFO: Standard output for task 4186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d5ccbfd-feda-4f2b-8818-9a40a22df10f_isrForCrosstalkSources_855444_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d5ccbfd-feda-4f2b-8818-9a40a22df10f_isrForCrosstalkSources_855444_24.stderr parsl.dataflow.dflow DEBUG: Task 4187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4187 has memoization hash a29d97be5572504a652175e11b1c713e parsl.dataflow.memoization INFO: Task 4187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4187 try 0 launched on executor multi with executor id 4188 parsl.dataflow.dflow INFO: Standard output for task 4187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/6068ea66-3b53-4491-bdfd-0958e520c5ed_isrForCrosstalkSources_855422_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/6068ea66-3b53-4491-bdfd-0958e520c5ed_isrForCrosstalkSources_855422_55.stderr parsl.dataflow.dflow DEBUG: Task 4188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4188 has memoization hash cbea3d9c19e8df475dde85292da71bff parsl.dataflow.memoization INFO: Task 4188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4188 try 0 launched on executor multi with executor id 4189 parsl.dataflow.dflow INFO: Standard output for task 4188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c5581362-f072-42e1-89a5-485914f05372_isrForCrosstalkSources_855415_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c5581362-f072-42e1-89a5-485914f05372_isrForCrosstalkSources_855415_37.stderr parsl.dataflow.dflow DEBUG: Task 4189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4189 has memoization hash ec4159a69ef97acd976287b4575f6ba4 parsl.dataflow.memoization INFO: Task 4189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4189 try 0 launched on executor multi with executor id 4190 parsl.dataflow.dflow INFO: Standard output for task 4189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/49b2897d-28ce-41eb-acaf-76e9f1cddb4e_isrForCrosstalkSources_855393_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/49b2897d-28ce-41eb-acaf-76e9f1cddb4e_isrForCrosstalkSources_855393_9.stderr parsl.dataflow.dflow DEBUG: Task 4190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4190 has memoization hash 9c0ac68cac9c4ed76b52ad63593b1991 parsl.dataflow.memoization INFO: Task 4190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4190 try 0 launched on executor multi with executor id 4191 parsl.dataflow.dflow INFO: Standard output for task 4190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0792edd5-eed0-4ee9-8856-73abebd07371_isrForCrosstalkSources_855390_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0792edd5-eed0-4ee9-8856-73abebd07371_isrForCrosstalkSources_855390_24.stderr parsl.dataflow.dflow DEBUG: Task 4191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4191 has memoization hash afef275d12c1fa67270fefcb09aa956b parsl.dataflow.memoization INFO: Task 4191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4191 try 0 launched on executor multi with executor id 4192 parsl.dataflow.dflow INFO: Standard output for task 4191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c6b4eda9-dece-4f30-8f7b-06f21aabf110_isrForCrosstalkSources_855387_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c6b4eda9-dece-4f30-8f7b-06f21aabf110_isrForCrosstalkSources_855387_51.stderr parsl.dataflow.dflow DEBUG: Task 4192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4192 has memoization hash d519b4d114d4ece8e02b9bf1074c33cb parsl.dataflow.memoization INFO: Task 4192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e93aa82cc639e51e885f61e966140a5b with result from task 525 2024-03-09 14:51:46 proc_lsst.multi:170 [INFO] [multi] job_id 16964185 proc_lsst.multi INFO: [multi] job_id 16964185 2024-03-09 14:51:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-09 14:51:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964185 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964185 parsl.executors.status_handling DEBUG: Launched block 159 on executor multi with job ID 16964185 parsl.executors.status_handling INFO: Allocated block ID 160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-160 2024-03-09 14:51:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-09 14:51:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4192 try 0 launched on executor multi with executor id 4193 parsl.dataflow.dflow INFO: Standard output for task 4192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/262640a6-1283-47c2-8696-98df33b3acaa_isrForCrosstalkSources_855426_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/262640a6-1283-47c2-8696-98df33b3acaa_isrForCrosstalkSources_855426_59.stderr parsl.dataflow.dflow DEBUG: Task 4193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4193 has memoization hash cb802b5b13dc9a2f7ff694d0ef44bcc6 parsl.dataflow.memoization INFO: Task 4193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0835e213-e26a-49de-bb90-ffd1b7483e0b_isrForCrosstalkSources_855402_26.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0835e213-e26a-49de-bb90-ffd1b7483e0b_isrForCrosstalkSources_855402_26.stderr parsl.dataflow.dflow INFO: Parsl task 4193 try 0 launched on executor multi with executor id 4194 parsl.dataflow.dflow INFO: Standard output for task 4193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/663bcc08-6a7e-4c08-b926-74c23a7f8a6d_isrForCrosstalkSources_855412_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/663bcc08-6a7e-4c08-b926-74c23a7f8a6d_isrForCrosstalkSources_855412_10.stderr parsl.dataflow.dflow DEBUG: Task 4194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4194 has memoization hash cb3d4eb9dc5b843b6eca46d1a88e1554 parsl.dataflow.memoization INFO: Task 4194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4194 try 0 launched on executor multi with executor id 4195 parsl.dataflow.dflow INFO: Standard output for task 4194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/bd2ede90-4b5e-4fc6-9c42-a39bb40fec7a_isrForCrosstalkSources_855433_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/bd2ede90-4b5e-4fc6-9c42-a39bb40fec7a_isrForCrosstalkSources_855433_25.stderr parsl.dataflow.dflow DEBUG: Task 4195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4195 has memoization hash cfb272677264fe7aff0b7a306b084727 parsl.dataflow.memoization INFO: Task 4195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4195 try 0 launched on executor multi with executor id 4196 parsl.dataflow.dflow INFO: Standard output for task 4195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/0e630fbb-87c3-4528-9292-8fd6977a4198_isrForCrosstalkSources_855449_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/0e630fbb-87c3-4528-9292-8fd6977a4198_isrForCrosstalkSources_855449_60.stderr parsl.dataflow.dflow DEBUG: Task 4196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4196 has memoization hash f7d6bc8be8f71797b94d3570011d8e49 parsl.dataflow.memoization INFO: Task 4196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4196 try 0 launched on executor multi with executor id 4197 parsl.dataflow.dflow INFO: Standard output for task 4196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f1a4ab5-c3cf-4e15-8121-0791ed0cfa2e_isrForCrosstalkSources_855441_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f1a4ab5-c3cf-4e15-8121-0791ed0cfa2e_isrForCrosstalkSources_855441_46.stderr parsl.dataflow.dflow DEBUG: Task 4197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4197 has memoization hash 7b512b82bba2de2ba17a47c18b7f3c1c parsl.dataflow.memoization INFO: Task 4197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4197 try 0 launched on executor multi with executor id 4198 parsl.dataflow.dflow INFO: Standard output for task 4197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78c2ceee-d589-4537-a27a-ee99e87d410b_isrForCrosstalkSources_855433_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78c2ceee-d589-4537-a27a-ee99e87d410b_isrForCrosstalkSources_855433_50.stderr parsl.dataflow.dflow DEBUG: Task 4198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4198 has memoization hash 83a1942de1ece6be2d91ecbf1dbdb8fa parsl.dataflow.memoization INFO: Task 4198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4198 try 0 launched on executor multi with executor id 4199 parsl.dataflow.dflow INFO: Standard output for task 4198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d4087853-e3fd-4caf-bba2-f60bcc1a8537_isrForCrosstalkSources_855414_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d4087853-e3fd-4caf-bba2-f60bcc1a8537_isrForCrosstalkSources_855414_23.stderr parsl.dataflow.dflow DEBUG: Task 4199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4199 has memoization hash 7ebcb2c66bbd39703fa4e825c555070b parsl.dataflow.memoization INFO: Task 4199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4199 try 0 launched on executor multi with executor id 4200 parsl.dataflow.dflow INFO: Standard output for task 4199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8680142e-17d6-4ec5-853e-5fe9272f74af_isrForCrosstalkSources_855454_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8680142e-17d6-4ec5-853e-5fe9272f74af_isrForCrosstalkSources_855454_54.stderr parsl.dataflow.dflow DEBUG: Task 4200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4200 has memoization hash 5ad202e7386a19faf1ac61c498f540cb parsl.dataflow.memoization INFO: Task 4200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4200 try 0 launched on executor multi with executor id 4201 parsl.dataflow.dflow INFO: Standard output for task 4200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/3a6d1326-b07c-45f9-a9ef-54b14e83bb23_isrForCrosstalkSources_855415_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/3a6d1326-b07c-45f9-a9ef-54b14e83bb23_isrForCrosstalkSources_855415_10.stderr parsl.dataflow.dflow DEBUG: Task 4201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4201 has memoization hash 097d8942503257d484a7b34be36371ba parsl.dataflow.memoization INFO: Task 4201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4201 try 0 launched on executor multi with executor id 4202 parsl.dataflow.dflow INFO: Standard output for task 4201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9348bb26-3850-421c-96fb-364e5234d9f7_isrForCrosstalkSources_855399_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9348bb26-3850-421c-96fb-364e5234d9f7_isrForCrosstalkSources_855399_15.stderr parsl.dataflow.dflow DEBUG: Task 4202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4202 has memoization hash 37d28a47cb2be03bcbddf272503ae9f3 parsl.dataflow.memoization INFO: Task 4202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4202 try 0 launched on executor multi with executor id 4203 parsl.dataflow.dflow INFO: Standard output for task 4202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/37c1d55a-1182-4296-936c-edcaa3275641_isrForCrosstalkSources_855406_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/37c1d55a-1182-4296-936c-edcaa3275641_isrForCrosstalkSources_855406_52.stderr parsl.dataflow.dflow DEBUG: Task 4203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4203 has memoization hash 33e5189863f6985a821958ee45136cb3 parsl.dataflow.memoization INFO: Task 4203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4203 try 0 launched on executor multi with executor id 4204 parsl.dataflow.dflow INFO: Standard output for task 4203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ecee7586-d847-4160-a244-8af8fe195fc8_isrForCrosstalkSources_855442_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ecee7586-d847-4160-a244-8af8fe195fc8_isrForCrosstalkSources_855442_45.stderr parsl.dataflow.dflow DEBUG: Task 4204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4204 has memoization hash ef51fb283dfe1dfafd846c884b297fc5 parsl.dataflow.memoization INFO: Task 4204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4204 try 0 launched on executor multi with executor id 4205 parsl.dataflow.dflow INFO: Standard output for task 4204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d383262-ba6b-452e-a5db-2d1e561f4193_isrForCrosstalkSources_855400_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d383262-ba6b-452e-a5db-2d1e561f4193_isrForCrosstalkSources_855400_35.stderr parsl.dataflow.dflow DEBUG: Task 4205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4205 has memoization hash b20429af0649b3cce84fd4cad88898a7 parsl.dataflow.memoization INFO: Task 4205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4205 try 0 launched on executor multi with executor id 4206 parsl.dataflow.dflow INFO: Standard output for task 4205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ba147929-c484-45b2-b4bf-063779ec3d4f_isrForCrosstalkSources_855405_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ba147929-c484-45b2-b4bf-063779ec3d4f_isrForCrosstalkSources_855405_14.stderr parsl.dataflow.dflow DEBUG: Task 4206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4206 has memoization hash fa9e0457714dde83605a8959465e6bba parsl.dataflow.memoization INFO: Task 4206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4206 try 0 launched on executor multi with executor id 4207 parsl.dataflow.dflow INFO: Standard output for task 4206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8bfe1cd4-0e97-4ad0-829d-5bfed62a1472_isrForCrosstalkSources_855397_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8bfe1cd4-0e97-4ad0-829d-5bfed62a1472_isrForCrosstalkSources_855397_26.stderr parsl.dataflow.dflow DEBUG: Task 4207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4207 has memoization hash 0258e8910276ed3447a05ffda13a58ff parsl.dataflow.memoization INFO: Task 4207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3d4dbf0d49c58287302d2326bf580b2 with result from task 565 parsl.dataflow.dflow INFO: Parsl task 4207 try 0 launched on executor multi with executor id 4208 parsl.dataflow.dflow INFO: Standard output for task 4207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/054940b6-48ff-4850-a47d-c45fec882897_isrForCrosstalkSources_855459_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/054940b6-48ff-4850-a47d-c45fec882897_isrForCrosstalkSources_855459_55.stderr parsl.dataflow.dflow DEBUG: Task 4208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4208 has memoization hash db1f2224d00a71336d02436c954d7020 parsl.dataflow.memoization INFO: Task 4208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/df0125b3-f877-4bfa-8df5-e62355942dfe_isrForCrosstalkSources_855386_49.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/df0125b3-f877-4bfa-8df5-e62355942dfe_isrForCrosstalkSources_855386_49.stderr parsl.dataflow.dflow INFO: Parsl task 4208 try 0 launched on executor multi with executor id 4209 parsl.dataflow.dflow INFO: Standard output for task 4208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/02f6dd88-d138-4d63-bcef-62e411218001_isrForCrosstalkSources_855393_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/02f6dd88-d138-4d63-bcef-62e411218001_isrForCrosstalkSources_855393_31.stderr parsl.dataflow.dflow DEBUG: Task 4209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4209 has memoization hash db795f9d6045217cdea8dcfc82dd24ee parsl.dataflow.memoization INFO: Task 4209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4209 try 0 launched on executor multi with executor id 4210 parsl.dataflow.dflow INFO: Standard output for task 4209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2a7048c2-89fb-4855-ad89-3ef5d3b6286f_isrForCrosstalkSources_855393_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2a7048c2-89fb-4855-ad89-3ef5d3b6286f_isrForCrosstalkSources_855393_25.stderr parsl.dataflow.dflow DEBUG: Task 4210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4210 has memoization hash ed0abba67a08c551b919d1cea1540179 parsl.dataflow.memoization INFO: Task 4210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4210 try 0 launched on executor multi with executor id 4211 parsl.dataflow.dflow INFO: Standard output for task 4210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/460df7b8-7b08-4911-bd0b-3e4e0e472295_isrForCrosstalkSources_855438_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/460df7b8-7b08-4911-bd0b-3e4e0e472295_isrForCrosstalkSources_855438_1.stderr parsl.dataflow.dflow DEBUG: Task 4211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4211 has memoization hash 2a1a117c0025a98f7012b0b0d4937e38 parsl.dataflow.memoization INFO: Task 4211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12bc0ac89b16e8e17ef40e9e0affe416 with result from task 568 parsl.dataflow.dflow INFO: Parsl task 4211 try 0 launched on executor multi with executor id 4212 parsl.dataflow.dflow INFO: Standard output for task 4211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f4bdb279-4a5a-4e7a-bd54-368fc063f722_isrForCrosstalkSources_855459_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f4bdb279-4a5a-4e7a-bd54-368fc063f722_isrForCrosstalkSources_855459_61.stderr parsl.dataflow.dflow DEBUG: Task 4212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Task 4212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9305cdcb-c43f-45c2-b23d-1d995b015fdb_isrForCrosstalkSources_855434_57.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9305cdcb-c43f-45c2-b23d-1d995b015fdb_isrForCrosstalkSources_855434_57.stderr parsl.dataflow.dflow DEBUG: Task 4212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4212 has memoization hash 9dbcc22d2d367e852cc6669a2454fa88 parsl.dataflow.memoization INFO: Task 4212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4212 try 0 launched on executor multi with executor id 4213 parsl.dataflow.dflow INFO: Standard output for task 4212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8d4e034e-d8f7-414c-b457-ba0be886e0b2_isrForCrosstalkSources_855410_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8d4e034e-d8f7-414c-b457-ba0be886e0b2_isrForCrosstalkSources_855410_9.stderr parsl.dataflow.dflow DEBUG: Task 4213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4213 has memoization hash d88c8ed67bf00317780233ef988709be parsl.dataflow.memoization INFO: Task 4213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4213 try 0 launched on executor multi with executor id 4214 parsl.dataflow.dflow INFO: Standard output for task 4213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bf13f712-c331-4580-a0db-1cc18b91bf0f_isrForCrosstalkSources_855442_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bf13f712-c331-4580-a0db-1cc18b91bf0f_isrForCrosstalkSources_855442_29.stderr parsl.dataflow.dflow DEBUG: Task 4214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4214 has memoization hash 3d72415f973f1f8565ca4a984d739bd1 parsl.dataflow.memoization INFO: Task 4214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4214 try 0 launched on executor multi with executor id 4215 parsl.dataflow.dflow INFO: Standard output for task 4214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/714e3474-f843-41ec-8faa-0ed5cc70dbc4_isrForCrosstalkSources_855432_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/714e3474-f843-41ec-8faa-0ed5cc70dbc4_isrForCrosstalkSources_855432_61.stderr parsl.dataflow.dflow DEBUG: Task 4215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4215 has memoization hash 0b31daf319f1848fc5e789d01208047e parsl.dataflow.memoization INFO: Task 4215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:47 proc_lsst.multi:170 [INFO] [multi] job_id 16964186 proc_lsst.multi INFO: [multi] job_id 16964186 2024-03-09 14:51:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-09 14:51:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964186 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964186 parsl.executors.status_handling DEBUG: Launched block 160 on executor multi with job ID 16964186 parsl.executors.status_handling INFO: Allocated block ID 161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-161 2024-03-09 14:51:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4215 try 0 launched on executor multi with executor id 4216 parsl.dataflow.dflow INFO: Standard output for task 4215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/724e32b7-9195-48f7-a7c6-4b4318242370_isrForCrosstalkSources_855388_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/724e32b7-9195-48f7-a7c6-4b4318242370_isrForCrosstalkSources_855388_55.stderr parsl.dataflow.dflow DEBUG: Task 4216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4216 has memoization hash e3001a92778606409a0c8d6c72311fc9 parsl.dataflow.memoization INFO: Task 4216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4216 try 0 launched on executor multi with executor id 4217 parsl.dataflow.dflow INFO: Standard output for task 4216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/169226f0-abe0-44b1-8bfb-a785c8de45bd_isrForCrosstalkSources_855446_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/169226f0-abe0-44b1-8bfb-a785c8de45bd_isrForCrosstalkSources_855446_35.stderr parsl.dataflow.dflow DEBUG: Task 4217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4217 has memoization hash 0d4f41fa0d8e6f39a779d89d92dac9d4 parsl.dataflow.memoization INFO: Task 4217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20777e4e79e756a06602f5deca6f61b0 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/122f7cc4-b143-4cfd-9c02-a2120bd92514_isrForCrosstalkSources_855412_33.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/122f7cc4-b143-4cfd-9c02-a2120bd92514_isrForCrosstalkSources_855412_33.stderr parsl.dataflow.dflow INFO: Parsl task 4217 try 0 launched on executor multi with executor id 4218 parsl.dataflow.dflow INFO: Standard output for task 4217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fce061c7-de25-44d4-8d6b-23e76b19c46c_isrForCrosstalkSources_855385_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fce061c7-de25-44d4-8d6b-23e76b19c46c_isrForCrosstalkSources_855385_47.stderr parsl.dataflow.dflow DEBUG: Task 4218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4218 has memoization hash d953c3165e7b50f2ef242ae3755ea879 parsl.dataflow.memoization INFO: Task 4218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4218 try 0 launched on executor multi with executor id 4219 parsl.dataflow.dflow INFO: Standard output for task 4218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3eb043b7-890a-405c-985f-8a68641f6862_isrForCrosstalkSources_855448_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3eb043b7-890a-405c-985f-8a68641f6862_isrForCrosstalkSources_855448_1.stderr parsl.dataflow.dflow DEBUG: Task 4219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4219 has memoization hash e003542f513477a92b90061c2e9fbfb4 parsl.dataflow.memoization INFO: Task 4219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50b0a263603aadc357a5673c13ab4afa with result from task 563 parsl.dataflow.dflow INFO: Parsl task 4219 try 0 launched on executor multi with executor id 4220 parsl.dataflow.dflow INFO: Standard output for task 4219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/369c815d-5537-494f-9736-3ab320aaa0d1_isrForCrosstalkSources_855449_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/369c815d-5537-494f-9736-3ab320aaa0d1_isrForCrosstalkSources_855449_35.stderr parsl.dataflow.dflow DEBUG: Task 4220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4220 has memoization hash d6bed900eedd2c761798a9f104a1d813 parsl.dataflow.memoization INFO: Task 4220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea343de0-be58-42ab-8e0a-61bd9c56785c_isrForCrosstalkSources_855382_61.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea343de0-be58-42ab-8e0a-61bd9c56785c_isrForCrosstalkSources_855382_61.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41c5e64a5ad12dd02a65f2bdca77eafa with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/2f4a6e06-f0af-4710-bd0f-554c7df5d493_isrForCrosstalkSources_855407_46.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/2f4a6e06-f0af-4710-bd0f-554c7df5d493_isrForCrosstalkSources_855407_46.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1173b033d93f412a514333a0353701a3 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/77f7a647-198d-47b1-bc77-27f4feab504c_isrForCrosstalkSources_855414_28.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/77f7a647-198d-47b1-bc77-27f4feab504c_isrForCrosstalkSources_855414_28.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13536d89379ec5d6c9717b86f0193cea with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0b294003-9f7e-44da-b9c8-449641df6628_isrForCrosstalkSources_855420_40.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0b294003-9f7e-44da-b9c8-449641df6628_isrForCrosstalkSources_855420_40.stderr parsl.dataflow.dflow INFO: Parsl task 4220 try 0 launched on executor multi with executor id 4221 parsl.dataflow.dflow INFO: Standard output for task 4220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3677a386-d1ce-4ff9-9805-e3bdd473eb87_isrForCrosstalkSources_855385_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3677a386-d1ce-4ff9-9805-e3bdd473eb87_isrForCrosstalkSources_855385_6.stderr parsl.dataflow.dflow DEBUG: Task 4221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4221 has memoization hash 009b68f0b7e5ebcd72e0290fcc3a8e6e parsl.dataflow.memoization INFO: Task 4221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4221 try 0 launched on executor multi with executor id 4222 parsl.dataflow.dflow INFO: Standard output for task 4221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/a3ed68a1-a281-4616-b1ab-115f1676ba83_isrForCrosstalkSources_855430_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/a3ed68a1-a281-4616-b1ab-115f1676ba83_isrForCrosstalkSources_855430_17.stderr parsl.dataflow.dflow DEBUG: Task 4222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4222 has memoization hash 1885695f7f360c072e1bd780e4196526 parsl.dataflow.memoization INFO: Task 4222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4222 try 0 launched on executor multi with executor id 4223 parsl.dataflow.dflow INFO: Standard output for task 4222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30745fe3-957e-46d4-9c79-7301e7c034b6_isrForCrosstalkSources_855436_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30745fe3-957e-46d4-9c79-7301e7c034b6_isrForCrosstalkSources_855436_58.stderr parsl.dataflow.dflow DEBUG: Task 4223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4223 has memoization hash edc164ca17d3bdff07b17da64966a66b parsl.dataflow.memoization INFO: Task 4223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4223 try 0 launched on executor multi with executor id 4224 parsl.dataflow.dflow INFO: Standard output for task 4223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/8cc7944a-e517-49a4-9c76-006994169e22_isrForCrosstalkSources_855394_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/8cc7944a-e517-49a4-9c76-006994169e22_isrForCrosstalkSources_855394_6.stderr parsl.dataflow.dflow DEBUG: Task 4224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4224 has memoization hash 1e4da5cea837629d95e52c0dabf9bcfa parsl.dataflow.memoization INFO: Task 4224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4224 try 0 launched on executor multi with executor id 4225 parsl.dataflow.dflow INFO: Standard output for task 4224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d7fe212-0e97-42bf-a4b0-3d80349a5569_isrForCrosstalkSources_855439_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d7fe212-0e97-42bf-a4b0-3d80349a5569_isrForCrosstalkSources_855439_7.stderr parsl.dataflow.dflow DEBUG: Task 4225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4225 has memoization hash 6657ddb7f3a20afaa7928639e08ac925 parsl.dataflow.memoization INFO: Task 4225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4225 try 0 launched on executor multi with executor id 4226 parsl.dataflow.dflow INFO: Standard output for task 4225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1eba9c1e-2367-4ffb-80c2-c4f0f2be6b9c_isrForCrosstalkSources_855431_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1eba9c1e-2367-4ffb-80c2-c4f0f2be6b9c_isrForCrosstalkSources_855431_3.stderr parsl.dataflow.dflow DEBUG: Task 4226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4226 has memoization hash c8233cec7b040000bda175b7b672ce9c parsl.dataflow.memoization INFO: Task 4226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4226 try 0 launched on executor multi with executor id 4227 parsl.dataflow.dflow INFO: Standard output for task 4226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9cac7f3d-4def-4ee5-96f4-97a7ed91742d_isrForCrosstalkSources_855380_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9cac7f3d-4def-4ee5-96f4-97a7ed91742d_isrForCrosstalkSources_855380_54.stderr parsl.dataflow.dflow DEBUG: Task 4227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4227 has memoization hash 24dbe3d69888e99b93994cdb02ff1dbd parsl.dataflow.memoization INFO: Task 4227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4227 try 0 launched on executor multi with executor id 4228 parsl.dataflow.dflow INFO: Standard output for task 4227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/495d90a6-889f-4904-8382-1bc1e49d2355_isrForCrosstalkSources_855379_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/495d90a6-889f-4904-8382-1bc1e49d2355_isrForCrosstalkSources_855379_45.stderr parsl.dataflow.dflow DEBUG: Task 4228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4228 has memoization hash fa438be7bc2de9db084d07cc95f602ac parsl.dataflow.memoization INFO: Task 4228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4228 try 0 launched on executor multi with executor id 4229 parsl.dataflow.dflow INFO: Standard output for task 4228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b6333fa2-2efc-44d9-9058-c139c82ec492_isrForCrosstalkSources_855433_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b6333fa2-2efc-44d9-9058-c139c82ec492_isrForCrosstalkSources_855433_28.stderr parsl.dataflow.dflow DEBUG: Task 4229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4229 has memoization hash d9933461ce0b74a7fc30525d22b823f1 parsl.dataflow.memoization INFO: Task 4229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4229 try 0 launched on executor multi with executor id 4230 parsl.dataflow.dflow INFO: Standard output for task 4229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/59809dc9-c3e0-4558-abf2-8517243640b3_isrForCrosstalkSources_855435_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/59809dc9-c3e0-4558-abf2-8517243640b3_isrForCrosstalkSources_855435_53.stderr parsl.dataflow.dflow DEBUG: Task 4230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4230 has memoization hash df82a9a9db6bb4f9732a662190084f9f parsl.dataflow.memoization INFO: Task 4230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4230 try 0 launched on executor multi with executor id 4231 parsl.dataflow.dflow INFO: Standard output for task 4230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/12744a18-8301-4664-bbe0-a46a40fd3868_isrForCrosstalkSources_855385_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/12744a18-8301-4664-bbe0-a46a40fd3868_isrForCrosstalkSources_855385_40.stderr parsl.dataflow.dflow DEBUG: Task 4231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4231 has memoization hash b02872a0eb610e8fd7afc7c45f6dfffa parsl.dataflow.memoization INFO: Task 4231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4231 try 0 launched on executor multi with executor id 4232 parsl.dataflow.dflow INFO: Standard output for task 4231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a582f427-c275-4e94-8903-dbc0be2576c8_isrForCrosstalkSources_855386_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a582f427-c275-4e94-8903-dbc0be2576c8_isrForCrosstalkSources_855386_24.stderr parsl.dataflow.dflow DEBUG: Task 4232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4232 has memoization hash dc118af96300aa0a69660ae69445448e parsl.dataflow.memoization INFO: Task 4232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4232 try 0 launched on executor multi with executor id 4233 parsl.dataflow.dflow INFO: Standard output for task 4232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ba2d5c5a-b0d9-4d82-ae4e-7bb7d4fa7f53_isrForCrosstalkSources_855400_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ba2d5c5a-b0d9-4d82-ae4e-7bb7d4fa7f53_isrForCrosstalkSources_855400_42.stderr parsl.dataflow.dflow DEBUG: Task 4233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4233 has memoization hash fef50db8180ed5a3e314bd9b0b9cb4b1 parsl.dataflow.memoization INFO: Task 4233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4233 try 0 launched on executor multi with executor id 4234 parsl.dataflow.dflow INFO: Standard output for task 4233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/34ec0bb8-3b32-457f-8422-b70dd031bf92_isrForCrosstalkSources_855416_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/34ec0bb8-3b32-457f-8422-b70dd031bf92_isrForCrosstalkSources_855416_41.stderr parsl.dataflow.dflow DEBUG: Task 4234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4234 has memoization hash d03df7eea69f3e199e7795fe1e276d6c parsl.dataflow.memoization INFO: Task 4234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4234 try 0 launched on executor multi with executor id 4235 parsl.dataflow.dflow INFO: Standard output for task 4234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/bf76530e-46a5-4420-9746-10ca5e5dca6a_isrForCrosstalkSources_855435_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/bf76530e-46a5-4420-9746-10ca5e5dca6a_isrForCrosstalkSources_855435_51.stderr parsl.dataflow.dflow DEBUG: Task 4235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4235 has memoization hash c4f16e489c8b926579bd9711d581e389 parsl.dataflow.memoization INFO: Task 4235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4235 try 0 launched on executor multi with executor id 4236 parsl.dataflow.dflow INFO: Standard output for task 4235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4769bf72-6e44-4dea-a79e-ea56f2ae7c38_isrForCrosstalkSources_855434_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4769bf72-6e44-4dea-a79e-ea56f2ae7c38_isrForCrosstalkSources_855434_7.stderr parsl.dataflow.dflow DEBUG: Task 4236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4236 has memoization hash 65639e1414363aca36a9e9c8472d2c41 parsl.dataflow.memoization INFO: Task 4236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4236 try 0 launched on executor multi with executor id 4237 parsl.dataflow.dflow INFO: Standard output for task 4236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/042f088d-014e-4784-93dd-db56ffc2af7b_isrForCrosstalkSources_855442_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/042f088d-014e-4784-93dd-db56ffc2af7b_isrForCrosstalkSources_855442_27.stderr parsl.dataflow.dflow DEBUG: Task 4237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4237 has memoization hash 7336aeb22891015279f4e68af00090a3 parsl.dataflow.memoization INFO: Task 4237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4237 try 0 launched on executor multi with executor id 4238 parsl.dataflow.dflow INFO: Standard output for task 4237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9e6132f6-076b-422e-8155-b1061863da06_isrForCrosstalkSources_855432_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9e6132f6-076b-422e-8155-b1061863da06_isrForCrosstalkSources_855432_45.stderr parsl.dataflow.dflow DEBUG: Task 4238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4238 has memoization hash 094e9c5f9ec078f2e039906496668bb0 parsl.dataflow.memoization INFO: Task 4238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4238 try 0 launched on executor multi with executor id 4239 parsl.dataflow.dflow INFO: Standard output for task 4238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/40ed2dca-8f20-4152-a44a-b46cd965e0fa_isrForCrosstalkSources_855406_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/40ed2dca-8f20-4152-a44a-b46cd965e0fa_isrForCrosstalkSources_855406_17.stderr parsl.dataflow.dflow DEBUG: Task 4239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4239 has memoization hash ed69b04c49a2a0d9e1731a7a2d086537 parsl.dataflow.memoization INFO: Task 4239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4239 try 0 launched on executor multi with executor id 4240 parsl.dataflow.dflow INFO: Standard output for task 4239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c66ca99f-56fa-461b-8c6e-8f867db9ba96_isrForCrosstalkSources_855419_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c66ca99f-56fa-461b-8c6e-8f867db9ba96_isrForCrosstalkSources_855419_4.stderr parsl.dataflow.dflow DEBUG: Task 4240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4240 has memoization hash eb0309392c9b818b16cd0ce7ba1c3fbf parsl.dataflow.memoization INFO: Task 4240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:47 proc_lsst.multi:170 [INFO] [multi] job_id 16964187 proc_lsst.multi INFO: [multi] job_id 16964187 2024-03-09 14:51:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-09 14:51:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964187 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964187 parsl.executors.status_handling DEBUG: Launched block 161 on executor multi with job ID 16964187 parsl.executors.status_handling INFO: Allocated block ID 162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-162 2024-03-09 14:51:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-09 14:51:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4240 try 0 launched on executor multi with executor id 4241 parsl.dataflow.dflow INFO: Standard output for task 4240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/176a52eb-25d5-4120-b611-5e4b0cda629b_isrForCrosstalkSources_855411_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/176a52eb-25d5-4120-b611-5e4b0cda629b_isrForCrosstalkSources_855411_5.stderr parsl.dataflow.dflow DEBUG: Task 4241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4241 has memoization hash 857420df1793cf93c3239e276c3e6755 parsl.dataflow.memoization INFO: Task 4241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4241 try 0 launched on executor multi with executor id 4242 parsl.dataflow.dflow INFO: Standard output for task 4241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/86198d71-f86b-48cb-b2cd-de604c9e252a_isrForCrosstalkSources_855399_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/86198d71-f86b-48cb-b2cd-de604c9e252a_isrForCrosstalkSources_855399_21.stderr parsl.dataflow.dflow DEBUG: Task 4242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4242 has memoization hash c896ad7eb86f77c3836a528cd4708823 parsl.dataflow.memoization INFO: Task 4242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4242 try 0 launched on executor multi with executor id 4243 parsl.dataflow.dflow INFO: Standard output for task 4242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fee46111-90bd-4f0f-b0bc-2e2bc73b72f8_isrForCrosstalkSources_855394_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fee46111-90bd-4f0f-b0bc-2e2bc73b72f8_isrForCrosstalkSources_855394_45.stderr parsl.dataflow.dflow DEBUG: Task 4243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4243 has memoization hash c7721c021b4ca5e83c0695387cb755f8 parsl.dataflow.memoization INFO: Task 4243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4243 try 0 launched on executor multi with executor id 4244 parsl.dataflow.dflow INFO: Standard output for task 4243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a1f2528-db5d-4431-93e2-06cd386d4a0f_isrForCrosstalkSources_855410_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a1f2528-db5d-4431-93e2-06cd386d4a0f_isrForCrosstalkSources_855410_41.stderr parsl.dataflow.dflow DEBUG: Task 4244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4244 has memoization hash 4ef01d190b0d6dca1f9b529552c8bfef parsl.dataflow.memoization INFO: Task 4244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4244 try 0 launched on executor multi with executor id 4245 parsl.dataflow.dflow INFO: Standard output for task 4244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b5c8932e-35b8-4e4b-9e1a-345584dc7935_isrForCrosstalkSources_855435_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b5c8932e-35b8-4e4b-9e1a-345584dc7935_isrForCrosstalkSources_855435_30.stderr parsl.dataflow.dflow DEBUG: Task 4245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4245 has memoization hash 827e69b61b20462734252eaf9024cbc8 parsl.dataflow.memoization INFO: Task 4245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5c1bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4245 try 0 launched on executor multi with executor id 4246 parsl.dataflow.dflow INFO: Standard output for task 4245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d66aef15-cce8-4c5f-a80c-c9b8c8e308af_isrForCrosstalkSources_855414_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d66aef15-cce8-4c5f-a80c-c9b8c8e308af_isrForCrosstalkSources_855414_43.stderr parsl.dataflow.dflow DEBUG: Task 4246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4246 has memoization hash 422babeeccd11f5cc8e3fc177c90157f parsl.dataflow.memoization INFO: Task 4246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4246 try 0 launched on executor multi with executor id 4247 parsl.dataflow.dflow INFO: Standard output for task 4246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/56e0d3a6-7bad-43a8-be5d-965479094e94_isrForCrosstalkSources_855379_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/56e0d3a6-7bad-43a8-be5d-965479094e94_isrForCrosstalkSources_855379_7.stderr parsl.dataflow.dflow DEBUG: Task 4247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4247 has memoization hash 67d02025ed26b1410dce0ac655f0bfd9 parsl.dataflow.memoization INFO: Task 4247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4247 try 0 launched on executor multi with executor id 4248 parsl.dataflow.dflow INFO: Standard output for task 4247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a37c415c-5407-449f-bd5c-5376a3f8c6d0_isrForCrosstalkSources_855404_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a37c415c-5407-449f-bd5c-5376a3f8c6d0_isrForCrosstalkSources_855404_58.stderr parsl.dataflow.dflow DEBUG: Task 4248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4248 has memoization hash 88c7068fd2fac22ef24d553e9baa85a4 parsl.dataflow.memoization INFO: Task 4248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4248 try 0 launched on executor multi with executor id 4249 parsl.dataflow.dflow INFO: Standard output for task 4248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032a29d0-ad81-48de-9470-92a374822501_isrForCrosstalkSources_855414_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032a29d0-ad81-48de-9470-92a374822501_isrForCrosstalkSources_855414_9.stderr parsl.dataflow.dflow DEBUG: Task 4249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4249 has memoization hash 40399a961d03d26c9862a04c4c0faf05 parsl.dataflow.memoization INFO: Task 4249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4249 try 0 launched on executor multi with executor id 4250 parsl.dataflow.dflow INFO: Standard output for task 4249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2604275-4513-41b1-b236-0f8ac4965e43_isrForCrosstalkSources_855385_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2604275-4513-41b1-b236-0f8ac4965e43_isrForCrosstalkSources_855385_57.stderr parsl.dataflow.dflow DEBUG: Task 4250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4250 has memoization hash c4635d3ceeb5788216ae3f7ccccc3311 parsl.dataflow.memoization INFO: Task 4250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4250 try 0 launched on executor multi with executor id 4251 parsl.dataflow.dflow INFO: Standard output for task 4250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8200eb69-bb52-4633-9672-033ee72c3a46_isrForCrosstalkSources_855399_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8200eb69-bb52-4633-9672-033ee72c3a46_isrForCrosstalkSources_855399_18.stderr parsl.dataflow.dflow DEBUG: Task 4251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4251 has memoization hash d39de8e1d3ebb333813ee69bafe97cba parsl.dataflow.memoization INFO: Task 4251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4251 try 0 launched on executor multi with executor id 4252 parsl.dataflow.dflow INFO: Standard output for task 4251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/58a56ceb-6424-4908-873d-197ce04f23af_isrForCrosstalkSources_855443_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/58a56ceb-6424-4908-873d-197ce04f23af_isrForCrosstalkSources_855443_43.stderr parsl.dataflow.dflow DEBUG: Task 4252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4252 has memoization hash 2f321f6a5943ac338a8454ab3d075f56 parsl.dataflow.memoization INFO: Task 4252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec589415e04ce54fb23e4f9f71d75c7 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f6f0be86-f5a7-4599-a682-a67e2d566152_isrForCrosstalkSources_855443_34.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f6f0be86-f5a7-4599-a682-a67e2d566152_isrForCrosstalkSources_855443_34.stderr parsl.dataflow.dflow INFO: Parsl task 4252 try 0 launched on executor multi with executor id 4253 parsl.dataflow.dflow INFO: Standard output for task 4252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72c75e58-9a3a-4aa7-8188-708ce7c516b4_isrForCrosstalkSources_855417_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72c75e58-9a3a-4aa7-8188-708ce7c516b4_isrForCrosstalkSources_855417_56.stderr parsl.dataflow.dflow DEBUG: Task 4253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4253 has memoization hash e02a70cf44968887f744e76bd3b7d7bd parsl.dataflow.memoization INFO: Task 4253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4253 try 0 launched on executor multi with executor id 4254 parsl.dataflow.dflow INFO: Standard output for task 4253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/83cb7257-9b4d-49a0-9306-a364ef4e1ee7_isrForCrosstalkSources_855437_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/83cb7257-9b4d-49a0-9306-a364ef4e1ee7_isrForCrosstalkSources_855437_34.stderr parsl.dataflow.dflow DEBUG: Task 4254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4254 has memoization hash f7edffde9cb9eeefc0ed2f39bef8a935 parsl.dataflow.memoization INFO: Task 4254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4254 try 0 launched on executor multi with executor id 4255 parsl.dataflow.dflow INFO: Standard output for task 4254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9bf5a57d-5bdd-4e9a-9558-258c84ce7715_isrForCrosstalkSources_855397_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9bf5a57d-5bdd-4e9a-9558-258c84ce7715_isrForCrosstalkSources_855397_47.stderr parsl.dataflow.dflow DEBUG: Task 4255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4255 has memoization hash b66dcf745aa496b7a2fa8763eeb24ba9 parsl.dataflow.memoization INFO: Task 4255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4255 try 0 launched on executor multi with executor id 4256 parsl.dataflow.dflow INFO: Standard output for task 4255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/db97ba76-012d-482f-91e9-7efbd8202a60_isrForCrosstalkSources_855452_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/db97ba76-012d-482f-91e9-7efbd8202a60_isrForCrosstalkSources_855452_36.stderr parsl.dataflow.dflow DEBUG: Task 4256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4256 has memoization hash db1a739d4a5cf246c2bbeb8879643457 parsl.dataflow.memoization INFO: Task 4256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4256 try 0 launched on executor multi with executor id 4257 parsl.dataflow.dflow INFO: Standard output for task 4256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/93f6f914-d439-4102-b52a-7be9d549e3db_isrForCrosstalkSources_855412_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/93f6f914-d439-4102-b52a-7be9d549e3db_isrForCrosstalkSources_855412_39.stderr parsl.dataflow.dflow DEBUG: Task 4257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4257 has memoization hash 1e8edec3a5d907ac7c6fc67f90a84392 parsl.dataflow.memoization INFO: Task 4257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4257 try 0 launched on executor multi with executor id 4258 parsl.dataflow.dflow INFO: Standard output for task 4257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/71771860-4808-4042-abf9-815b1203e9e9_isrForCrosstalkSources_855453_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/71771860-4808-4042-abf9-815b1203e9e9_isrForCrosstalkSources_855453_8.stderr parsl.dataflow.dflow DEBUG: Task 4258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4258 has memoization hash 4740c7a8bf98af4a8925af732fdd9190 parsl.dataflow.memoization INFO: Task 4258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4258 try 0 launched on executor multi with executor id 4259 parsl.dataflow.dflow INFO: Standard output for task 4258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1ec98d01-bf9b-40bd-a3e8-144b2fba6626_isrForCrosstalkSources_855446_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1ec98d01-bf9b-40bd-a3e8-144b2fba6626_isrForCrosstalkSources_855446_25.stderr parsl.dataflow.dflow DEBUG: Task 4259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4259 has memoization hash 00dcf7ca3b339b22fa50319f65a513d9 parsl.dataflow.memoization INFO: Task 4259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4259 try 0 launched on executor multi with executor id 4260 parsl.dataflow.dflow INFO: Standard output for task 4259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/805f8f68-cc2a-4970-ae48-daff7ff4a079_isrForCrosstalkSources_855384_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/805f8f68-cc2a-4970-ae48-daff7ff4a079_isrForCrosstalkSources_855384_55.stderr parsl.dataflow.dflow DEBUG: Task 4260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4260 has memoization hash f283a7c9e96ec8b2dc6b858131f8ad4b parsl.dataflow.memoization INFO: Task 4260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4260 try 0 launched on executor multi with executor id 4261 parsl.dataflow.dflow INFO: Standard output for task 4260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/56b535f2-785b-461f-b9e8-1e06f77e3b0f_isrForCrosstalkSources_855455_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/56b535f2-785b-461f-b9e8-1e06f77e3b0f_isrForCrosstalkSources_855455_15.stderr parsl.dataflow.dflow DEBUG: Task 4261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4261 has memoization hash 277ee15874c80b7f179f3945bfc842ac parsl.dataflow.memoization INFO: Task 4261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4261 try 0 launched on executor multi with executor id 4262 parsl.dataflow.dflow INFO: Standard output for task 4261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5fa1d87a-6169-49f9-b8ef-55c8cbd19c41_isrForCrosstalkSources_855416_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5fa1d87a-6169-49f9-b8ef-55c8cbd19c41_isrForCrosstalkSources_855416_31.stderr parsl.dataflow.dflow DEBUG: Task 4262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4262 has memoization hash 879a06e7720b2115facfe4e01f4b6b5a parsl.dataflow.memoization INFO: Task 4262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4262 try 0 launched on executor multi with executor id 4263 parsl.dataflow.dflow INFO: Standard output for task 4262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fd4bfa10-71ab-4519-b962-b68a4641431a_isrForCrosstalkSources_855409_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fd4bfa10-71ab-4519-b962-b68a4641431a_isrForCrosstalkSources_855409_14.stderr parsl.dataflow.dflow DEBUG: Task 4263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4263 has memoization hash ad2c439adf0fc3ea27aa60b401002214 parsl.dataflow.memoization INFO: Task 4263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a600e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4263 try 0 launched on executor multi with executor id 4264 parsl.dataflow.dflow INFO: Standard output for task 4263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c51382cb-0adc-47d7-995a-2cbe18822d3d_isrForCrosstalkSources_855395_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c51382cb-0adc-47d7-995a-2cbe18822d3d_isrForCrosstalkSources_855395_60.stderr parsl.dataflow.dflow DEBUG: Task 4264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4264 has memoization hash 9851635817bc2a1fae4f9f442674dcef parsl.dataflow.memoization INFO: Task 4264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4264 try 0 launched on executor multi with executor id 4265 parsl.dataflow.dflow INFO: Standard output for task 4264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b9da4ce2-e1d3-4acc-a981-5499f10115a0_isrForCrosstalkSources_855391_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b9da4ce2-e1d3-4acc-a981-5499f10115a0_isrForCrosstalkSources_855391_35.stderr parsl.dataflow.dflow DEBUG: Task 4265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4265 has memoization hash e056e35d69ac526d2142d2af0c215c97 parsl.dataflow.memoization INFO: Task 4265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4265 try 0 launched on executor multi with executor id 4266 parsl.dataflow.dflow INFO: Standard output for task 4265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7876e18-3e28-4dbc-8fa2-62b2d7c2fe19_isrForCrosstalkSources_855382_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7876e18-3e28-4dbc-8fa2-62b2d7c2fe19_isrForCrosstalkSources_855382_49.stderr parsl.dataflow.dflow DEBUG: Task 4266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4266 has memoization hash d14263569a823548a4c8484e9a5b9755 parsl.dataflow.memoization INFO: Task 4266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a602c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4266 try 0 launched on executor multi with executor id 4267 parsl.dataflow.dflow INFO: Standard output for task 4266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3c07092c-b8ed-4908-8541-80e7c7dda3ec_isrForCrosstalkSources_855442_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3c07092c-b8ed-4908-8541-80e7c7dda3ec_isrForCrosstalkSources_855442_49.stderr parsl.dataflow.dflow DEBUG: Task 4267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4267 has memoization hash 37540f5e42757eaa526a93c661f52cde parsl.dataflow.memoization INFO: Task 4267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4267 try 0 launched on executor multi with executor id 4268 parsl.dataflow.dflow INFO: Standard output for task 4267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/71e7b0c4-1fe8-425f-b453-2e0fbecb5930_isrForCrosstalkSources_855441_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/71e7b0c4-1fe8-425f-b453-2e0fbecb5930_isrForCrosstalkSources_855441_39.stderr parsl.dataflow.dflow DEBUG: Task 4268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4268 has memoization hash 8048e2a8d41ffda64632132597e8ca28 parsl.dataflow.memoization INFO: Task 4268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4268 try 0 launched on executor multi with executor id 4269 parsl.dataflow.dflow INFO: Standard output for task 4268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f3e2bf66-f251-4c1a-b796-4a0ec5570b40_isrForCrosstalkSources_855388_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f3e2bf66-f251-4c1a-b796-4a0ec5570b40_isrForCrosstalkSources_855388_51.stderr parsl.dataflow.dflow DEBUG: Task 4269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4269 has memoization hash f3a254b575c57b0504ac9bc0746876e7 parsl.dataflow.memoization INFO: Task 4269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4269 try 0 launched on executor multi with executor id 4270 parsl.dataflow.dflow INFO: Standard output for task 4269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d8229bc-8dae-4087-8b71-d14fb7fae80d_isrForCrosstalkSources_855380_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d8229bc-8dae-4087-8b71-d14fb7fae80d_isrForCrosstalkSources_855380_5.stderr parsl.dataflow.dflow DEBUG: Task 4270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4270 has memoization hash a5cd65bf5fa75c15f67f68d6296ec345 parsl.dataflow.memoization INFO: Task 4270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4270 try 0 launched on executor multi with executor id 4271 parsl.dataflow.dflow INFO: Standard output for task 4270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/41c6fb97-d1ca-48ff-86a3-ba04ee6f2454_isrForCrosstalkSources_855427_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/41c6fb97-d1ca-48ff-86a3-ba04ee6f2454_isrForCrosstalkSources_855427_62.stderr parsl.dataflow.dflow DEBUG: Task 4271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4271 has memoization hash 844d5528d3b475336f2350beb5387297 parsl.dataflow.memoization INFO: Task 4271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a605e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4271 try 0 launched on executor multi with executor id 4272 parsl.dataflow.dflow INFO: Standard output for task 4271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/33e7c969-99f7-4349-988e-6178dc2d520e_isrForCrosstalkSources_855399_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/33e7c969-99f7-4349-988e-6178dc2d520e_isrForCrosstalkSources_855399_61.stderr parsl.dataflow.dflow DEBUG: Task 4272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4272 has memoization hash f870b456622f92bc862fee710f369a0f parsl.dataflow.memoization INFO: Task 4272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4272 try 0 launched on executor multi with executor id 4273 parsl.dataflow.dflow INFO: Standard output for task 4272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a08a5c21-10ab-4332-a2ee-73a9cc5152ae_isrForCrosstalkSources_855453_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a08a5c21-10ab-4332-a2ee-73a9cc5152ae_isrForCrosstalkSources_855453_2.stderr parsl.dataflow.dflow DEBUG: Task 4273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4273 has memoization hash 70c4eb512ef245d0f69ed995c40041b9 parsl.dataflow.memoization INFO: Task 4273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4273 try 0 launched on executor multi with executor id 4274 parsl.dataflow.dflow INFO: Standard output for task 4273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a34cc138-b2e7-44b2-a77a-d6b4bd2446af_isrForCrosstalkSources_855394_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a34cc138-b2e7-44b2-a77a-d6b4bd2446af_isrForCrosstalkSources_855394_36.stderr parsl.dataflow.dflow DEBUG: Task 4274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4274 has memoization hash a3ecf98874364639e444ece78eb60ede parsl.dataflow.memoization INFO: Task 4274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4274 try 0 launched on executor multi with executor id 4275 parsl.dataflow.dflow INFO: Standard output for task 4274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ab04e1a2-cfee-4e51-a782-7abc344179b6_isrForCrosstalkSources_855434_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ab04e1a2-cfee-4e51-a782-7abc344179b6_isrForCrosstalkSources_855434_36.stderr parsl.dataflow.dflow DEBUG: Task 4275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4275 has memoization hash 81477f2c8e05e2972ac3c0247c28ee75 parsl.dataflow.memoization INFO: Task 4275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4275 try 0 launched on executor multi with executor id 4276 parsl.dataflow.dflow INFO: Standard output for task 4275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/205700b1-c287-42ad-b66f-2f522dc7d6f9_isrForCrosstalkSources_855418_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/205700b1-c287-42ad-b66f-2f522dc7d6f9_isrForCrosstalkSources_855418_17.stderr parsl.dataflow.dflow DEBUG: Task 4276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4276 has memoization hash aa34766cedbea4da2b1d99e704c2bbc4 parsl.dataflow.memoization INFO: Task 4276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4276 try 0 launched on executor multi with executor id 4277 parsl.dataflow.dflow INFO: Standard output for task 4276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/00e91dfa-368c-4152-b80f-da31dccae1e5_isrForCrosstalkSources_855393_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/00e91dfa-368c-4152-b80f-da31dccae1e5_isrForCrosstalkSources_855393_14.stderr parsl.dataflow.dflow DEBUG: Task 4277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4277 has memoization hash c42dea64ebc2f9cf00dd649db3abb005 parsl.dataflow.memoization INFO: Task 4277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4277 try 0 launched on executor multi with executor id 4278 parsl.dataflow.dflow INFO: Standard output for task 4277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/50784c65-be87-47eb-96af-56624b822d11_isrForCrosstalkSources_855414_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/50784c65-be87-47eb-96af-56624b822d11_isrForCrosstalkSources_855414_44.stderr parsl.dataflow.dflow DEBUG: Task 4278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4278 has memoization hash f76141cf2ad62fd3077fd12120f5e572 parsl.dataflow.memoization INFO: Task 4278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a609a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4278 try 0 launched on executor multi with executor id 4279 parsl.dataflow.dflow INFO: Standard output for task 4278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0a35b54a-5793-4155-824d-ea948c2261d4_isrForCrosstalkSources_855456_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0a35b54a-5793-4155-824d-ea948c2261d4_isrForCrosstalkSources_855456_30.stderr parsl.dataflow.dflow DEBUG: Task 4279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4279 has memoization hash af45f5b882446ee8a0972c3991f09a15 parsl.dataflow.memoization INFO: Task 4279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4279 try 0 launched on executor multi with executor id 4280 parsl.dataflow.dflow INFO: Standard output for task 4279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/396af17f-6a54-4a30-ae02-d38e601a9b00_isrForCrosstalkSources_855386_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/396af17f-6a54-4a30-ae02-d38e601a9b00_isrForCrosstalkSources_855386_7.stderr parsl.dataflow.dflow DEBUG: Task 4280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4280 has memoization hash 9e61cadf4195565a9887b2ac25423053 parsl.dataflow.memoization INFO: Task 4280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4280 try 0 launched on executor multi with executor id 4281 parsl.dataflow.dflow INFO: Standard output for task 4280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6dc09dcc-b68d-46fa-8028-3dabcb00b4af_isrForCrosstalkSources_855417_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6dc09dcc-b68d-46fa-8028-3dabcb00b4af_isrForCrosstalkSources_855417_40.stderr parsl.dataflow.dflow DEBUG: Task 4281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4281 has memoization hash 05e7141e914155b7633e713be22a12ef parsl.dataflow.memoization INFO: Task 4281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4281 try 0 launched on executor multi with executor id 4282 parsl.dataflow.dflow INFO: Standard output for task 4281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a91fd2fd-42ff-4cba-86af-e85409eb06a3_isrForCrosstalkSources_855420_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a91fd2fd-42ff-4cba-86af-e85409eb06a3_isrForCrosstalkSources_855420_32.stderr parsl.dataflow.dflow DEBUG: Task 4282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4282 has memoization hash 335b4456111771bb50b1ca5aee0c3a03 parsl.dataflow.memoization INFO: Task 4282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4282 try 0 launched on executor multi with executor id 4283 parsl.dataflow.dflow INFO: Standard output for task 4282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0ba84ccb-1602-483b-a7d7-3b2c31092ac3_isrForCrosstalkSources_855408_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0ba84ccb-1602-483b-a7d7-3b2c31092ac3_isrForCrosstalkSources_855408_54.stderr parsl.dataflow.dflow DEBUG: Task 4283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4283 has memoization hash 169b1f9341eb5a1514bf9c40e27e1dac parsl.dataflow.memoization INFO: Task 4283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4283 try 0 launched on executor multi with executor id 4284 parsl.dataflow.dflow INFO: Standard output for task 4283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a652c21-eaca-40a0-87a0-185f80c1e7ab_isrForCrosstalkSources_855396_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a652c21-eaca-40a0-87a0-185f80c1e7ab_isrForCrosstalkSources_855396_60.stderr parsl.dataflow.dflow DEBUG: Task 4284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4284 has memoization hash 7cbad348ffb823684164694461b27979 parsl.dataflow.memoization INFO: Task 4284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:48 proc_lsst.multi:170 [INFO] [multi] job_id 16964188 proc_lsst.multi INFO: [multi] job_id 16964188 2024-03-09 14:51:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-09 14:51:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964188 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964188 parsl.executors.status_handling DEBUG: Launched block 162 on executor multi with job ID 16964188 parsl.executors.status_handling INFO: Allocated block ID 163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-163 2024-03-09 14:51:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4284 try 0 launched on executor multi with executor id 4285 parsl.dataflow.dflow INFO: Standard output for task 4284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/515fb82d-b9cd-4b08-8cf4-75429d5f945a_isrForCrosstalkSources_855390_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/515fb82d-b9cd-4b08-8cf4-75429d5f945a_isrForCrosstalkSources_855390_49.stderr parsl.dataflow.dflow DEBUG: Task 4285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4285 has memoization hash dbe99fc883178b4709d50c7978e392a6 parsl.dataflow.memoization INFO: Task 4285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4285 try 0 launched on executor multi with executor id 4286 parsl.dataflow.dflow INFO: Standard output for task 4285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/78bd9115-97a5-4eee-a7b2-d117cc10752a_isrForCrosstalkSources_855428_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/78bd9115-97a5-4eee-a7b2-d117cc10752a_isrForCrosstalkSources_855428_1.stderr parsl.dataflow.dflow DEBUG: Task 4286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4286 has memoization hash 9e4f24cbb7394c6c780879ccfc48a57e parsl.dataflow.memoization INFO: Task 4286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4286 try 0 launched on executor multi with executor id 4287 parsl.dataflow.dflow INFO: Standard output for task 4286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9a5ea6ae-e895-460a-9951-263a3985bc45_isrForCrosstalkSources_855392_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9a5ea6ae-e895-460a-9951-263a3985bc45_isrForCrosstalkSources_855392_58.stderr parsl.dataflow.dflow DEBUG: Task 4287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4287 has memoization hash 36970f888c7616b7fa8e189e7cd26843 parsl.dataflow.memoization INFO: Task 4287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4287 try 0 launched on executor multi with executor id 4288 parsl.dataflow.dflow INFO: Standard output for task 4287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fab6202b-21da-44a1-a265-6746e65ac78b_isrForCrosstalkSources_855379_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fab6202b-21da-44a1-a265-6746e65ac78b_isrForCrosstalkSources_855379_30.stderr parsl.dataflow.dflow DEBUG: Task 4288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4288 has memoization hash 6d6aecd97d51d5a76c19ddc230167d8e parsl.dataflow.memoization INFO: Task 4288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4288 try 0 launched on executor multi with executor id 4289 parsl.dataflow.dflow INFO: Standard output for task 4288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1554b1c4-0a1a-4401-93b6-e56378f94cd4_isrForCrosstalkSources_855387_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1554b1c4-0a1a-4401-93b6-e56378f94cd4_isrForCrosstalkSources_855387_62.stderr parsl.dataflow.dflow DEBUG: Task 4289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4289 has memoization hash 4c908d119c9eadb9fd64012389759a89 parsl.dataflow.memoization INFO: Task 4289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4289 try 0 launched on executor multi with executor id 4290 parsl.dataflow.dflow INFO: Standard output for task 4289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6411571b-b776-4ac4-ae59-b14b02c796ad_isrForCrosstalkSources_855436_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6411571b-b776-4ac4-ae59-b14b02c796ad_isrForCrosstalkSources_855436_23.stderr parsl.dataflow.dflow DEBUG: Task 4290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4290 has memoization hash 7d6669c2d3cf0145c03c48190216bdbb parsl.dataflow.memoization INFO: Task 4290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4290 try 0 launched on executor multi with executor id 4291 parsl.dataflow.dflow INFO: Standard output for task 4290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8c8e81f3-de92-45db-bedc-b8bdfbe0bdff_isrForCrosstalkSources_855435_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8c8e81f3-de92-45db-bedc-b8bdfbe0bdff_isrForCrosstalkSources_855435_38.stderr parsl.dataflow.dflow DEBUG: Task 4291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4291 has memoization hash 1fdf16f925f9357d52c5f2b09aaa98f1 parsl.dataflow.memoization INFO: Task 4291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4291 try 0 launched on executor multi with executor id 4292 parsl.dataflow.dflow INFO: Standard output for task 4291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ef82b9e6-7212-49e5-af7d-07ae9c4b9da3_isrForCrosstalkSources_855449_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ef82b9e6-7212-49e5-af7d-07ae9c4b9da3_isrForCrosstalkSources_855449_51.stderr parsl.dataflow.dflow DEBUG: Task 4292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4292 has memoization hash caee9518d533df6bdfd0b8110a210835 parsl.dataflow.memoization INFO: Task 4292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73da8aa755ea934601e4c177f7468a7 with result from task 168 parsl.dataflow.dflow INFO: Parsl task 4292 try 0 launched on executor multi with executor id 4293 parsl.dataflow.dflow INFO: Standard output for task 4292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3c869758-f155-42ce-ad0e-7a0c1f34568f_isrForCrosstalkSources_855438_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3c869758-f155-42ce-ad0e-7a0c1f34568f_isrForCrosstalkSources_855438_28.stderr parsl.dataflow.dflow DEBUG: Task 4293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4293 has memoization hash f49c5a433a195a3bc80dbe8a4b5f74b8 parsl.dataflow.memoization INFO: Task 4293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4293 try 0 launched on executor multi with executor id 4294 parsl.dataflow.dflow INFO: Standard output for task 4293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5b30f73c-f8a1-48d5-858a-319cc0842653_isrForCrosstalkSources_855427_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5b30f73c-f8a1-48d5-858a-319cc0842653_isrForCrosstalkSources_855427_58.stderr parsl.dataflow.dflow DEBUG: Task 4294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4294 has memoization hash b5ccf1874b07b1f8be5e463c5e4820aa parsl.dataflow.memoization INFO: Task 4294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4294 try 0 launched on executor multi with executor id 4295 parsl.dataflow.dflow INFO: Standard output for task 4294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/e6d112da-6181-429b-9ff5-1b0c8ccc5815_isrForCrosstalkSources_855410_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/e6d112da-6181-429b-9ff5-1b0c8ccc5815_isrForCrosstalkSources_855410_11.stderr parsl.dataflow.dflow DEBUG: Task 4295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4295 has memoization hash b879b051653cdb402722c70602ddb9b6 parsl.dataflow.memoization INFO: Task 4295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5db58ed1-f632-4362-85fc-87a62823e911_isrForCrosstalkSources_855416_57.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5db58ed1-f632-4362-85fc-87a62823e911_isrForCrosstalkSources_855416_57.stderr parsl.dataflow.dflow INFO: Parsl task 4295 try 0 launched on executor multi with executor id 4296 parsl.dataflow.dflow INFO: Standard output for task 4295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0bd859d6-8bfb-4c5f-89fa-513b1ea68a34_isrForCrosstalkSources_855387_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0bd859d6-8bfb-4c5f-89fa-513b1ea68a34_isrForCrosstalkSources_855387_4.stderr parsl.dataflow.dflow DEBUG: Task 4296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4296 has memoization hash e6bb33c1d6487d27b230520f5e724f5c parsl.dataflow.memoization INFO: Task 4296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4296 try 0 launched on executor multi with executor id 4297 parsl.dataflow.dflow INFO: Standard output for task 4296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/87cc0c1c-c4e4-4893-abec-c94a170da88c_isrForCrosstalkSources_855426_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/87cc0c1c-c4e4-4893-abec-c94a170da88c_isrForCrosstalkSources_855426_56.stderr parsl.dataflow.dflow DEBUG: Task 4297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4297 has memoization hash ba27e8582495a14cf06bdf2945651797 parsl.dataflow.memoization INFO: Task 4297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4297 try 0 launched on executor multi with executor id 4298 parsl.dataflow.dflow INFO: Standard output for task 4297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e97547da-140d-4e04-8a67-9f0da856fcee_isrForCrosstalkSources_855439_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e97547da-140d-4e04-8a67-9f0da856fcee_isrForCrosstalkSources_855439_10.stderr parsl.dataflow.dflow DEBUG: Task 4298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4298 has memoization hash 111638c31e11a03fbce276f94b8ce829 parsl.dataflow.memoization INFO: Task 4298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a614e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4298 try 0 launched on executor multi with executor id 4299 parsl.dataflow.dflow INFO: Standard output for task 4298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dda2ad34-605b-463f-a99c-06f701fc3b1c_isrForCrosstalkSources_855387_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dda2ad34-605b-463f-a99c-06f701fc3b1c_isrForCrosstalkSources_855387_25.stderr parsl.dataflow.dflow DEBUG: Task 4299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4299 has memoization hash 8f60a80562405fc96ae446f84c9b2cce parsl.dataflow.memoization INFO: Task 4299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4299 try 0 launched on executor multi with executor id 4300 parsl.dataflow.dflow INFO: Standard output for task 4299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/1914859f-9cba-4d0e-846f-9b4a4c6684dc_isrForCrosstalkSources_855415_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/1914859f-9cba-4d0e-846f-9b4a4c6684dc_isrForCrosstalkSources_855415_59.stderr parsl.dataflow.dflow DEBUG: Task 4300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4300 has memoization hash 356638f89de0894f2c150e09551d8a09 parsl.dataflow.memoization INFO: Task 4300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4300 try 0 launched on executor multi with executor id 4301 parsl.dataflow.dflow INFO: Standard output for task 4300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/de5c6215-a127-4db6-ba02-b1a8da047f6f_isrForCrosstalkSources_855387_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/de5c6215-a127-4db6-ba02-b1a8da047f6f_isrForCrosstalkSources_855387_48.stderr parsl.dataflow.dflow DEBUG: Task 4301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4301 has memoization hash 253b6f7cb1a3d3b4ce9144e56bca3687 parsl.dataflow.memoization INFO: Task 4301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a613a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4301 try 0 launched on executor multi with executor id 4302 parsl.dataflow.dflow INFO: Standard output for task 4301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a20412f8-ecd1-47a5-9575-47a06441b008_isrForCrosstalkSources_855418_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a20412f8-ecd1-47a5-9575-47a06441b008_isrForCrosstalkSources_855418_41.stderr parsl.dataflow.dflow DEBUG: Task 4302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4302 has memoization hash 67956917ca5c735047673a30ecb2fa25 parsl.dataflow.memoization INFO: Task 4302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4302 try 0 launched on executor multi with executor id 4303 parsl.dataflow.dflow INFO: Standard output for task 4302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4985e6b5-85b3-4b43-95fb-fc6daf8c133a_isrForCrosstalkSources_855383_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4985e6b5-85b3-4b43-95fb-fc6daf8c133a_isrForCrosstalkSources_855383_30.stderr parsl.dataflow.dflow DEBUG: Task 4303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4303 has memoization hash c92bfa071066d46180b593ab1892ef79 parsl.dataflow.memoization INFO: Task 4303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4303 try 0 launched on executor multi with executor id 4304 parsl.dataflow.dflow INFO: Standard output for task 4303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ae6c3658-420a-4b6b-97d7-05db0a2f91e2_isrForCrosstalkSources_855424_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ae6c3658-420a-4b6b-97d7-05db0a2f91e2_isrForCrosstalkSources_855424_8.stderr parsl.dataflow.dflow DEBUG: Task 4304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4304 has memoization hash 9b9e1543d750fbe4b760dadcc2c78e04 parsl.dataflow.memoization INFO: Task 4304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a616c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4304 try 0 launched on executor multi with executor id 4305 parsl.dataflow.dflow INFO: Standard output for task 4304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e416f927-4d8c-4665-8237-05ec79c33a0b_isrForCrosstalkSources_855391_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e416f927-4d8c-4665-8237-05ec79c33a0b_isrForCrosstalkSources_855391_29.stderr parsl.dataflow.dflow DEBUG: Task 4305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4305 has memoization hash 4a11c7214358e6948cf8b02160bfd4d6 parsl.dataflow.memoization INFO: Task 4305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4305 try 0 launched on executor multi with executor id 4306 parsl.dataflow.dflow INFO: Standard output for task 4305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/47808243-0548-4a62-9e1b-a04fbc2f9a4a_isrForCrosstalkSources_855452_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/47808243-0548-4a62-9e1b-a04fbc2f9a4a_isrForCrosstalkSources_855452_55.stderr parsl.dataflow.dflow DEBUG: Task 4306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4306 has memoization hash cc61ee90f9894092b81db31a3e10e5aa parsl.dataflow.memoization INFO: Task 4306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a628e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4306 try 0 launched on executor multi with executor id 4307 parsl.dataflow.dflow INFO: Standard output for task 4306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d1f4fae9-8085-45f8-b79e-9b6c0e1e3d88_isrForCrosstalkSources_855430_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d1f4fae9-8085-45f8-b79e-9b6c0e1e3d88_isrForCrosstalkSources_855430_29.stderr parsl.dataflow.dflow DEBUG: Task 4307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4307 has memoization hash 4f0075e94b6e34ab70ea13e472f3c07d parsl.dataflow.memoization INFO: Task 4307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4307 try 0 launched on executor multi with executor id 4308 parsl.dataflow.dflow INFO: Standard output for task 4307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c70b02c9-8c15-4091-ab61-78e50b4f68c8_isrForCrosstalkSources_855440_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c70b02c9-8c15-4091-ab61-78e50b4f68c8_isrForCrosstalkSources_855440_6.stderr parsl.dataflow.dflow DEBUG: Task 4308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4308 has memoization hash f6be9ce9302c3b77783f335ed8a427d4 parsl.dataflow.memoization INFO: Task 4308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4308 try 0 launched on executor multi with executor id 4309 parsl.dataflow.dflow INFO: Standard output for task 4308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/392b882f-3600-484f-9748-13cb40acaafc_isrForCrosstalkSources_855382_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/392b882f-3600-484f-9748-13cb40acaafc_isrForCrosstalkSources_855382_39.stderr parsl.dataflow.dflow DEBUG: Task 4309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4309 has memoization hash 099356d4d2bb8e39a85c67d68ba13fcf parsl.dataflow.memoization INFO: Task 4309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4309 try 0 launched on executor multi with executor id 4310 parsl.dataflow.dflow INFO: Standard output for task 4309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d251d0f6-0c9b-45fc-a314-822d4fc3bd77_isrForCrosstalkSources_855432_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d251d0f6-0c9b-45fc-a314-822d4fc3bd77_isrForCrosstalkSources_855432_33.stderr parsl.dataflow.dflow DEBUG: Task 4310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4310 has memoization hash d995a816153c8165d51dfaaec7d59e35 parsl.dataflow.memoization INFO: Task 4310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a627a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4310 try 0 launched on executor multi with executor id 4311 parsl.dataflow.dflow INFO: Standard output for task 4310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a4ee7da2-f2ef-413c-af27-d51219a36388_isrForCrosstalkSources_855381_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a4ee7da2-f2ef-413c-af27-d51219a36388_isrForCrosstalkSources_855381_55.stderr parsl.dataflow.dflow DEBUG: Task 4311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4311 has memoization hash d65f07991b3ce2bc79ddb8f2b6407c6d parsl.dataflow.memoization INFO: Task 4311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4311 try 0 launched on executor multi with executor id 4312 parsl.dataflow.dflow INFO: Standard output for task 4311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a1bf5ad9-0425-4cd6-84fd-0aa3f1de06c1_isrForCrosstalkSources_855439_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a1bf5ad9-0425-4cd6-84fd-0aa3f1de06c1_isrForCrosstalkSources_855439_43.stderr parsl.dataflow.dflow DEBUG: Task 4312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4312 has memoization hash d37f36fa9d2ea9460fdf22d5a68e9948 parsl.dataflow.memoization INFO: Task 4312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4312 try 0 launched on executor multi with executor id 4313 parsl.dataflow.dflow INFO: Standard output for task 4312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/22581263-b603-437b-8f8e-99ce46ba51bc_isrForCrosstalkSources_855393_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/22581263-b603-437b-8f8e-99ce46ba51bc_isrForCrosstalkSources_855393_57.stderr parsl.dataflow.dflow DEBUG: Task 4313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4313 has memoization hash 8f971958c6643d6fba6fd716c50ce233 parsl.dataflow.memoization INFO: Task 4313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4313 try 0 launched on executor multi with executor id 4314 parsl.dataflow.dflow INFO: Standard output for task 4313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1320388-92ad-44f6-8d1f-3d065e48c5c8_isrForCrosstalkSources_855395_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1320388-92ad-44f6-8d1f-3d065e48c5c8_isrForCrosstalkSources_855395_25.stderr parsl.dataflow.dflow DEBUG: Task 4314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4314 has memoization hash 5ee7a1112583c6fdc44a35f6745ce42b parsl.dataflow.memoization INFO: Task 4314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4314 try 0 launched on executor multi with executor id 4315 parsl.dataflow.dflow INFO: Standard output for task 4314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ba2ab040-6fae-45db-9f0a-1717891bbeab_isrForCrosstalkSources_855419_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ba2ab040-6fae-45db-9f0a-1717891bbeab_isrForCrosstalkSources_855419_46.stderr parsl.dataflow.dflow DEBUG: Task 4315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4315 has memoization hash 19a53b6502c92841cf183b4d766c36f7 parsl.dataflow.memoization INFO: Task 4315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a622a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4315 try 0 launched on executor multi with executor id 4316 parsl.dataflow.dflow INFO: Standard output for task 4315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a6b04538-33c6-4edf-9798-3fcfb7c8f696_isrForCrosstalkSources_855451_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a6b04538-33c6-4edf-9798-3fcfb7c8f696_isrForCrosstalkSources_855451_10.stderr parsl.dataflow.dflow DEBUG: Task 4316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4316 has memoization hash 77fe694e2e41a06f5057244921731df8 parsl.dataflow.memoization INFO: Task 4316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4316 try 0 launched on executor multi with executor id 4317 parsl.dataflow.dflow INFO: Standard output for task 4316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/5779e492-7865-4e3a-809c-3e3856837bd5_isrForCrosstalkSources_855415_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/5779e492-7865-4e3a-809c-3e3856837bd5_isrForCrosstalkSources_855415_27.stderr parsl.dataflow.dflow DEBUG: Task 4317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4317 has memoization hash a3493a50b360b77df2ac15152fa266f2 parsl.dataflow.memoization INFO: Task 4317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a623e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:48 proc_lsst.multi:170 [INFO] [multi] job_id 16964189 proc_lsst.multi INFO: [multi] job_id 16964189 2024-03-09 14:51:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-09 14:51:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964189 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964189 parsl.executors.status_handling DEBUG: Launched block 163 on executor multi with job ID 16964189 parsl.executors.status_handling INFO: Allocated block ID 164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-164 2024-03-09 14:51:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-09 14:51:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4317 try 0 launched on executor multi with executor id 4318 parsl.dataflow.dflow INFO: Standard output for task 4317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/605a2946-06cb-4841-ba0e-7ade089fabab_isrForCrosstalkSources_855392_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/605a2946-06cb-4841-ba0e-7ade089fabab_isrForCrosstalkSources_855392_50.stderr parsl.dataflow.dflow DEBUG: Task 4318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4318 has memoization hash 09ea11be2ca1798fa762e4ed4d3e8a3f parsl.dataflow.memoization INFO: Task 4318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4318 try 0 launched on executor multi with executor id 4319 parsl.dataflow.dflow INFO: Standard output for task 4318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c64e15d0-63e1-4daf-987f-e44baedf8025_isrForCrosstalkSources_855421_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c64e15d0-63e1-4daf-987f-e44baedf8025_isrForCrosstalkSources_855421_49.stderr parsl.dataflow.dflow DEBUG: Task 4319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4319 has memoization hash 574025372894836882beb8ef1489062a parsl.dataflow.memoization INFO: Task 4319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4319 try 0 launched on executor multi with executor id 4320 parsl.dataflow.dflow INFO: Standard output for task 4319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/016f279b-dd4d-4798-9264-06fe125ab113_isrForCrosstalkSources_855411_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/016f279b-dd4d-4798-9264-06fe125ab113_isrForCrosstalkSources_855411_19.stderr parsl.dataflow.dflow DEBUG: Task 4320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4320 has memoization hash 435649ce3ed2fe5508f536146cd56947 parsl.dataflow.memoization INFO: Task 4320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4320 try 0 launched on executor multi with executor id 4321 parsl.dataflow.dflow INFO: Standard output for task 4320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3298eae9-9c6e-4462-9c57-b5d8d5624ac2_isrForCrosstalkSources_855457_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3298eae9-9c6e-4462-9c57-b5d8d5624ac2_isrForCrosstalkSources_855457_37.stderr parsl.dataflow.dflow DEBUG: Task 4321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4321 has memoization hash a386668887e1056fa31129cca8e36c51 parsl.dataflow.memoization INFO: Task 4321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4321 try 0 launched on executor multi with executor id 4322 parsl.dataflow.dflow INFO: Standard output for task 4321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8f97bd81-7abf-4f1a-a8fa-6714ca40f4f9_isrForCrosstalkSources_855423_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8f97bd81-7abf-4f1a-a8fa-6714ca40f4f9_isrForCrosstalkSources_855423_56.stderr parsl.dataflow.dflow DEBUG: Task 4322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4322 has memoization hash a0d363e3675c61ea3730ebc10b3a88ab parsl.dataflow.memoization INFO: Task 4322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a611c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4322 try 0 launched on executor multi with executor id 4323 parsl.dataflow.dflow INFO: Standard output for task 4322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4c888835-e1ed-4def-9500-8c3a81f0fe64_isrForCrosstalkSources_855421_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4c888835-e1ed-4def-9500-8c3a81f0fe64_isrForCrosstalkSources_855421_48.stderr parsl.dataflow.dflow DEBUG: Task 4323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4323 has memoization hash 77dda5a931dfde8ac39837141cd17cf3 parsl.dataflow.memoization INFO: Task 4323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4323 try 0 launched on executor multi with executor id 4324 parsl.dataflow.dflow INFO: Standard output for task 4323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/e93b57d3-e999-46c0-8a71-55d9b1c3b7b8_isrForCrosstalkSources_855395_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/e93b57d3-e999-46c0-8a71-55d9b1c3b7b8_isrForCrosstalkSources_855395_15.stderr parsl.dataflow.dflow DEBUG: Task 4324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4324 has memoization hash c556d8d0efaae0cf04b52fdc97cb0e01 parsl.dataflow.memoization INFO: Task 4324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4324 try 0 launched on executor multi with executor id 4325 parsl.dataflow.dflow INFO: Standard output for task 4324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04b89321-1378-4bbb-a4bf-a8cca6969019_isrForCrosstalkSources_855412_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04b89321-1378-4bbb-a4bf-a8cca6969019_isrForCrosstalkSources_855412_21.stderr parsl.dataflow.dflow DEBUG: Task 4325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4325 has memoization hash 6dae2f8d21840e88816fc106d51ab6ac parsl.dataflow.memoization INFO: Task 4325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4325 try 0 launched on executor multi with executor id 4326 parsl.dataflow.dflow INFO: Standard output for task 4325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3c3ba501-f18c-4c91-b2c4-68173f520963_isrForCrosstalkSources_855432_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3c3ba501-f18c-4c91-b2c4-68173f520963_isrForCrosstalkSources_855432_56.stderr parsl.dataflow.dflow DEBUG: Task 4326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4326 has memoization hash ef71501c5e43ace9d537b208e7cadce3 parsl.dataflow.memoization INFO: Task 4326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4326 try 0 launched on executor multi with executor id 4327 parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Standard output for task 4326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e121a2e2-674f-4a18-bf37-64c9f3821d06_isrForCrosstalkSources_855394_58.stdout parsl.dataflow.memoization DEBUG: Storing app cache entry ef6e26b132a4017671d4d0ee7cd1d63a with result from task 524 parsl.dataflow.dflow INFO: Standard error for task 4326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e121a2e2-674f-4a18-bf37-64c9f3821d06_isrForCrosstalkSources_855394_58.stderr parsl.dataflow.dflow DEBUG: Task 4327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4327 has memoization hash 0bfcfddf484ce1ac7bf4097ec387daf1 parsl.dataflow.memoization INFO: Task 4327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c73e254d-06cf-431e-be32-f9ac212573d8_isrForCrosstalkSources_855458_3.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c73e254d-06cf-431e-be32-f9ac212573d8_isrForCrosstalkSources_855458_3.stderr parsl.dataflow.dflow INFO: Parsl task 4327 try 0 launched on executor multi with executor id 4328 parsl.dataflow.dflow INFO: Standard output for task 4327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e26f81aa-4f8a-435c-ab48-a4c07446d2f2_isrForCrosstalkSources_855457_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e26f81aa-4f8a-435c-ab48-a4c07446d2f2_isrForCrosstalkSources_855457_28.stderr parsl.dataflow.dflow DEBUG: Task 4328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4328 has memoization hash fe1b4daee445af4359c5eb01339136f9 parsl.dataflow.memoization INFO: Task 4328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4328 try 0 launched on executor multi with executor id 4329 parsl.dataflow.dflow INFO: Standard output for task 4328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a8fdbe8-868e-489f-bd54-d7e86d65a8f0_isrForCrosstalkSources_855442_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a8fdbe8-868e-489f-bd54-d7e86d65a8f0_isrForCrosstalkSources_855442_19.stderr parsl.dataflow.dflow DEBUG: Task 4329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4329 has memoization hash 1e2318cb479d815cd3bbda5411f0a197 parsl.dataflow.memoization INFO: Task 4329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4329 try 0 launched on executor multi with executor id 4330 parsl.dataflow.dflow INFO: Standard output for task 4329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a8f2aef3-8a7e-46f9-9e7d-c5d07ddf3619_isrForCrosstalkSources_855450_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a8f2aef3-8a7e-46f9-9e7d-c5d07ddf3619_isrForCrosstalkSources_855450_30.stderr parsl.dataflow.dflow DEBUG: Task 4330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4330 has memoization hash 2c6670fe4da91c322a05d18a118ce2bd parsl.dataflow.memoization INFO: Task 4330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a619e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4330 try 0 launched on executor multi with executor id 4331 parsl.dataflow.dflow INFO: Standard output for task 4330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d7a6c32e-f06c-47c8-9b83-bd323f382fa3_isrForCrosstalkSources_855417_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d7a6c32e-f06c-47c8-9b83-bd323f382fa3_isrForCrosstalkSources_855417_2.stderr parsl.dataflow.dflow DEBUG: Task 4331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4331 has memoization hash 550c9b49dedd06d1d827b19031511bc9 parsl.dataflow.memoization INFO: Task 4331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a618a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4331 try 0 launched on executor multi with executor id 4332 parsl.dataflow.dflow INFO: Standard output for task 4331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d14fcdf0-43e7-4e8b-9b09-deca2241f98d_isrForCrosstalkSources_855420_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d14fcdf0-43e7-4e8b-9b09-deca2241f98d_isrForCrosstalkSources_855420_46.stderr parsl.dataflow.dflow DEBUG: Task 4332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4332 has memoization hash e8185e7cdd9bebca80ff1c22cdc7c65a parsl.dataflow.memoization INFO: Task 4332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4332 try 0 launched on executor multi with executor id 4333 parsl.dataflow.dflow INFO: Standard output for task 4332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/56f34de0-2f67-4d1b-b801-14a5826e852b_isrForCrosstalkSources_855399_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/56f34de0-2f67-4d1b-b801-14a5826e852b_isrForCrosstalkSources_855399_13.stderr parsl.dataflow.dflow DEBUG: Task 4333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4333 has memoization hash 005acd61eff21c7c260fd183f6a97c93 parsl.dataflow.memoization INFO: Task 4333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4333 try 0 launched on executor multi with executor id 4334 parsl.dataflow.dflow INFO: Standard output for task 4333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/faf62389-0918-4dc3-813f-ca0cb073ffc8_isrForCrosstalkSources_855396_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/faf62389-0918-4dc3-813f-ca0cb073ffc8_isrForCrosstalkSources_855396_24.stderr parsl.dataflow.dflow DEBUG: Task 4334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4334 has memoization hash 3ca2fc84cb85916278cd5a3daac2c637 parsl.dataflow.memoization INFO: Task 4334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4334 try 0 launched on executor multi with executor id 4335 parsl.dataflow.dflow INFO: Standard output for task 4334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11f54cca-4f8c-41cf-b818-df17022ddfa3_isrForCrosstalkSources_855426_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11f54cca-4f8c-41cf-b818-df17022ddfa3_isrForCrosstalkSources_855426_31.stderr parsl.dataflow.dflow DEBUG: Task 4335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4335 has memoization hash 7b57944e2d17918da0f70739f5d90d9c parsl.dataflow.memoization INFO: Task 4335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4335 try 0 launched on executor multi with executor id 4336 parsl.dataflow.dflow INFO: Standard output for task 4335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0d471279-df8a-4989-a03f-a18ad4fb21e9_isrForCrosstalkSources_855441_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0d471279-df8a-4989-a03f-a18ad4fb21e9_isrForCrosstalkSources_855441_8.stderr parsl.dataflow.dflow DEBUG: Task 4336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4336 has memoization hash a8aec7fbf2996844587e13e2f90a0149 parsl.dataflow.memoization INFO: Task 4336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4336 try 0 launched on executor multi with executor id 4337 parsl.dataflow.dflow INFO: Standard output for task 4336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/280a5be6-1799-4df5-91f2-5605b3aefe84_isrForCrosstalkSources_855380_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/280a5be6-1799-4df5-91f2-5605b3aefe84_isrForCrosstalkSources_855380_50.stderr parsl.dataflow.dflow DEBUG: Task 4337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4337 has memoization hash dcc439a10d044757c9c2cba77f8649a0 parsl.dataflow.memoization INFO: Task 4337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4337 try 0 launched on executor multi with executor id 4338 parsl.dataflow.dflow INFO: Standard output for task 4337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b2b5fabe-4c6a-469d-87a6-96e1bf44b926_isrForCrosstalkSources_855424_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b2b5fabe-4c6a-469d-87a6-96e1bf44b926_isrForCrosstalkSources_855424_1.stderr parsl.dataflow.dflow DEBUG: Task 4338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4338 has memoization hash b8c6c0036395fdfe0040bf487c6154ca parsl.dataflow.memoization INFO: Task 4338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4338 try 0 launched on executor multi with executor id 4339 parsl.dataflow.dflow INFO: Standard output for task 4338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d609c42f-2e4c-4211-885e-842f4d545d7f_isrForCrosstalkSources_855430_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d609c42f-2e4c-4211-885e-842f4d545d7f_isrForCrosstalkSources_855430_48.stderr parsl.dataflow.dflow DEBUG: Task 4339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4339 has memoization hash 9755881d4f11256096ff71bc9de4879e parsl.dataflow.memoization INFO: Task 4339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4339 try 0 launched on executor multi with executor id 4340 parsl.dataflow.dflow INFO: Standard output for task 4339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/84bfc195-7682-4c27-89d2-be325a2d8cea_isrForCrosstalkSources_855381_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/84bfc195-7682-4c27-89d2-be325a2d8cea_isrForCrosstalkSources_855381_33.stderr parsl.dataflow.dflow DEBUG: Task 4340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4340 has memoization hash 959a9cdb0fb5c482c667018eacb91596 parsl.dataflow.memoization INFO: Task 4340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:49 proc_lsst.multi:170 [INFO] [multi] job_id 16964190 proc_lsst.multi INFO: [multi] job_id 16964190 2024-03-09 14:51:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-09 14:51:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964190 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964190 parsl.dataflow.dflow INFO: Parsl task 4340 try 0 launched on executor multi with executor id 4341 parsl.dataflow.dflow INFO: Standard output for task 4340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e7afe286-d946-4c62-a24d-5b81e59c8afe_isrForCrosstalkSources_855399_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e7afe286-d946-4c62-a24d-5b81e59c8afe_isrForCrosstalkSources_855399_43.stderr parsl.dataflow.dflow DEBUG: Task 4341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4341 has memoization hash 23c5ffc7a7b29d5e65ee91f0965aa062 parsl.dataflow.memoization INFO: Task 4341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4341 try 0 launched on executor multi with executor id 4342 parsl.dataflow.dflow INFO: Standard output for task 4341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c53b3e1d-81e2-4fac-9c8f-1e6b7509f0ba_isrForCrosstalkSources_855408_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c53b3e1d-81e2-4fac-9c8f-1e6b7509f0ba_isrForCrosstalkSources_855408_22.stderr parsl.dataflow.dflow DEBUG: Task 4342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4342 has memoization hash f012e7cf5a48aa0e68bb8fc7e371b963 parsl.dataflow.memoization INFO: Task 4342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5be68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4342 try 0 launched on executor multi with executor id 4343 parsl.dataflow.dflow INFO: Standard output for task 4342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a1140311-00d8-4046-becb-6aab00b1ebfc_isrForCrosstalkSources_855418_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a1140311-00d8-4046-becb-6aab00b1ebfc_isrForCrosstalkSources_855418_58.stderr parsl.dataflow.dflow DEBUG: Task 4343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4343 has memoization hash 94639724a823488051d76489d4424a19 parsl.dataflow.memoization INFO: Task 4343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f4e101c604a1b70e98041bf02f272b3 with result from task 533 parsl.executors.status_handling DEBUG: Launched block 164 on executor multi with job ID 16964190 parsl.executors.status_handling INFO: Allocated block ID 165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-165 2024-03-09 14:51:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 2024-03-09 14:51:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-09 14:51:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4343 try 0 launched on executor multi with executor id 4344 parsl.dataflow.dflow INFO: Standard output for task 4343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/45da5ece-174d-4d1d-8bff-e1019e5c055e_isrForCrosstalkSources_855386_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/45da5ece-174d-4d1d-8bff-e1019e5c055e_isrForCrosstalkSources_855386_45.stderr parsl.dataflow.dflow DEBUG: Task 4344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4344 has memoization hash ea905992f51731cc5bcb414ffd5a7513 parsl.dataflow.memoization INFO: Task 4344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4344 try 0 launched on executor multi with executor id 4345 parsl.dataflow.dflow INFO: Standard output for task 4344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/034cbe0e-7ee6-4b2b-b767-ef264a3014fd_isrForCrosstalkSources_855430_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/034cbe0e-7ee6-4b2b-b767-ef264a3014fd_isrForCrosstalkSources_855430_3.stderr parsl.dataflow.dflow DEBUG: Task 4345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4345 has memoization hash e8864c21e7279373f86c3209b0630c02 parsl.dataflow.memoization INFO: Task 4345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4345 try 0 launched on executor multi with executor id 4346 parsl.dataflow.dflow INFO: Standard output for task 4345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c85f8162-c9d2-4c78-8d01-5e750fc80a6a_isrForCrosstalkSources_855390_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c85f8162-c9d2-4c78-8d01-5e750fc80a6a_isrForCrosstalkSources_855390_47.stderr parsl.dataflow.dflow DEBUG: Task 4346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Task 4346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/16f630aa-fbba-488d-acab-47a6bcc9c8b4_isrForCrosstalkSources_855400_43.stdout parsl.dataflow.dflow DEBUG: Task 4346 set to pending state with AppFuture: parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/16f630aa-fbba-488d-acab-47a6bcc9c8b4_isrForCrosstalkSources_855400_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4346 has memoization hash 1ad018f34916556651d1a1760129c489 parsl.dataflow.memoization INFO: Task 4346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a62fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.memoization DEBUG: Storing app cache entry a750ed23b47c9991f8851bbe1326acea with result from task 304 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/9ff45d83-3f49-4e70-853b-29c18af91409_isrForCrosstalkSources_855391_59.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/9ff45d83-3f49-4e70-853b-29c18af91409_isrForCrosstalkSources_855391_59.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2ac86e9931c22ef461f36e3c9ccbf5d with result from task 173 parsl.dataflow.dflow INFO: Parsl task 4346 try 0 launched on executor multi with executor id 4347 parsl.dataflow.dflow INFO: Standard output for task 4346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2a7c569b-89a9-49f1-bc1d-b7b544989edf_isrForCrosstalkSources_855419_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2a7c569b-89a9-49f1-bc1d-b7b544989edf_isrForCrosstalkSources_855419_20.stderr parsl.dataflow.dflow DEBUG: Task 4347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4347 has memoization hash 0b621b1f55f28e8cd48d6bf3f7ff9767 parsl.dataflow.memoization INFO: Task 4347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3ab1f7a6-9d8b-470d-bad7-729b4483cbf0_isrForCrosstalkSources_855403_59.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3ab1f7a6-9d8b-470d-bad7-729b4483cbf0_isrForCrosstalkSources_855403_59.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f5e4d865855573affd860bb6d608771 with result from task 589 parsl.dataflow.dflow INFO: Parsl task 4347 try 0 launched on executor multi with executor id 4348 parsl.dataflow.dflow INFO: Standard output for task 4347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20298ed1-f6cd-4c96-96a3-b4ff1f4709bd_isrForCrosstalkSources_855419_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20298ed1-f6cd-4c96-96a3-b4ff1f4709bd_isrForCrosstalkSources_855419_41.stderr parsl.dataflow.dflow DEBUG: Task 4348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4348 has memoization hash 85820042dfa0e03b007217d06e6c17fe parsl.dataflow.memoization INFO: Task 4348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a629eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4348 try 0 launched on executor multi with executor id 4349 parsl.dataflow.dflow INFO: Standard output for task 4348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bf81d2c1-cf4e-4c39-921c-06066c330c7d_isrForCrosstalkSources_855430_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bf81d2c1-cf4e-4c39-921c-06066c330c7d_isrForCrosstalkSources_855430_11.stderr parsl.dataflow.dflow DEBUG: Task 4349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4349 has memoization hash f34b1b0a612ae9785c3af05e7acc6505 parsl.dataflow.memoization INFO: Task 4349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f6f15a12-7311-49b1-90f5-151a699a40c3_isrForCrosstalkSources_855428_8.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f6f15a12-7311-49b1-90f5-151a699a40c3_isrForCrosstalkSources_855428_8.stderr parsl.dataflow.dflow INFO: Parsl task 4349 try 0 launched on executor multi with executor id 4350 parsl.dataflow.dflow INFO: Standard output for task 4349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2568793-13d0-4029-9a2a-b749a655ed11_isrForCrosstalkSources_855385_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2568793-13d0-4029-9a2a-b749a655ed11_isrForCrosstalkSources_855385_25.stderr parsl.dataflow.dflow DEBUG: Task 4350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4350 has memoization hash e1e9f023bf27175d07583dabd91e811f parsl.dataflow.memoization INFO: Task 4350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a634c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4350 try 0 launched on executor multi with executor id 4351 parsl.dataflow.dflow INFO: Standard output for task 4350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e73b2c18-1ac7-4fca-b1ec-c5f4b948f000_isrForCrosstalkSources_855416_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e73b2c18-1ac7-4fca-b1ec-c5f4b948f000_isrForCrosstalkSources_855416_30.stderr parsl.dataflow.dflow DEBUG: Task 4351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4351 has memoization hash d54f918420c7fc5ec26cc211bd036005 parsl.dataflow.memoization INFO: Task 4351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4351 try 0 launched on executor multi with executor id 4352 parsl.dataflow.dflow INFO: Standard output for task 4351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/be8c3e03-9c64-44bf-8995-2a09eefac86e_isrForCrosstalkSources_855388_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/be8c3e03-9c64-44bf-8995-2a09eefac86e_isrForCrosstalkSources_855388_20.stderr parsl.dataflow.dflow DEBUG: Task 4352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4352 has memoization hash bf80e760d16128dde3b96fc4ff074f17 parsl.dataflow.memoization INFO: Task 4352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4352 try 0 launched on executor multi with executor id 4353 parsl.dataflow.dflow INFO: Standard output for task 4352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2530b659-1ad2-418c-bbb0-c7cabc85e22b_isrForCrosstalkSources_855406_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2530b659-1ad2-418c-bbb0-c7cabc85e22b_isrForCrosstalkSources_855406_37.stderr parsl.dataflow.dflow DEBUG: Task 4353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4353 has memoization hash b3b68194b8f7b0f103129d8c194b7d4c parsl.dataflow.memoization INFO: Task 4353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4353 try 0 launched on executor multi with executor id 4354 parsl.dataflow.dflow INFO: Standard output for task 4353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cfdeaba8-05c2-40ab-9e42-50bc0e566918_isrForCrosstalkSources_855407_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cfdeaba8-05c2-40ab-9e42-50bc0e566918_isrForCrosstalkSources_855407_51.stderr parsl.dataflow.dflow DEBUG: Task 4354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4354 has memoization hash 34fcffefe719bd4ff1b48e754977a9b2 parsl.dataflow.memoization INFO: Task 4354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4354 try 0 launched on executor multi with executor id 4355 parsl.dataflow.dflow INFO: Standard output for task 4354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c35cbf34-ebe3-4364-a94e-3c17139cd353_isrForCrosstalkSources_855442_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c35cbf34-ebe3-4364-a94e-3c17139cd353_isrForCrosstalkSources_855442_12.stderr parsl.dataflow.dflow DEBUG: Task 4355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4355 has memoization hash 3874b679bb11ee05d633e529ec7628e0 parsl.dataflow.memoization INFO: Task 4355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4355 try 0 launched on executor multi with executor id 4356 parsl.dataflow.dflow INFO: Standard output for task 4355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/888f5f1c-e305-470e-a352-91a28cd5361f_isrForCrosstalkSources_855382_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/888f5f1c-e305-470e-a352-91a28cd5361f_isrForCrosstalkSources_855382_1.stderr parsl.dataflow.dflow DEBUG: Task 4356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4356 has memoization hash 899306a43e0c9aa0b6a04107d9dd71f3 parsl.dataflow.memoization INFO: Task 4356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a639c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4356 try 0 launched on executor multi with executor id 4357 parsl.dataflow.dflow INFO: Standard output for task 4356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8ccb4dd0-6455-4105-9c84-3200ef1d6669_isrForCrosstalkSources_855442_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8ccb4dd0-6455-4105-9c84-3200ef1d6669_isrForCrosstalkSources_855442_26.stderr parsl.dataflow.dflow DEBUG: Task 4357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4357 has memoization hash fe56f862b9db7e599647722d3168d9cc parsl.dataflow.memoization INFO: Task 4357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4357 try 0 launched on executor multi with executor id 4358 parsl.dataflow.dflow INFO: Standard output for task 4357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91306bef-3d40-47e2-9646-a1c315afaf9f_isrForCrosstalkSources_855390_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91306bef-3d40-47e2-9646-a1c315afaf9f_isrForCrosstalkSources_855390_31.stderr parsl.dataflow.dflow DEBUG: Task 4358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4358 has memoization hash 1be5c91b3d30721afa48e8e3d188c9fe parsl.dataflow.memoization INFO: Task 4358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4358 try 0 launched on executor multi with executor id 4359 parsl.dataflow.dflow INFO: Standard output for task 4358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6cf854e3-546d-426a-b28a-01c66bb65b6c_isrForCrosstalkSources_855410_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6cf854e3-546d-426a-b28a-01c66bb65b6c_isrForCrosstalkSources_855410_40.stderr parsl.dataflow.dflow DEBUG: Task 4359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4359 has memoization hash 275701ac1c43b979930051671d9ef932 parsl.dataflow.memoization INFO: Task 4359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4359 try 0 launched on executor multi with executor id 4360 parsl.dataflow.dflow INFO: Standard output for task 4359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b5e28533-b472-4df0-b82f-29c193e87f2a_isrForCrosstalkSources_855388_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b5e28533-b472-4df0-b82f-29c193e87f2a_isrForCrosstalkSources_855388_30.stderr parsl.dataflow.dflow DEBUG: Task 4360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4360 has memoization hash 649bd2506b37e6a0d6ee0abaf3b8a5b5 parsl.dataflow.memoization INFO: Task 4360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4360 try 0 launched on executor multi with executor id 4361 parsl.dataflow.dflow INFO: Standard output for task 4360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/fe706ac7-5dd1-479c-8544-46e4c090b1c0_isrForCrosstalkSources_855386_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/fe706ac7-5dd1-479c-8544-46e4c090b1c0_isrForCrosstalkSources_855386_53.stderr parsl.dataflow.dflow DEBUG: Task 4361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4361 has memoization hash c8e3f6a72d08bc2ea7d30c61c037396c parsl.dataflow.memoization INFO: Task 4361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a637e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4361 try 0 launched on executor multi with executor id 4362 parsl.dataflow.dflow INFO: Standard output for task 4361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9d3edea4-2c68-455d-8f45-71cc0e8543e5_isrForCrosstalkSources_855403_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9d3edea4-2c68-455d-8f45-71cc0e8543e5_isrForCrosstalkSources_855403_3.stderr parsl.dataflow.dflow DEBUG: Task 4362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4362 has memoization hash cddcab48801d96241b59789740fda6a5 parsl.dataflow.memoization INFO: Task 4362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4362 try 0 launched on executor multi with executor id 4363 parsl.dataflow.dflow INFO: Standard output for task 4362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/eece1abb-9683-44bd-aeea-aa2442abc8d9_isrForCrosstalkSources_855415_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/eece1abb-9683-44bd-aeea-aa2442abc8d9_isrForCrosstalkSources_855415_50.stderr parsl.dataflow.dflow DEBUG: Task 4363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4363 has memoization hash 390865dd61f4675f14193d00f83e635e parsl.dataflow.memoization INFO: Task 4363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4363 try 0 launched on executor multi with executor id 4364 parsl.dataflow.dflow INFO: Standard output for task 4363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/32d12c6e-64d8-45b3-8241-5acd6d32c16c_isrForCrosstalkSources_855445_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/32d12c6e-64d8-45b3-8241-5acd6d32c16c_isrForCrosstalkSources_855445_59.stderr parsl.dataflow.dflow DEBUG: Task 4364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4364 has memoization hash 04c5a2070f9e0c43cac371a8ff4bb439 parsl.dataflow.memoization INFO: Task 4364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4364 try 0 launched on executor multi with executor id 4365 parsl.dataflow.dflow INFO: Standard output for task 4364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/260908fd-91e3-4816-b5d3-1050e1112e56_isrForCrosstalkSources_855435_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/260908fd-91e3-4816-b5d3-1050e1112e56_isrForCrosstalkSources_855435_13.stderr parsl.dataflow.dflow DEBUG: Task 4365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4365 has memoization hash 9d589356c17738528f01097a510d9bb7 parsl.dataflow.memoization INFO: Task 4365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4365 try 0 launched on executor multi with executor id 4366 parsl.dataflow.dflow INFO: Standard output for task 4365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0b3c2fc-616f-409a-8a4b-4e8be36fea1f_isrForCrosstalkSources_855380_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0b3c2fc-616f-409a-8a4b-4e8be36fea1f_isrForCrosstalkSources_855380_15.stderr parsl.dataflow.dflow DEBUG: Task 4366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4366 has memoization hash 2b7e265b75c3593496906817e00ed3b2 parsl.dataflow.memoization INFO: Task 4366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4366 try 0 launched on executor multi with executor id 4367 parsl.dataflow.dflow INFO: Standard output for task 4366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/07e94a09-8c5a-49b5-8322-c97a8195a171_isrForCrosstalkSources_855394_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/07e94a09-8c5a-49b5-8322-c97a8195a171_isrForCrosstalkSources_855394_19.stderr parsl.dataflow.dflow DEBUG: Task 4367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4367 has memoization hash 8ab13e8f3f0cbff3671ae748f862358b parsl.dataflow.memoization INFO: Task 4367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4367 try 0 launched on executor multi with executor id 4368 parsl.dataflow.dflow INFO: Standard output for task 4367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7a6daf8f-eed1-48de-b47f-be567fb22812_isrForCrosstalkSources_855393_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7a6daf8f-eed1-48de-b47f-be567fb22812_isrForCrosstalkSources_855393_35.stderr parsl.dataflow.dflow DEBUG: Task 4368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4368 has memoization hash 1004d97258f4177212c085eb712df3b8 parsl.dataflow.memoization INFO: Task 4368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4368 try 0 launched on executor multi with executor id 4369 parsl.dataflow.dflow INFO: Standard output for task 4368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/ebe34178-d732-45f3-8181-56a0f2e43f55_isrForCrosstalkSources_855428_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/ebe34178-d732-45f3-8181-56a0f2e43f55_isrForCrosstalkSources_855428_18.stderr parsl.dataflow.dflow DEBUG: Task 4369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4369 has memoization hash 4c073412bac669d86e68de239ef07a7e parsl.dataflow.memoization INFO: Task 4369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4369 try 0 launched on executor multi with executor id 4370 parsl.dataflow.dflow INFO: Standard output for task 4369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23588e2d-f71e-43cc-9fd2-df5fc89f50c6_isrForCrosstalkSources_855396_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23588e2d-f71e-43cc-9fd2-df5fc89f50c6_isrForCrosstalkSources_855396_42.stderr parsl.dataflow.dflow DEBUG: Task 4370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4370 has memoization hash 67f466e704b5650ae2d9a8de3c82a5e4 parsl.dataflow.memoization INFO: Task 4370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4370 try 0 launched on executor multi with executor id 4371 parsl.dataflow.dflow INFO: Standard output for task 4370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/28bad6e2-57fc-4502-8f27-725571624724_isrForCrosstalkSources_855421_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/28bad6e2-57fc-4502-8f27-725571624724_isrForCrosstalkSources_855421_41.stderr parsl.dataflow.dflow DEBUG: Task 4371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4371 has memoization hash 81ae21f1b2893a4495d36522ee0363fc parsl.dataflow.memoization INFO: Task 4371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:50 proc_lsst.multi:170 [INFO] [multi] job_id 16964191 proc_lsst.multi INFO: [multi] job_id 16964191 2024-03-09 14:51:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-09 14:51:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964191 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964191 parsl.executors.status_handling DEBUG: Launched block 165 on executor multi with job ID 16964191 parsl.executors.status_handling INFO: Allocated block ID 166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-166 2024-03-09 14:51:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4371 try 0 launched on executor multi with executor id 4372 parsl.dataflow.dflow INFO: Standard output for task 4371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1500d13c-29b9-4fec-81c0-184ddde0efdc_isrForCrosstalkSources_855397_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1500d13c-29b9-4fec-81c0-184ddde0efdc_isrForCrosstalkSources_855397_39.stderr parsl.dataflow.dflow DEBUG: Task 4372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4372 has memoization hash 628704ef831effcc9e272980d5d430c7 parsl.dataflow.memoization INFO: Task 4372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4372 try 0 launched on executor multi with executor id 4373 parsl.dataflow.dflow INFO: Standard output for task 4372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/92090860-1d56-4d54-8946-6f56d2970484_isrForCrosstalkSources_855379_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/92090860-1d56-4d54-8946-6f56d2970484_isrForCrosstalkSources_855379_10.stderr parsl.dataflow.dflow DEBUG: Task 4373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4373 has memoization hash 6ec64000a4368cffc36f5c7ac2c92a07 parsl.dataflow.memoization INFO: Task 4373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a631a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4373 try 0 launched on executor multi with executor id 4374 parsl.dataflow.dflow INFO: Standard output for task 4373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/9868d12e-0c2c-49f1-8ea6-7c644d319fa0_isrForCrosstalkSources_855453_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/9868d12e-0c2c-49f1-8ea6-7c644d319fa0_isrForCrosstalkSources_855453_24.stderr parsl.dataflow.dflow DEBUG: Task 4374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4374 has memoization hash f6956d27926b8887ce206a5cf91eb77a parsl.dataflow.memoization INFO: Task 4374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4374 try 0 launched on executor multi with executor id 4375 parsl.dataflow.dflow INFO: Standard output for task 4374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/609fd9c9-974a-4b7a-b802-68f1f95c936f_isrForCrosstalkSources_855424_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/609fd9c9-974a-4b7a-b802-68f1f95c936f_isrForCrosstalkSources_855424_50.stderr parsl.dataflow.dflow DEBUG: Task 4375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4375 has memoization hash d70dd8f88b4e074e432105025ecbabe6 parsl.dataflow.memoization INFO: Task 4375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4375 try 0 launched on executor multi with executor id 4376 parsl.dataflow.dflow INFO: Standard output for task 4375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/71c041a9-0c97-4ca6-ae4e-2e5022970a1e_isrForCrosstalkSources_855428_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/71c041a9-0c97-4ca6-ae4e-2e5022970a1e_isrForCrosstalkSources_855428_36.stderr parsl.dataflow.dflow DEBUG: Task 4376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4376 has memoization hash e1c8956e3ac13dfdea208e36d840ba4a parsl.dataflow.memoization INFO: Task 4376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4376 try 0 launched on executor multi with executor id 4377 parsl.dataflow.dflow INFO: Standard output for task 4376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7cfc70d-46cc-42bd-ab00-e366d4755953_isrForCrosstalkSources_855382_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7cfc70d-46cc-42bd-ab00-e366d4755953_isrForCrosstalkSources_855382_24.stderr parsl.dataflow.dflow DEBUG: Task 4377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4377 has memoization hash e71cca44fd8b6059c5060334e2c7781a parsl.dataflow.memoization INFO: Task 4377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4377 try 0 launched on executor multi with executor id 4378 parsl.dataflow.dflow INFO: Standard output for task 4377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6ca5fd3d-6cd0-447b-b6fa-b9c1c399cbaa_isrForCrosstalkSources_855410_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6ca5fd3d-6cd0-447b-b6fa-b9c1c399cbaa_isrForCrosstalkSources_855410_46.stderr parsl.dataflow.dflow DEBUG: Task 4378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4378 has memoization hash 0b583b6e29b0cae8483666540d297e17 parsl.dataflow.memoization INFO: Task 4378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4378 try 0 launched on executor multi with executor id 4379 parsl.dataflow.dflow INFO: Standard output for task 4378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2559b0f2-abf9-4d5a-a2c8-23e8197484c5_isrForCrosstalkSources_855424_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2559b0f2-abf9-4d5a-a2c8-23e8197484c5_isrForCrosstalkSources_855424_58.stderr parsl.dataflow.dflow DEBUG: Task 4379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4379 has memoization hash d8aab19a1e72cc053314212eb4c8e7c2 parsl.dataflow.memoization INFO: Task 4379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4379 try 0 launched on executor multi with executor id 4380 parsl.dataflow.dflow INFO: Standard output for task 4379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/26bfe208-086e-4142-b69f-85b83af5e597_isrForCrosstalkSources_855392_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/26bfe208-086e-4142-b69f-85b83af5e597_isrForCrosstalkSources_855392_37.stderr parsl.dataflow.dflow DEBUG: Task 4380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4380 has memoization hash 8bcd1837d4239523a59b336ad8e3f738 parsl.dataflow.memoization INFO: Task 4380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4380 try 0 launched on executor multi with executor id 4381 parsl.dataflow.dflow INFO: Standard output for task 4380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6d72d1d2-e407-4db5-857d-4039541801f6_isrForCrosstalkSources_855431_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6d72d1d2-e407-4db5-857d-4039541801f6_isrForCrosstalkSources_855431_53.stderr parsl.dataflow.dflow DEBUG: Task 4381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4381 has memoization hash 36595643a29f5f30795c6edd17f3c9c8 parsl.dataflow.memoization INFO: Task 4381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4381 try 0 launched on executor multi with executor id 4382 parsl.dataflow.dflow INFO: Standard output for task 4381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/47674b6d-e83a-4abb-90b3-8f5bad1719d1_isrForCrosstalkSources_855456_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/47674b6d-e83a-4abb-90b3-8f5bad1719d1_isrForCrosstalkSources_855456_14.stderr parsl.dataflow.dflow DEBUG: Task 4382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4382 has memoization hash 6be5ddbf02c65c8aa9c7513abfe43e7c parsl.dataflow.memoization INFO: Task 4382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4382 try 0 launched on executor multi with executor id 4383 parsl.dataflow.dflow INFO: Standard output for task 4382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8feb2d1b-61c7-4046-a24a-b585563d6d6e_isrForCrosstalkSources_855444_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8feb2d1b-61c7-4046-a24a-b585563d6d6e_isrForCrosstalkSources_855444_61.stderr parsl.dataflow.dflow DEBUG: Task 4383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4383 has memoization hash 2e93f38797e81c06c6f8931fe9678287 parsl.dataflow.memoization INFO: Task 4383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4383 try 0 launched on executor multi with executor id 4384 parsl.dataflow.dflow INFO: Standard output for task 4383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4393b30e-1934-4044-b920-bdbe31fb6d94_isrForCrosstalkSources_855415_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4393b30e-1934-4044-b920-bdbe31fb6d94_isrForCrosstalkSources_855415_36.stderr parsl.dataflow.dflow DEBUG: Task 4384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4384 has memoization hash 15cdd48327cb1bf50a98e48a54a36da4 parsl.dataflow.memoization INFO: Task 4384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4384 try 0 launched on executor multi with executor id 4385 parsl.dataflow.dflow INFO: Standard output for task 4384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/de92076b-d802-4278-858c-d8a45f125bd4_isrForCrosstalkSources_855389_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/de92076b-d802-4278-858c-d8a45f125bd4_isrForCrosstalkSources_855389_58.stderr parsl.dataflow.dflow DEBUG: Task 4385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4385 has memoization hash 349eb536da0be0cd838a1cf63db16cc4 parsl.dataflow.memoization INFO: Task 4385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4385 try 0 launched on executor multi with executor id 4386 parsl.dataflow.dflow INFO: Standard output for task 4385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/aa0d2277-8f6a-437c-993f-65549dbed3c4_isrForCrosstalkSources_855379_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/aa0d2277-8f6a-437c-993f-65549dbed3c4_isrForCrosstalkSources_855379_53.stderr parsl.dataflow.dflow DEBUG: Task 4386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4386 has memoization hash ec2047f31863743997855a39730d8031 parsl.dataflow.memoization INFO: Task 4386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4386 try 0 launched on executor multi with executor id 4387 parsl.dataflow.dflow INFO: Standard output for task 4386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/96e1848b-76ac-4ed4-a6a9-0cabca0f790b_isrForCrosstalkSources_855394_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/96e1848b-76ac-4ed4-a6a9-0cabca0f790b_isrForCrosstalkSources_855394_23.stderr parsl.dataflow.dflow DEBUG: Task 4387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4387 has memoization hash 57b8de7bec4e8241f8d4e2172707bc5a parsl.dataflow.memoization INFO: Task 4387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4387 try 0 launched on executor multi with executor id 4388 parsl.dataflow.dflow INFO: Standard output for task 4387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/780a115a-890a-470b-849a-9b53fa0db762_isrForCrosstalkSources_855435_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/780a115a-890a-470b-849a-9b53fa0db762_isrForCrosstalkSources_855435_58.stderr parsl.dataflow.dflow DEBUG: Task 4388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4388 has memoization hash c8b6e9eb42be07b121c1d6e0be86cc55 parsl.dataflow.memoization INFO: Task 4388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4388 try 0 launched on executor multi with executor id 4389 parsl.dataflow.dflow INFO: Standard output for task 4388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/055575f9-03a0-4ea5-915e-5212e42c7ab0_isrForCrosstalkSources_855425_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/055575f9-03a0-4ea5-915e-5212e42c7ab0_isrForCrosstalkSources_855425_4.stderr parsl.dataflow.dflow DEBUG: Task 4389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4389 has memoization hash c24feea3d81f2d62370ec62d94ca91ac parsl.dataflow.memoization INFO: Task 4389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4389 try 0 launched on executor multi with executor id 4390 parsl.dataflow.dflow INFO: Standard output for task 4389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2dc70254-7757-4770-8dbb-8a7341a3344f_isrForCrosstalkSources_855418_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2dc70254-7757-4770-8dbb-8a7341a3344f_isrForCrosstalkSources_855418_53.stderr parsl.dataflow.dflow DEBUG: Task 4390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4390 has memoization hash 8dc5d6fcf88ba10a6bc80cc48a8500f1 parsl.dataflow.memoization INFO: Task 4390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4390 try 0 launched on executor multi with executor id 4391 parsl.dataflow.dflow INFO: Standard output for task 4390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4480618f-6af2-40ab-a3d7-c576baa2ea76_isrForCrosstalkSources_855401_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4480618f-6af2-40ab-a3d7-c576baa2ea76_isrForCrosstalkSources_855401_62.stderr parsl.dataflow.dflow DEBUG: Task 4391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4391 has memoization hash d07b160ae148606c3198746df8e3e429 parsl.dataflow.memoization INFO: Task 4391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4391 try 0 launched on executor multi with executor id 4392 parsl.dataflow.dflow INFO: Standard output for task 4391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/495c3577-204e-4fc3-a090-555caa1a71cb_isrForCrosstalkSources_855403_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/495c3577-204e-4fc3-a090-555caa1a71cb_isrForCrosstalkSources_855403_10.stderr parsl.dataflow.dflow DEBUG: Task 4392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4392 has memoization hash 53a51384d9a8b9e28043db1e36bdc2b1 parsl.dataflow.memoization INFO: Task 4392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4392 try 0 launched on executor multi with executor id 4393 parsl.dataflow.dflow INFO: Standard output for task 4392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce10fdcb-7614-4c64-8926-364ea6de0ca4_isrForCrosstalkSources_855445_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce10fdcb-7614-4c64-8926-364ea6de0ca4_isrForCrosstalkSources_855445_45.stderr parsl.dataflow.dflow DEBUG: Task 4393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4393 has memoization hash 41397e1c46997cc2af6fa0606fb58ea4 parsl.dataflow.memoization INFO: Task 4393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4393 try 0 launched on executor multi with executor id 4394 parsl.dataflow.dflow INFO: Standard output for task 4393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ad25e0d0-81dd-412c-a105-1d219dc9e66c_isrForCrosstalkSources_855409_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ad25e0d0-81dd-412c-a105-1d219dc9e66c_isrForCrosstalkSources_855409_4.stderr parsl.dataflow.dflow DEBUG: Task 4394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4394 has memoization hash 7912cb3fe2c89b2e4112d00fc3c1e998 parsl.dataflow.memoization INFO: Task 4394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4394 try 0 launched on executor multi with executor id 4395 parsl.dataflow.dflow INFO: Standard output for task 4394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/23be3b5e-e824-409d-b8b5-c99841d86b06_isrForCrosstalkSources_855440_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/23be3b5e-e824-409d-b8b5-c99841d86b06_isrForCrosstalkSources_855440_18.stderr parsl.dataflow.dflow DEBUG: Task 4395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4395 has memoization hash 276c8f66ac6c0285f5fc33397baa807a parsl.dataflow.memoization INFO: Task 4395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4395 try 0 launched on executor multi with executor id 4396 parsl.dataflow.dflow INFO: Standard output for task 4395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c27df980-0470-4dca-93e6-fac8ca858183_isrForCrosstalkSources_855387_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c27df980-0470-4dca-93e6-fac8ca858183_isrForCrosstalkSources_855387_20.stderr parsl.dataflow.dflow DEBUG: Task 4396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4396 has memoization hash 250e9bdd62c5c6847b1558216de655a1 parsl.dataflow.memoization INFO: Task 4396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4396 try 0 launched on executor multi with executor id 4397 parsl.dataflow.dflow INFO: Standard output for task 4396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a9305ad1-e2d3-4592-97e4-4e254f2d435d_isrForCrosstalkSources_855398_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a9305ad1-e2d3-4592-97e4-4e254f2d435d_isrForCrosstalkSources_855398_16.stderr parsl.dataflow.dflow DEBUG: Task 4397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4397 has memoization hash 07b63a33c7415c9791ac57e0e3ba5a3a parsl.dataflow.memoization INFO: Task 4397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4397 try 0 launched on executor multi with executor id 4398 parsl.dataflow.dflow INFO: Standard output for task 4397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/56ee23b6-8516-4a5b-8789-bc9695c09d2c_isrForCrosstalkSources_855403_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/56ee23b6-8516-4a5b-8789-bc9695c09d2c_isrForCrosstalkSources_855403_62.stderr parsl.dataflow.dflow DEBUG: Task 4398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4398 has memoization hash 5f7526ed6f9a0e4387c34ab7b6148ce5 parsl.dataflow.memoization INFO: Task 4398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4398 try 0 launched on executor multi with executor id 4399 parsl.dataflow.dflow INFO: Standard output for task 4398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/20a7d027-d24c-4738-a788-0e93bc9fb656_isrForCrosstalkSources_855457_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/20a7d027-d24c-4738-a788-0e93bc9fb656_isrForCrosstalkSources_855457_62.stderr parsl.dataflow.dflow DEBUG: Task 4399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4399 has memoization hash 63bcac0318828eaa09721c704ca78041 parsl.dataflow.memoization INFO: Task 4399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4399 try 0 launched on executor multi with executor id 4400 parsl.dataflow.dflow INFO: Standard output for task 4399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3c6ebad6-4ac6-4b3d-b298-c56c767e962d_isrForCrosstalkSources_855401_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3c6ebad6-4ac6-4b3d-b298-c56c767e962d_isrForCrosstalkSources_855401_53.stderr parsl.dataflow.dflow DEBUG: Task 4400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4400 has memoization hash 7be6b078192efa117b0322ec9807d72f parsl.dataflow.memoization INFO: Task 4400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4400 try 0 launched on executor multi with executor id 4401 parsl.dataflow.dflow INFO: Standard output for task 4400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/db3f70bc-87dc-4306-b8c1-906b450157c0_isrForCrosstalkSources_855399_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/db3f70bc-87dc-4306-b8c1-906b450157c0_isrForCrosstalkSources_855399_1.stderr parsl.dataflow.dflow DEBUG: Task 4401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4401 has memoization hash 4ec2ed0d028b0cd3e84cff8146dc3b53 parsl.dataflow.memoization INFO: Task 4401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4401 try 0 launched on executor multi with executor id 4402 parsl.dataflow.dflow INFO: Standard output for task 4401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7c46f8bb-cee0-46c1-9345-fd9bf3f53a0b_isrForCrosstalkSources_855391_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7c46f8bb-cee0-46c1-9345-fd9bf3f53a0b_isrForCrosstalkSources_855391_3.stderr parsl.dataflow.dflow DEBUG: Task 4402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4402 has memoization hash d68aacf9eaa3f01762459d60def162f2 parsl.dataflow.memoization INFO: Task 4402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4402 try 0 launched on executor multi with executor id 4403 parsl.dataflow.dflow INFO: Standard output for task 4402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/392ba925-34cc-4ea5-8b7e-6187bfe47afd_isrForCrosstalkSources_855394_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/392ba925-34cc-4ea5-8b7e-6187bfe47afd_isrForCrosstalkSources_855394_34.stderr parsl.dataflow.dflow DEBUG: Task 4403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4403 has memoization hash ddf6d147c26ae49c6d888d3bc6625845 parsl.dataflow.memoization INFO: Task 4403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4403 try 0 launched on executor multi with executor id 4404 parsl.dataflow.dflow INFO: Standard output for task 4403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/60783d28-d268-4aeb-9474-9703d9cf18c4_isrForCrosstalkSources_855381_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/60783d28-d268-4aeb-9474-9703d9cf18c4_isrForCrosstalkSources_855381_8.stderr parsl.dataflow.dflow DEBUG: Task 4404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4404 has memoization hash 66586e4830694bce0590fb2e0f310608 parsl.dataflow.memoization INFO: Task 4404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4404 try 0 launched on executor multi with executor id 4405 parsl.dataflow.dflow INFO: Standard output for task 4404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b443d601-f43c-4b0d-859b-c86996e417b1_isrForCrosstalkSources_855436_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b443d601-f43c-4b0d-859b-c86996e417b1_isrForCrosstalkSources_855436_22.stderr parsl.dataflow.dflow DEBUG: Task 4405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4405 has memoization hash 47167d30940208add098550c0200a0dc parsl.dataflow.memoization INFO: Task 4405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4405 try 0 launched on executor multi with executor id 4406 parsl.dataflow.dflow INFO: Standard output for task 4405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ec87c72f-117f-4eb9-9ba5-d290194dfa44_isrForCrosstalkSources_855430_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ec87c72f-117f-4eb9-9ba5-d290194dfa44_isrForCrosstalkSources_855430_2.stderr parsl.dataflow.dflow DEBUG: Task 4406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4406 has memoization hash fd178c3673c569f0f3504e4b75eec7d3 parsl.dataflow.memoization INFO: Task 4406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4406 try 0 launched on executor multi with executor id 4407 parsl.dataflow.dflow INFO: Standard output for task 4406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0c91b94d-1bda-4abc-b501-00c8b03c9a6e_isrForCrosstalkSources_855389_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0c91b94d-1bda-4abc-b501-00c8b03c9a6e_isrForCrosstalkSources_855389_49.stderr parsl.dataflow.dflow DEBUG: Task 4407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4407 has memoization hash c1b15d4ed8ee8371e67f293934c855fc parsl.dataflow.memoization INFO: Task 4407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4407 try 0 launched on executor multi with executor id 4408 parsl.dataflow.dflow INFO: Standard output for task 4407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6864b0cf-8312-48e8-88d8-fd52acef28fb_isrForCrosstalkSources_855391_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6864b0cf-8312-48e8-88d8-fd52acef28fb_isrForCrosstalkSources_855391_19.stderr parsl.dataflow.dflow DEBUG: Task 4408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4408 has memoization hash d84f756ce0b55e7c7471edd838ea4b5c parsl.dataflow.memoization INFO: Task 4408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4408 try 0 launched on executor multi with executor id 4409 parsl.dataflow.dflow INFO: Standard output for task 4408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2222936d-d76a-47bd-9983-220a8690326c_isrForCrosstalkSources_855419_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2222936d-d76a-47bd-9983-220a8690326c_isrForCrosstalkSources_855419_1.stderr parsl.dataflow.dflow DEBUG: Task 4409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4409 has memoization hash 5b0174d700b497a750ddfbd5c7ece765 parsl.dataflow.memoization INFO: Task 4409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4409 try 0 launched on executor multi with executor id 4410 parsl.dataflow.dflow INFO: Standard output for task 4409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e5176301-dd6f-4e0a-be3a-dd1d9506ffd6_isrForCrosstalkSources_855432_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e5176301-dd6f-4e0a-be3a-dd1d9506ffd6_isrForCrosstalkSources_855432_32.stderr parsl.dataflow.dflow DEBUG: Task 4410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4410 has memoization hash f2eb32a486a95c4a3b801a18bfeaf99e parsl.dataflow.memoization INFO: Task 4410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4410 try 0 launched on executor multi with executor id 4411 parsl.dataflow.dflow INFO: Standard output for task 4410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2bbfffd9-4733-4c15-9097-67b14fc20fb2_isrForCrosstalkSources_855396_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2bbfffd9-4733-4c15-9097-67b14fc20fb2_isrForCrosstalkSources_855396_38.stderr parsl.dataflow.dflow DEBUG: Task 4411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4411 has memoization hash e4f1474f03dfcd3c943092b4319d4d08 parsl.dataflow.memoization INFO: Task 4411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4411 try 0 launched on executor multi with executor id 4412 parsl.dataflow.dflow INFO: Standard output for task 4411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c34e469b-65d1-402b-a621-1c5016386edb_isrForCrosstalkSources_855387_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c34e469b-65d1-402b-a621-1c5016386edb_isrForCrosstalkSources_855387_37.stderr parsl.dataflow.dflow DEBUG: Task 4412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4412 has memoization hash dccacc65852845092c97f80632dbf8d2 parsl.dataflow.memoization INFO: Task 4412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4412 try 0 launched on executor multi with executor id 4413 parsl.dataflow.dflow INFO: Standard output for task 4412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9a95bac9-ff8d-491c-bee2-c2cfc51d275f_isrForCrosstalkSources_855444_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9a95bac9-ff8d-491c-bee2-c2cfc51d275f_isrForCrosstalkSources_855444_29.stderr parsl.dataflow.dflow DEBUG: Task 4413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4413 has memoization hash 0d4a64e632f63e48cfb9a8e4383f779a parsl.dataflow.memoization INFO: Task 4413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4413 try 0 launched on executor multi with executor id 4414 parsl.dataflow.dflow INFO: Standard output for task 4413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a1406514-23a4-4a32-bf35-8211cd1e1eb0_isrForCrosstalkSources_855387_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a1406514-23a4-4a32-bf35-8211cd1e1eb0_isrForCrosstalkSources_855387_50.stderr parsl.dataflow.dflow DEBUG: Task 4414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4414 has memoization hash 5336b4b2bac76b5b0d82e9a7769d909b parsl.dataflow.memoization INFO: Task 4414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4414 try 0 launched on executor multi with executor id 4415 parsl.dataflow.dflow INFO: Standard output for task 4414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/625a8c04-2085-4f56-b9bc-458d53bff5a6_isrForCrosstalkSources_855380_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/625a8c04-2085-4f56-b9bc-458d53bff5a6_isrForCrosstalkSources_855380_12.stderr parsl.dataflow.dflow DEBUG: Task 4415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4415 has memoization hash 55430cb38b8e33aeaad7c3f11795eb4c parsl.dataflow.memoization INFO: Task 4415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4415 try 0 launched on executor multi with executor id 4416 parsl.dataflow.dflow INFO: Standard output for task 4415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4017f72c-931c-42bb-95d3-73f341feda8c_isrForCrosstalkSources_855384_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4017f72c-931c-42bb-95d3-73f341feda8c_isrForCrosstalkSources_855384_8.stderr parsl.dataflow.dflow DEBUG: Task 4416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4416 has memoization hash 8af07ce41e6a22543a94e1b1345466dc parsl.dataflow.memoization INFO: Task 4416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4416 try 0 launched on executor multi with executor id 4417 parsl.dataflow.dflow INFO: Standard output for task 4416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/763de7f7-9a3b-4641-83a6-3ccd826bbe54_isrForCrosstalkSources_855394_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/763de7f7-9a3b-4641-83a6-3ccd826bbe54_isrForCrosstalkSources_855394_60.stderr parsl.dataflow.dflow DEBUG: Task 4417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4417 has memoization hash 157650aac1da472a188f35ed1acba636 parsl.dataflow.memoization INFO: Task 4417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4417 try 0 launched on executor multi with executor id 4418 parsl.dataflow.dflow INFO: Standard output for task 4417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b4127af3-f4b9-406a-803f-ed35a360ed06_isrForCrosstalkSources_855451_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b4127af3-f4b9-406a-803f-ed35a360ed06_isrForCrosstalkSources_855451_34.stderr parsl.dataflow.dflow DEBUG: Task 4418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4418 has memoization hash eae4c250589b6eaa3eefd40310db0389 parsl.dataflow.memoization INFO: Task 4418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4418 try 0 launched on executor multi with executor id 4419 parsl.dataflow.dflow INFO: Standard output for task 4418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bc40a678-c62a-4593-bbdd-8188bf6cc3c7_isrForCrosstalkSources_855436_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bc40a678-c62a-4593-bbdd-8188bf6cc3c7_isrForCrosstalkSources_855436_30.stderr parsl.dataflow.dflow DEBUG: Task 4419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies 2024-03-09 14:51:50 proc_lsst.multi:170 [INFO] [multi] job_id 16964192 proc_lsst.multi INFO: [multi] job_id 16964192 2024-03-09 14:51:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-09 14:51:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964192 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964192 parsl.dataflow.dflow INFO: Task 4419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Launched block 166 on executor multi with job ID 16964192 parsl.dataflow.dflow DEBUG: Task 4419 set to pending state with AppFuture: parsl.executors.status_handling INFO: Allocated block ID 167 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-167 2024-03-09 14:51:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-09 14:51:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Task 4419 has memoization hash 8bb141d4ba10020b59a8d21a50546873 parsl.dataflow.memoization INFO: Task 4419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81aa09f8c93627c81995c2c50448e2ed with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c0acadf4-497d-41ff-a992-e8bc83427770_isrForCrosstalkSources_855399_6.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c0acadf4-497d-41ff-a992-e8bc83427770_isrForCrosstalkSources_855399_6.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4de3acd361dbcce3058416d1ddd111fa with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/74b480ce-26bc-4fa7-b2e0-a0d60a3e8a28_isrForCrosstalkSources_855452_7.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/74b480ce-26bc-4fa7-b2e0-a0d60a3e8a28_isrForCrosstalkSources_855452_7.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7003d70f5c9a65592682ee0c071b02f with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/72e6ea9d-75e9-4c12-9db9-a60236e226e8_isrForCrosstalkSources_855384_51.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/72e6ea9d-75e9-4c12-9db9-a60236e226e8_isrForCrosstalkSources_855384_51.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ccd8358dda532b40f6a44d60c137b06 with result from task 296 parsl.dataflow.dflow INFO: Parsl task 4419 try 0 launched on executor multi with executor id 4420 parsl.dataflow.dflow INFO: Standard output for task 4419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e5682352-1ad8-4ee4-bc1e-08578ae07939_isrForCrosstalkSources_855455_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e5682352-1ad8-4ee4-bc1e-08578ae07939_isrForCrosstalkSources_855455_8.stderr parsl.dataflow.dflow DEBUG: Task 4420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4420 has memoization hash 625aecfbbafe6de46319d975291a6c4d parsl.dataflow.memoization INFO: Task 4420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a60fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/aceb60a0-40b5-424e-b3b7-bbae4694f4fc_isrForCrosstalkSources_855443_8.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/aceb60a0-40b5-424e-b3b7-bbae4694f4fc_isrForCrosstalkSources_855443_8.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57bcc044aeb63a86a02e65afca5c851d with result from task 300 parsl.dataflow.dflow INFO: Parsl task 4420 try 0 launched on executor multi with executor id 4421 parsl.dataflow.dflow INFO: Standard output for task 4420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/cdc3b09a-2c1a-49d8-bf6b-8b5134b921a2_isrForCrosstalkSources_855451_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/cdc3b09a-2c1a-49d8-bf6b-8b5134b921a2_isrForCrosstalkSources_855451_19.stderr parsl.dataflow.dflow DEBUG: Task 4421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4421 has memoization hash d853315941d709e1158ce4585b8e72bd parsl.dataflow.memoization INFO: Task 4421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/eee6febc-9114-4cc1-be4f-3d2afda0215d_isrForCrosstalkSources_855430_38.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/eee6febc-9114-4cc1-be4f-3d2afda0215d_isrForCrosstalkSources_855430_38.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 081c49cc880d3ca4a81b028f240b8213 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/a3946805-3bb5-4b82-bf46-6c0113e5ab4b_isrForCrosstalkSources_855417_50.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/a3946805-3bb5-4b82-bf46-6c0113e5ab4b_isrForCrosstalkSources_855417_50.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb9f9d1e749556e39520849d349cdf40 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/64f33b78-2366-47b5-8b7b-f9eeac3bc9b1_isrForCrosstalkSources_855454_32.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/64f33b78-2366-47b5-8b7b-f9eeac3bc9b1_isrForCrosstalkSources_855454_32.stderr parsl.dataflow.dflow INFO: Parsl task 4421 try 0 launched on executor multi with executor id 4422 parsl.dataflow.dflow INFO: Standard output for task 4421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/dccd7e6c-766a-449e-bd9c-c5ad72854b75_isrForCrosstalkSources_855379_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/dccd7e6c-766a-449e-bd9c-c5ad72854b75_isrForCrosstalkSources_855379_1.stderr parsl.dataflow.dflow DEBUG: Task 4422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4422 has memoization hash fca9f3174de1d5cdaa8028b3a59b96aa parsl.dataflow.memoization INFO: Task 4422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4ab5006ba370cdbc00fdecf641daadb with result from task 287 parsl.dataflow.dflow INFO: Parsl task 4422 try 0 launched on executor multi with executor id 4423 parsl.dataflow.dflow INFO: Standard output for task 4422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1b42e89f-c7bb-4c4b-a678-6dc024310e4d_isrForCrosstalkSources_855399_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1b42e89f-c7bb-4c4b-a678-6dc024310e4d_isrForCrosstalkSources_855399_37.stderr parsl.dataflow.dflow DEBUG: Task 4423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4423 has memoization hash 6fd9329e11273c339b1a3e4c3efa05f0 parsl.dataflow.memoization INFO: Task 4423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/48953295-bdb2-4e40-8cab-0c60644a9554_isrForCrosstalkSources_855435_48.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/48953295-bdb2-4e40-8cab-0c60644a9554_isrForCrosstalkSources_855435_48.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c352ee5ad22e4a3474337660275d453c with result from task 299 parsl.dataflow.dflow INFO: Parsl task 4423 try 0 launched on executor multi with executor id 4424 parsl.dataflow.dflow INFO: Standard output for task 4423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e016955a-906c-4e1b-894a-088f835dffa5_isrForCrosstalkSources_855413_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e016955a-906c-4e1b-894a-088f835dffa5_isrForCrosstalkSources_855413_26.stderr parsl.dataflow.dflow DEBUG: Task 4424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4424 has memoization hash 235f3c56e2403d04093ce76768f5325a parsl.dataflow.memoization INFO: Task 4424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1beec3e7-7172-4b5c-a776-900fa2f8acb6_isrForCrosstalkSources_855442_38.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1beec3e7-7172-4b5c-a776-900fa2f8acb6_isrForCrosstalkSources_855442_38.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3ce7d2c45f6db38afcef4a9c7b68f04 with result from task 268 parsl.dataflow.dflow INFO: Parsl task 4424 try 0 launched on executor multi with executor id 4425 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Standard output for task 4424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8cc45309-f079-4483-ac33-dd64d08d4c7d_isrForCrosstalkSources_855423_43.stdout parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard error for task 4424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8cc45309-f079-4483-ac33-dd64d08d4c7d_isrForCrosstalkSources_855423_43.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f14284f8-8630-4d86-8816-30c4393a1a76_isrForCrosstalkSources_855381_39.stdout parsl.dataflow.dflow DEBUG: Task 4425 will be sent to executor multi parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f14284f8-8630-4d86-8816-30c4393a1a76_isrForCrosstalkSources_855381_39.stderr parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Task 4425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.memoization DEBUG: Storing app cache entry 955653f72d87415974831d36980c27e6 with result from task 269 parsl.dataflow.dflow DEBUG: Task 4425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Task 4425 has memoization hash 9abe4f48b58329d30ca277f7965236aa parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dc0b69b8-bba5-449d-8991-bb80fa7161db_isrForCrosstalkSources_855447_39.stdout parsl.dataflow.memoization INFO: Task 4425 had no result in cache parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dc0b69b8-bba5-449d-8991-bb80fa7161db_isrForCrosstalkSources_855447_39.stderr parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6630bbfb01314696996f28604389174 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a47b955-1676-47b4-b2d1-a9303368c958_isrForCrosstalkSources_855425_13.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a47b955-1676-47b4-b2d1-a9303368c958_isrForCrosstalkSources_855425_13.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 427a46e58184dcf0cb3feda67b9fc2e5 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/060ef822-6bb3-4c97-bff2-8c4b33352262_isrForCrosstalkSources_855413_45.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/060ef822-6bb3-4c97-bff2-8c4b33352262_isrForCrosstalkSources_855413_45.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e4ede53bb3b4f925217848b4c4a6702 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9eb8d864-b490-4f61-929f-549eef900be5_isrForCrosstalkSources_855423_4.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9eb8d864-b490-4f61-929f-549eef900be5_isrForCrosstalkSources_855423_4.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89aa57d617d2c16e30a122a9b6dee7ca with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/085d84ee-6df1-4a03-98f4-d9596201b736_isrForCrosstalkSources_855391_34.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/085d84ee-6df1-4a03-98f4-d9596201b736_isrForCrosstalkSources_855391_34.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c91f15103d3f8c25121c7f7759a759 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0d494811-b4d8-4e66-9a46-d3d13c04143b_isrForCrosstalkSources_855451_33.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0d494811-b4d8-4e66-9a46-d3d13c04143b_isrForCrosstalkSources_855451_33.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bedb98c057c7c392e4eeae7e85c88289 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/34eecb6e-f21c-4bc9-b2cf-3e8ad526a8d0_isrForCrosstalkSources_855380_52.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/34eecb6e-f21c-4bc9-b2cf-3e8ad526a8d0_isrForCrosstalkSources_855380_52.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bc7a01f21a7149d158a701df151f5ba with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5a7ade19-74de-4c0a-b187-85de3a59272b_isrForCrosstalkSources_855449_57.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5a7ade19-74de-4c0a-b187-85de3a59272b_isrForCrosstalkSources_855449_57.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2029458be2114a30436cf0dac9076844 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a530cf7-984d-4135-afbd-51c35189fdee_isrForCrosstalkSources_855393_2.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a530cf7-984d-4135-afbd-51c35189fdee_isrForCrosstalkSources_855393_2.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa65d51c9a6115f8433bc1164df842f7 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5015968a-c083-4106-804d-5d316be1febf_isrForCrosstalkSources_855397_38.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5015968a-c083-4106-804d-5d316be1febf_isrForCrosstalkSources_855397_38.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe87e2eff1562c016a8c4a3acd1b325 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f4c01a5c-745e-4933-a021-826db741bd2b_isrForCrosstalkSources_855439_27.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f4c01a5c-745e-4933-a021-826db741bd2b_isrForCrosstalkSources_855439_27.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feb756b438752c33e6819125ac52dbbd with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e24d56e0-09c2-4f9a-ab42-cdd20bb78a00_isrForCrosstalkSources_855391_56.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e24d56e0-09c2-4f9a-ab42-cdd20bb78a00_isrForCrosstalkSources_855391_56.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 459e16b891afa56557ebd128d8fc67a5 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7b325447-964a-47a6-ab13-69109dcac9cd_isrForCrosstalkSources_855453_11.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7b325447-964a-47a6-ab13-69109dcac9cd_isrForCrosstalkSources_855453_11.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a02dd1e7868134932bbf2865677dec with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bc4d8770-bb2a-4fb5-9040-ec4e67d8894c_isrForCrosstalkSources_855458_22.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bc4d8770-bb2a-4fb5-9040-ec4e67d8894c_isrForCrosstalkSources_855458_22.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4d1a5475e31eb1446b96d38a76989a1 with result from task 446 parsl.dataflow.dflow INFO: Parsl task 4425 try 0 launched on executor multi with executor id 4426 parsl.dataflow.dflow INFO: Standard output for task 4425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/684e13f8-0370-4cea-8c9a-0737f0ba0955_isrForCrosstalkSources_855442_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/684e13f8-0370-4cea-8c9a-0737f0ba0955_isrForCrosstalkSources_855442_59.stderr parsl.dataflow.dflow DEBUG: Task 4426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4426 has memoization hash 63c4a81e5effd7f4e692b88fd2364acd parsl.dataflow.memoization INFO: Task 4426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4426 try 0 launched on executor multi with executor id 4427 parsl.dataflow.dflow INFO: Standard output for task 4426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1bd01b9a-96dd-46ac-bb93-a85bde693b5a_isrForCrosstalkSources_855459_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1bd01b9a-96dd-46ac-bb93-a85bde693b5a_isrForCrosstalkSources_855459_35.stderr parsl.dataflow.dflow DEBUG: Task 4427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4427 has memoization hash cbe320c5d31c804e76530820a6c10a6f parsl.dataflow.memoization INFO: Task 4427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5fa40abb-3ad6-41ee-a9e2-f1da7fd4fb81_isrForCrosstalkSources_855456_19.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5fa40abb-3ad6-41ee-a9e2-f1da7fd4fb81_isrForCrosstalkSources_855456_19.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2194659c97ee49fd308da982731ede4 with result from task 267 parsl.dataflow.dflow INFO: Parsl task 4427 try 0 launched on executor multi with executor id 4428 parsl.dataflow.dflow INFO: Standard output for task 4427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6a8c3c2c-a8b3-44c9-9d2e-4ce98b12ca60_isrForCrosstalkSources_855456_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6a8c3c2c-a8b3-44c9-9d2e-4ce98b12ca60_isrForCrosstalkSources_855456_49.stderr parsl.dataflow.dflow DEBUG: Task 4428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4428 has memoization hash 0ebcad6dc582b5e84cafba170d2542f8 parsl.dataflow.memoization INFO: Task 4428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/137f03c5-ce71-41a1-a78d-6723b06779cd_isrForCrosstalkSources_855418_5.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/137f03c5-ce71-41a1-a78d-6723b06779cd_isrForCrosstalkSources_855418_5.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b26484a10e3486caf541814daa047a0d with result from task 282 parsl.dataflow.dflow INFO: Parsl task 4428 try 0 launched on executor multi with executor id 4429 parsl.dataflow.dflow INFO: Standard output for task 4428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/43cd52a0-13f3-459a-b649-9b44ee95c3e4_isrForCrosstalkSources_855398_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/43cd52a0-13f3-459a-b649-9b44ee95c3e4_isrForCrosstalkSources_855398_39.stderr parsl.dataflow.dflow DEBUG: Task 4429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4429 has memoization hash ce0d92a3eef8bde4954f5958026ee473 parsl.dataflow.memoization INFO: Task 4429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/aa969765-d7d5-4f02-bcfd-8eab2fb98ef1_isrForCrosstalkSources_855391_13.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/aa969765-d7d5-4f02-bcfd-8eab2fb98ef1_isrForCrosstalkSources_855391_13.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0692dad9289f650bea5c5ecd8a804fd with result from task 288 parsl.dataflow.dflow INFO: Parsl task 4429 try 0 launched on executor multi with executor id 4430 parsl.dataflow.dflow INFO: Standard output for task 4429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c8209a57-ef40-4de5-be89-a35638a88170_isrForCrosstalkSources_855400_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c8209a57-ef40-4de5-be89-a35638a88170_isrForCrosstalkSources_855400_40.stderr parsl.dataflow.dflow DEBUG: Task 4430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4430 has memoization hash 38e635cdf22ebe3086119508915aed12 parsl.dataflow.memoization INFO: Task 4430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/e2a0305c-3514-4973-bf1a-47026e94bb99_isrForCrosstalkSources_855452_6.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/e2a0305c-3514-4973-bf1a-47026e94bb99_isrForCrosstalkSources_855452_6.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0958a97cce6f8479ea5ede8f4fdd2d83 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fa180342-ae9d-4aa2-a184-c907040f31db_isrForCrosstalkSources_855438_37.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fa180342-ae9d-4aa2-a184-c907040f31db_isrForCrosstalkSources_855438_37.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4877de6b68fdcda28e293daf13e78f30 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e4a5d1b3-2875-40f9-93a5-7dd8cacdfed8_isrForCrosstalkSources_855398_33.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e4a5d1b3-2875-40f9-93a5-7dd8cacdfed8_isrForCrosstalkSources_855398_33.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8feefbb2810c6978c37cd0cde83699b4 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/5d5a616a-2dba-4065-980e-4af35f1707e3_isrForCrosstalkSources_855438_8.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/5d5a616a-2dba-4065-980e-4af35f1707e3_isrForCrosstalkSources_855438_8.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09d699e546957b1792080b1cea70b48b with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/732268b6-6607-48da-8ec0-35740f9ac431_isrForCrosstalkSources_855429_19.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/732268b6-6607-48da-8ec0-35740f9ac431_isrForCrosstalkSources_855429_19.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4afc1a0a14518eda4f4798f7c9302480 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6eb8134e-d231-4227-992b-baf1ac987625_isrForCrosstalkSources_855453_5.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6eb8134e-d231-4227-992b-baf1ac987625_isrForCrosstalkSources_855453_5.stderr parsl.dataflow.dflow INFO: Parsl task 4430 try 0 launched on executor multi with executor id 4431 parsl.dataflow.dflow INFO: Standard output for task 4430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/79f9398d-762c-49f1-82a0-187dba53a126_isrForCrosstalkSources_855456_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/79f9398d-762c-49f1-82a0-187dba53a126_isrForCrosstalkSources_855456_51.stderr parsl.dataflow.dflow DEBUG: Task 4431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4431 has memoization hash d1549330d0b20e12d600aeb4e3bbae55 parsl.dataflow.memoization INFO: Task 4431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4431 try 0 launched on executor multi with executor id 4432 parsl.dataflow.dflow INFO: Standard output for task 4431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a2fcbb1-964d-4f30-ac1a-1066b33f7747_isrForCrosstalkSources_855379_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a2fcbb1-964d-4f30-ac1a-1066b33f7747_isrForCrosstalkSources_855379_6.stderr parsl.dataflow.dflow DEBUG: Task 4432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4432 has memoization hash 574bf2238b67b9b31572dcca75eab076 parsl.dataflow.memoization INFO: Task 4432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4432 try 0 launched on executor multi with executor id 4433 parsl.dataflow.dflow INFO: Standard output for task 4432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/2ee420dd-8d3d-48f5-8047-0615a201292c_isrForCrosstalkSources_855430_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/2ee420dd-8d3d-48f5-8047-0615a201292c_isrForCrosstalkSources_855430_15.stderr parsl.dataflow.dflow DEBUG: Task 4433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4433 has memoization hash 9af54a221ad9e946e90d798b5060ef90 parsl.dataflow.memoization INFO: Task 4433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4433 try 0 launched on executor multi with executor id 4434 parsl.dataflow.dflow INFO: Standard output for task 4433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7c9809a2-0d40-4e39-ba39-de55d191ff02_isrForCrosstalkSources_855453_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7c9809a2-0d40-4e39-ba39-de55d191ff02_isrForCrosstalkSources_855453_22.stderr parsl.dataflow.dflow DEBUG: Task 4434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4434 has memoization hash 8b059be8e5e647d1a7c20281efd8e91f parsl.dataflow.memoization INFO: Task 4434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4434 try 0 launched on executor multi with executor id 4435 parsl.dataflow.dflow INFO: Standard output for task 4434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/002a4cb0-e615-4c63-8a84-e8dbf1b70dd7_isrForCrosstalkSources_855418_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/002a4cb0-e615-4c63-8a84-e8dbf1b70dd7_isrForCrosstalkSources_855418_10.stderr parsl.dataflow.dflow DEBUG: Task 4435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4435 has memoization hash 2d5e5899a4b7d9385ab4711f57938377 parsl.dataflow.memoization INFO: Task 4435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4435 try 0 launched on executor multi with executor id 4436 parsl.dataflow.dflow INFO: Standard output for task 4435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d70a2e-149e-444b-9602-5168ed804e78_isrForCrosstalkSources_855426_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d70a2e-149e-444b-9602-5168ed804e78_isrForCrosstalkSources_855426_57.stderr parsl.dataflow.dflow DEBUG: Task 4436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4436 has memoization hash b481ffc588974150a7c49a57982e7e9a parsl.dataflow.memoization INFO: Task 4436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4436 try 0 launched on executor multi with executor id 4437 parsl.dataflow.dflow INFO: Standard output for task 4436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/db9ff831-3340-4348-8786-53c56050d689_isrForCrosstalkSources_855457_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/db9ff831-3340-4348-8786-53c56050d689_isrForCrosstalkSources_855457_61.stderr parsl.dataflow.dflow DEBUG: Task 4437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4437 has memoization hash e5c02922c6489407472d1d20977e289c parsl.dataflow.memoization INFO: Task 4437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4437 try 0 launched on executor multi with executor id 4438 parsl.dataflow.dflow INFO: Standard output for task 4437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/36abda30-4635-4ffa-a893-7b529fd1f123_isrForCrosstalkSources_855383_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/36abda30-4635-4ffa-a893-7b529fd1f123_isrForCrosstalkSources_855383_36.stderr parsl.dataflow.dflow DEBUG: Task 4438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4438 has memoization hash 168672c8010d1021b4554c9a1ec9699b parsl.dataflow.memoization INFO: Task 4438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4438 try 0 launched on executor multi with executor id 4439 parsl.dataflow.dflow INFO: Standard output for task 4438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/7e3d3aad-7bd6-4374-b37f-beb02cd2d543_isrForCrosstalkSources_855425_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/7e3d3aad-7bd6-4374-b37f-beb02cd2d543_isrForCrosstalkSources_855425_57.stderr parsl.dataflow.dflow DEBUG: Task 4439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4439 has memoization hash 6083b5c2bcc3fbd130545762f602f5c2 parsl.dataflow.memoization INFO: Task 4439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5862c5803815e301478befe72b0b4fc with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c54733a9-ee71-4bb8-b9bd-4f3c20d6a9b2_isrForCrosstalkSources_855443_6.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c54733a9-ee71-4bb8-b9bd-4f3c20d6a9b2_isrForCrosstalkSources_855443_6.stderr parsl.dataflow.dflow INFO: Parsl task 4439 try 0 launched on executor multi with executor id 4440 parsl.dataflow.dflow INFO: Standard output for task 4439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ec17e1d7-4c7c-4593-bcf2-561fb3d4c432_isrForCrosstalkSources_855403_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ec17e1d7-4c7c-4593-bcf2-561fb3d4c432_isrForCrosstalkSources_855403_47.stderr parsl.dataflow.dflow DEBUG: Task 4440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4440 has memoization hash 45bbf2082d1b98b16a3fe34f4cdcf6c0 parsl.dataflow.memoization INFO: Task 4440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:51 proc_lsst.multi:170 [INFO] [multi] job_id 16964193 proc_lsst.multi INFO: [multi] job_id 16964193 2024-03-09 14:51:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-09 14:51:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964193 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964193 parsl.executors.status_handling DEBUG: Launched block 167 on executor multi with job ID 16964193 parsl.executors.status_handling INFO: Allocated block ID 168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-168 2024-03-09 14:51:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 2024-03-09 14:51:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-09 14:51:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4440 try 0 launched on executor multi with executor id 4441 parsl.dataflow.dflow INFO: Standard output for task 4440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b5ff75ff-bbbe-4034-84e4-6760b6066f3c_isrForCrosstalkSources_855446_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b5ff75ff-bbbe-4034-84e4-6760b6066f3c_isrForCrosstalkSources_855446_3.stderr parsl.dataflow.dflow DEBUG: Task 4441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4441 has memoization hash e6a052bf8e7e3e28db0aa0bc42c1fd55 parsl.dataflow.memoization INFO: Task 4441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4441 try 0 launched on executor multi with executor id 4442 parsl.dataflow.dflow INFO: Standard output for task 4441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4a4b74fd-ee3e-4e0e-b669-435d838e8091_isrForCrosstalkSources_855433_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4a4b74fd-ee3e-4e0e-b669-435d838e8091_isrForCrosstalkSources_855433_16.stderr parsl.dataflow.dflow DEBUG: Task 4442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4442 has memoization hash 41e9e27187cefc797e5a957a06740c6e parsl.dataflow.memoization INFO: Task 4442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4442 try 0 launched on executor multi with executor id 4443 parsl.dataflow.dflow INFO: Standard output for task 4442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/94288910-0a97-4915-901a-c940fda1a86a_isrForCrosstalkSources_855389_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/94288910-0a97-4915-901a-c940fda1a86a_isrForCrosstalkSources_855389_19.stderr parsl.dataflow.dflow DEBUG: Task 4443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4443 has memoization hash ccc4ef244e3cb292990e0709acfbb971 parsl.dataflow.memoization INFO: Task 4443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4443 try 0 launched on executor multi with executor id 4444 parsl.dataflow.dflow INFO: Standard output for task 4443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/59ffcacf-5b23-4747-b2a0-f41a7dfc36ea_isrForCrosstalkSources_855458_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/59ffcacf-5b23-4747-b2a0-f41a7dfc36ea_isrForCrosstalkSources_855458_31.stderr parsl.dataflow.dflow DEBUG: Task 4444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4444 has memoization hash 4d75c8fb3d23acc1c908b33c022fcbcd parsl.dataflow.memoization INFO: Task 4444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4444 try 0 launched on executor multi with executor id 4445 parsl.dataflow.dflow INFO: Standard output for task 4444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ace41b0-9a4d-4563-9770-a75fa37d4620_isrForCrosstalkSources_855429_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ace41b0-9a4d-4563-9770-a75fa37d4620_isrForCrosstalkSources_855429_14.stderr parsl.dataflow.dflow DEBUG: Task 4445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4445 has memoization hash 8fbacfe7d532e1fbdd365b90bac207c2 parsl.dataflow.memoization INFO: Task 4445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4445 try 0 launched on executor multi with executor id 4446 parsl.dataflow.dflow INFO: Standard output for task 4445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b868ba9e-24e7-478c-817e-99fb5dd4b4bc_isrForCrosstalkSources_855415_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b868ba9e-24e7-478c-817e-99fb5dd4b4bc_isrForCrosstalkSources_855415_16.stderr parsl.dataflow.dflow DEBUG: Task 4446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4446 has memoization hash 7ea5544ffbf3c27f29abd5391fb70337 parsl.dataflow.memoization INFO: Task 4446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4446 try 0 launched on executor multi with executor id 4447 parsl.dataflow.dflow INFO: Standard output for task 4446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b444e26-211d-4c34-aff5-8cc3fee68fb2_isrForCrosstalkSources_855447_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b444e26-211d-4c34-aff5-8cc3fee68fb2_isrForCrosstalkSources_855447_32.stderr parsl.dataflow.dflow DEBUG: Task 4447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4447 has memoization hash ba93d5d4aee2402d859367f7f432ff33 parsl.dataflow.memoization INFO: Task 4447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4447 try 0 launched on executor multi with executor id 4448 parsl.dataflow.dflow INFO: Standard output for task 4447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5e867190-e099-4fe5-8e87-b3426e78f217_isrForCrosstalkSources_855391_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5e867190-e099-4fe5-8e87-b3426e78f217_isrForCrosstalkSources_855391_38.stderr parsl.dataflow.dflow DEBUG: Task 4448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4448 has memoization hash 49241e116990e177c5e6d3bfb403478d parsl.dataflow.memoization INFO: Task 4448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4448 try 0 launched on executor multi with executor id 4449 parsl.dataflow.dflow INFO: Standard output for task 4448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bc9ec2c9-54c7-47eb-a645-3acc633ab1d0_isrForCrosstalkSources_855426_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bc9ec2c9-54c7-47eb-a645-3acc633ab1d0_isrForCrosstalkSources_855426_45.stderr parsl.dataflow.dflow DEBUG: Task 4449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4449 has memoization hash 4a215f900bec2bf6827b579d4032b267 parsl.dataflow.memoization INFO: Task 4449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4449 try 0 launched on executor multi with executor id 4450 parsl.dataflow.dflow INFO: Standard output for task 4449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ea2e2923-f1f3-4c12-8840-bde6c5a69092_isrForCrosstalkSources_855397_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ea2e2923-f1f3-4c12-8840-bde6c5a69092_isrForCrosstalkSources_855397_45.stderr parsl.dataflow.dflow DEBUG: Task 4450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4450 has memoization hash ddfa42fa2d6d0941646c69632b24f0a4 parsl.dataflow.memoization INFO: Task 4450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4450 try 0 launched on executor multi with executor id 4451 parsl.dataflow.dflow INFO: Standard output for task 4450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/064bfd6b-aecd-41f7-bf67-ca7e708861a5_isrForCrosstalkSources_855422_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/064bfd6b-aecd-41f7-bf67-ca7e708861a5_isrForCrosstalkSources_855422_38.stderr parsl.dataflow.dflow DEBUG: Task 4451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4451 has memoization hash f528ce6982f538117aa89db9f13ec6cb parsl.dataflow.memoization INFO: Task 4451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05abb6838ccf7dbcca886768f245ff1e with result from task 603 parsl.dataflow.dflow INFO: Parsl task 4451 try 0 launched on executor multi with executor id 4452 parsl.dataflow.dflow INFO: Standard output for task 4451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3cecf6ef-e8e2-4bda-8059-25af86f61286_isrForCrosstalkSources_855432_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3cecf6ef-e8e2-4bda-8059-25af86f61286_isrForCrosstalkSources_855432_3.stderr parsl.dataflow.dflow DEBUG: Task 4452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4452 has memoization hash 6eb6aede96b4462d5ddc18c45b7d95ea parsl.dataflow.memoization INFO: Task 4452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/74c9a681-9062-42bf-8ef4-6491132ef8dc_isrForCrosstalkSources_855416_37.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/74c9a681-9062-42bf-8ef4-6491132ef8dc_isrForCrosstalkSources_855416_37.stderr parsl.dataflow.dflow INFO: Parsl task 4452 try 0 launched on executor multi with executor id 4453 parsl.dataflow.dflow INFO: Standard output for task 4452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/7e2883a7-4bbe-4932-90a4-4e830790a7b8_isrForCrosstalkSources_855430_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/7e2883a7-4bbe-4932-90a4-4e830790a7b8_isrForCrosstalkSources_855430_60.stderr parsl.dataflow.dflow DEBUG: Task 4453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4453 has memoization hash 04c6d143d13ed2a05d64788be9c95a46 parsl.dataflow.memoization INFO: Task 4453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4453 try 0 launched on executor multi with executor id 4454 parsl.dataflow.dflow INFO: Standard output for task 4453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/9a622bef-6fd7-4e59-8f44-e79d2169fe9e_isrForCrosstalkSources_855455_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/9a622bef-6fd7-4e59-8f44-e79d2169fe9e_isrForCrosstalkSources_855455_20.stderr parsl.dataflow.dflow DEBUG: Task 4454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4454 has memoization hash c105598ce7d3052aaae337e76382b84c parsl.dataflow.memoization INFO: Task 4454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4454 try 0 launched on executor multi with executor id 4455 parsl.dataflow.dflow INFO: Standard output for task 4454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fa9852de-f4ae-4023-8762-8c9713a27039_isrForCrosstalkSources_855425_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fa9852de-f4ae-4023-8762-8c9713a27039_isrForCrosstalkSources_855425_36.stderr parsl.dataflow.dflow DEBUG: Task 4455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4455 has memoization hash 41c9df3e65300a2f8b85d434eed8014b parsl.dataflow.memoization INFO: Task 4455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4455 try 0 launched on executor multi with executor id 4456 parsl.dataflow.dflow INFO: Standard output for task 4455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7730ad6f-78f8-43e6-9e21-cb8b0c7b5247_isrForCrosstalkSources_855446_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7730ad6f-78f8-43e6-9e21-cb8b0c7b5247_isrForCrosstalkSources_855446_61.stderr parsl.dataflow.dflow DEBUG: Task 4456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4456 has memoization hash 7e29cf5c7fbe1bd6c5016b6c406f909a parsl.dataflow.memoization INFO: Task 4456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4456 try 0 launched on executor multi with executor id 4457 parsl.dataflow.dflow INFO: Standard output for task 4456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc5c6add-5038-4f63-a7cf-b6946a30764e_isrForCrosstalkSources_855450_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc5c6add-5038-4f63-a7cf-b6946a30764e_isrForCrosstalkSources_855450_37.stderr parsl.dataflow.dflow DEBUG: Task 4457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4457 has memoization hash 7f8e2b400311ea0b8ce5e3cf0a0f031c parsl.dataflow.memoization INFO: Task 4457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4457 try 0 launched on executor multi with executor id 4458 parsl.dataflow.dflow INFO: Standard output for task 4457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e3faf19-b4c5-432a-8788-8f38fe27d848_isrForCrosstalkSources_855429_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e3faf19-b4c5-432a-8788-8f38fe27d848_isrForCrosstalkSources_855429_8.stderr parsl.dataflow.dflow DEBUG: Task 4458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4458 has memoization hash e7527f1d7d694d8520d3bc58a74fa05e parsl.dataflow.memoization INFO: Task 4458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4458 try 0 launched on executor multi with executor id 4459 parsl.dataflow.dflow INFO: Standard output for task 4458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b854d41-fd2d-4b61-98fd-49038971167b_isrForCrosstalkSources_855459_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b854d41-fd2d-4b61-98fd-49038971167b_isrForCrosstalkSources_855459_4.stderr parsl.dataflow.dflow DEBUG: Task 4459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4459 has memoization hash b89c3a152d0d07ff065a287eedd2ab87 parsl.dataflow.memoization INFO: Task 4459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aefe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4459 try 0 launched on executor multi with executor id 4460 parsl.dataflow.dflow INFO: Standard output for task 4459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/8facd480-b791-4113-b0ec-0291825fe8e4_isrForCrosstalkSources_855437_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/8facd480-b791-4113-b0ec-0291825fe8e4_isrForCrosstalkSources_855437_55.stderr parsl.dataflow.dflow DEBUG: Task 4460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4460 has memoization hash 0a3f3a329486062e1402629d354e5dbf parsl.dataflow.memoization INFO: Task 4460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aeff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4460 try 0 launched on executor multi with executor id 4461 parsl.dataflow.dflow INFO: Standard output for task 4460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73a28890-b65f-429e-be36-190d29235c92_isrForCrosstalkSources_855444_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73a28890-b65f-429e-be36-190d29235c92_isrForCrosstalkSources_855444_40.stderr parsl.dataflow.dflow DEBUG: Task 4461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4461 has memoization hash f291b0f3ad41cb5366c990d46ebde57f parsl.dataflow.memoization INFO: Task 4461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4461 try 0 launched on executor multi with executor id 4462 parsl.dataflow.dflow INFO: Standard output for task 4461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/448fc953-6bd2-457b-a495-3c502d7345b9_isrForCrosstalkSources_855426_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/448fc953-6bd2-457b-a495-3c502d7345b9_isrForCrosstalkSources_855426_52.stderr parsl.dataflow.dflow DEBUG: Task 4462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4462 has memoization hash ea88e04bd949e94d411a3141dc79d50f parsl.dataflow.memoization INFO: Task 4462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4462 try 0 launched on executor multi with executor id 4463 parsl.dataflow.dflow INFO: Standard output for task 4462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6a98b60d-dee1-4a41-ad76-c92a59979f81_isrForCrosstalkSources_855429_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6a98b60d-dee1-4a41-ad76-c92a59979f81_isrForCrosstalkSources_855429_3.stderr parsl.dataflow.dflow DEBUG: Task 4463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4463 has memoization hash dda6c9df288a2baf0ad6faa1169a2500 parsl.dataflow.memoization INFO: Task 4463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4463 try 0 launched on executor multi with executor id 4464 parsl.dataflow.dflow INFO: Standard output for task 4463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ed4f14de-d393-4282-8c52-5d6be4fddd5e_isrForCrosstalkSources_855429_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ed4f14de-d393-4282-8c52-5d6be4fddd5e_isrForCrosstalkSources_855429_55.stderr parsl.dataflow.dflow DEBUG: Task 4464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4464 has memoization hash 8e2f2ca9dc00ab4cdea76806cc0d6239 parsl.dataflow.memoization INFO: Task 4464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4464 try 0 launched on executor multi with executor id 4465 parsl.dataflow.dflow INFO: Standard output for task 4464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8f247bcc-d81b-4552-a96d-40ee46084706_isrForCrosstalkSources_855404_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8f247bcc-d81b-4552-a96d-40ee46084706_isrForCrosstalkSources_855404_40.stderr parsl.dataflow.dflow DEBUG: Task 4465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4465 has memoization hash c25c7be26eb062e8124cd471d78d6266 parsl.dataflow.memoization INFO: Task 4465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4465 try 0 launched on executor multi with executor id 4466 parsl.dataflow.dflow INFO: Standard output for task 4465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fc4565d0-62d7-4ad1-8b26-5881202c1b04_isrForCrosstalkSources_855450_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fc4565d0-62d7-4ad1-8b26-5881202c1b04_isrForCrosstalkSources_855450_50.stderr parsl.dataflow.dflow DEBUG: Task 4466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4466 has memoization hash 471018880cbaf33acf90c412d15e0901 parsl.dataflow.memoization INFO: Task 4466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4466 try 0 launched on executor multi with executor id 4467 parsl.dataflow.dflow INFO: Standard output for task 4466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14113fc8-7660-43e6-80e7-90dfe46fd2ac_isrForCrosstalkSources_855442_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14113fc8-7660-43e6-80e7-90dfe46fd2ac_isrForCrosstalkSources_855442_32.stderr parsl.dataflow.dflow DEBUG: Task 4467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4467 has memoization hash 305bc049731a3fd553d36b0b792a0863 parsl.dataflow.memoization INFO: Task 4467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4467 try 0 launched on executor multi with executor id 4468 parsl.dataflow.dflow INFO: Standard output for task 4467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/33eca51f-c423-4fc7-94c4-8e59192ff3a8_isrForCrosstalkSources_855400_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/33eca51f-c423-4fc7-94c4-8e59192ff3a8_isrForCrosstalkSources_855400_61.stderr parsl.dataflow.dflow DEBUG: Task 4468 will be sent to executor multi 2024-03-09 14:51:52 proc_lsst.multi:170 [INFO] [multi] job_id 16964194 proc_lsst.multi INFO: [multi] job_id 16964194 2024-03-09 14:51:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-09 14:51:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964194 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964194 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling DEBUG: Launched block 168 on executor multi with job ID 16964194 parsl.dataflow.dflow INFO: Task 4468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling INFO: Allocated block ID 169 parsl.dataflow.dflow DEBUG: Task 4468 set to pending state with AppFuture: parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-169 2024-03-09 14:51:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4468 has memoization hash 67f6d27cc8b84f7a11079e876c561ed0 parsl.dataflow.memoization INFO: Task 4468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4468 try 0 launched on executor multi with executor id 4469 parsl.dataflow.dflow INFO: Standard output for task 4468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2df8a800-6bee-4bef-aec5-730fb69d127a_isrForCrosstalkSources_855456_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2df8a800-6bee-4bef-aec5-730fb69d127a_isrForCrosstalkSources_855456_3.stderr parsl.dataflow.dflow DEBUG: Task 4469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4469 has memoization hash ca8b5343086e105699d786f0542cdafa parsl.dataflow.memoization INFO: Task 4469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a61120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4469 try 0 launched on executor multi with executor id 4470 parsl.dataflow.dflow INFO: Standard output for task 4469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/06f791a9-fa7c-4400-9c34-e13d5460520a_isrForCrosstalkSources_855453_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/06f791a9-fa7c-4400-9c34-e13d5460520a_isrForCrosstalkSources_855453_23.stderr parsl.dataflow.dflow DEBUG: Task 4470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4470 has memoization hash eb96732c5f3c4d79c0ae85696724f8d1 parsl.dataflow.memoization INFO: Task 4470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a63380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4470 try 0 launched on executor multi with executor id 4471 parsl.dataflow.dflow INFO: Standard output for task 4470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/01bf9351-9e61-4559-8d42-6fe20690474f_isrForCrosstalkSources_855427_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/01bf9351-9e61-4559-8d42-6fe20690474f_isrForCrosstalkSources_855427_34.stderr parsl.dataflow.dflow DEBUG: Task 4471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4471 has memoization hash 733380dea5a8d8f2a535a6d87c079575 parsl.dataflow.memoization INFO: Task 4471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5a632e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4471 try 0 launched on executor multi with executor id 4472 parsl.dataflow.dflow INFO: Standard output for task 4471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a270fc16-02d1-4c8b-b724-96a953998d97_isrForCrosstalkSources_855382_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a270fc16-02d1-4c8b-b724-96a953998d97_isrForCrosstalkSources_855382_29.stderr parsl.dataflow.dflow DEBUG: Task 4472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4472 has memoization hash 27fec0bb3a8ce97c0324d2efc82dae7d parsl.dataflow.memoization INFO: Task 4472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4472 try 0 launched on executor multi with executor id 4473 parsl.dataflow.dflow INFO: Standard output for task 4472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/70061a86-416a-4240-b250-ce3f18d16000_isrForCrosstalkSources_855428_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/70061a86-416a-4240-b250-ce3f18d16000_isrForCrosstalkSources_855428_12.stderr parsl.dataflow.dflow DEBUG: Task 4473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4473 has memoization hash 7e2f60936f9def38263a6149fe7d9d80 parsl.dataflow.memoization INFO: Task 4473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4473 try 0 launched on executor multi with executor id 4474 parsl.dataflow.dflow INFO: Standard output for task 4473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/bf5168fa-a09d-4fc1-9706-e413d35ef3eb_isrForCrosstalkSources_855457_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/bf5168fa-a09d-4fc1-9706-e413d35ef3eb_isrForCrosstalkSources_855457_4.stderr parsl.dataflow.dflow DEBUG: Task 4474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4474 has memoization hash 9944fc4e5c2d05fe9a7a46a35205dc6c parsl.dataflow.memoization INFO: Task 4474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4474 try 0 launched on executor multi with executor id 4475 parsl.dataflow.dflow INFO: Standard output for task 4474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/06dba6f5-2d2a-4a88-b61d-0f8e62e89de2_isrForCrosstalkSources_855408_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/06dba6f5-2d2a-4a88-b61d-0f8e62e89de2_isrForCrosstalkSources_855408_29.stderr parsl.dataflow.dflow DEBUG: Task 4475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4475 has memoization hash 00bef76857175750574f5982d44353db parsl.dataflow.memoization INFO: Task 4475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4475 try 0 launched on executor multi with executor id 4476 parsl.dataflow.dflow INFO: Standard output for task 4475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/9f0fd6cc-15c2-4865-be11-1ced50bcaedd_isrForCrosstalkSources_855415_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/9f0fd6cc-15c2-4865-be11-1ced50bcaedd_isrForCrosstalkSources_855415_32.stderr parsl.dataflow.dflow DEBUG: Task 4476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4476 has memoization hash 174063b25bc4489da5fbfd2f48244d7b parsl.dataflow.memoization INFO: Task 4476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84c92ce2103ec37d058f7dfcdc31e926 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/41037bec-b935-44f4-8bbb-40325e6a668a_isrForCrosstalkSources_855430_34.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/41037bec-b935-44f4-8bbb-40325e6a668a_isrForCrosstalkSources_855430_34.stderr parsl.dataflow.dflow INFO: Parsl task 4476 try 0 launched on executor multi with executor id 4477 parsl.dataflow.dflow INFO: Standard output for task 4476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0bb67001-ff7b-42fe-86b2-562599b6da3d_isrForCrosstalkSources_855444_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0bb67001-ff7b-42fe-86b2-562599b6da3d_isrForCrosstalkSources_855444_6.stderr parsl.dataflow.dflow DEBUG: Task 4477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4477 has memoization hash bb3ee60c98249b3f86628b52b579e0cf parsl.dataflow.memoization INFO: Task 4477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4477 try 0 launched on executor multi with executor id 4478 parsl.dataflow.dflow INFO: Standard output for task 4477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca06a1de-5591-4005-94d4-c5a320c5f1d5_isrForCrosstalkSources_855400_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca06a1de-5591-4005-94d4-c5a320c5f1d5_isrForCrosstalkSources_855400_3.stderr parsl.dataflow.dflow DEBUG: Task 4478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4478 has memoization hash a2ca1459d719138872916d0b65f90b53 parsl.dataflow.memoization INFO: Task 4478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4478 try 0 launched on executor multi with executor id 4479 parsl.dataflow.dflow INFO: Standard output for task 4478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/162faa76-1de6-4ab5-acc1-5226d6571c85_isrForCrosstalkSources_855416_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/162faa76-1de6-4ab5-acc1-5226d6571c85_isrForCrosstalkSources_855416_7.stderr parsl.dataflow.dflow DEBUG: Task 4479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4479 has memoization hash 37934079d3d45837df3ef0542f795c1f parsl.dataflow.memoization INFO: Task 4479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4479 try 0 launched on executor multi with executor id 4480 parsl.dataflow.dflow INFO: Standard output for task 4479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/427325b0-8d01-45d8-b5e6-0fb9867e4367_isrForCrosstalkSources_855389_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/427325b0-8d01-45d8-b5e6-0fb9867e4367_isrForCrosstalkSources_855389_48.stderr parsl.dataflow.dflow DEBUG: Task 4480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4480 has memoization hash 2a4389e3825b8267e37d0a071089087a parsl.dataflow.memoization INFO: Task 4480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4480 try 0 launched on executor multi with executor id 4481 parsl.dataflow.dflow INFO: Standard output for task 4480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac8ba7bc-85a4-44b4-b953-76bb1cafcc77_isrForCrosstalkSources_855394_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac8ba7bc-85a4-44b4-b953-76bb1cafcc77_isrForCrosstalkSources_855394_37.stderr parsl.dataflow.dflow DEBUG: Task 4481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4481 has memoization hash c9371846c6d0339a8415acb3a034fffa parsl.dataflow.memoization INFO: Task 4481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4481 try 0 launched on executor multi with executor id 4482 parsl.dataflow.dflow INFO: Standard output for task 4481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/f07e2776-9c7d-468a-be8c-143392393dbb_isrForCrosstalkSources_855403_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/f07e2776-9c7d-468a-be8c-143392393dbb_isrForCrosstalkSources_855403_48.stderr parsl.dataflow.dflow DEBUG: Task 4482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4482 has memoization hash 05ca77d45af73edad21894d3eccb6234 parsl.dataflow.memoization INFO: Task 4482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4482 try 0 launched on executor multi with executor id 4483 parsl.dataflow.dflow INFO: Standard output for task 4482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e0abd165-1fee-44e0-8027-85d183b94b13_isrForCrosstalkSources_855400_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e0abd165-1fee-44e0-8027-85d183b94b13_isrForCrosstalkSources_855400_44.stderr parsl.dataflow.dflow DEBUG: Task 4483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4483 has memoization hash 15fa53643c4c51c1df26c349eee8fd51 parsl.dataflow.memoization INFO: Task 4483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4483 try 0 launched on executor multi with executor id 4484 parsl.dataflow.dflow INFO: Standard output for task 4483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/70adbb9e-200b-4af9-b408-02fe3d229a6d_isrForCrosstalkSources_855439_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/70adbb9e-200b-4af9-b408-02fe3d229a6d_isrForCrosstalkSources_855439_57.stderr parsl.dataflow.dflow DEBUG: Task 4484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4484 has memoization hash b6a51768b4c0f523300d5dc80c26a0e1 parsl.dataflow.memoization INFO: Task 4484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4484 try 0 launched on executor multi with executor id 4485 parsl.dataflow.dflow INFO: Standard output for task 4484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f4694f4a-d31c-4d5c-9e30-e42bd0a20736_isrForCrosstalkSources_855398_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f4694f4a-d31c-4d5c-9e30-e42bd0a20736_isrForCrosstalkSources_855398_44.stderr parsl.dataflow.dflow DEBUG: Task 4485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4485 has memoization hash c431942dc402d10041716c14a5a3434a parsl.dataflow.memoization INFO: Task 4485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4485 try 0 launched on executor multi with executor id 4486 parsl.dataflow.dflow INFO: Standard output for task 4485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/effa8f22-eb80-40ce-8746-77c9a6116ba9_isrForCrosstalkSources_855409_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/effa8f22-eb80-40ce-8746-77c9a6116ba9_isrForCrosstalkSources_855409_15.stderr parsl.dataflow.dflow DEBUG: Task 4486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4486 has memoization hash 7af0934870566dc6f1f62cfe7ff6dd97 parsl.dataflow.memoization INFO: Task 4486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4486 try 0 launched on executor multi with executor id 4487 parsl.dataflow.dflow INFO: Standard output for task 4486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dd2fc06b-d15a-4081-b9ab-d5b924376e19_isrForCrosstalkSources_855447_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dd2fc06b-d15a-4081-b9ab-d5b924376e19_isrForCrosstalkSources_855447_34.stderr parsl.dataflow.dflow DEBUG: Task 4487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4487 has memoization hash ec33f7955f5642d940d860600b5b1488 parsl.dataflow.memoization INFO: Task 4487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4487 try 0 launched on executor multi with executor id 4488 parsl.dataflow.dflow INFO: Standard output for task 4487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b646b5c3-1777-45f5-b117-f2208b9b428a_isrForCrosstalkSources_855453_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b646b5c3-1777-45f5-b117-f2208b9b428a_isrForCrosstalkSources_855453_42.stderr parsl.dataflow.dflow DEBUG: Task 4488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4488 has memoization hash 0bb4368937044a6bb4dc166c852cd832 parsl.dataflow.memoization INFO: Task 4488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4488 try 0 launched on executor multi with executor id 4489 parsl.dataflow.dflow INFO: Standard output for task 4488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca178794-770c-4820-9cbf-2c84abbab139_isrForCrosstalkSources_855437_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca178794-770c-4820-9cbf-2c84abbab139_isrForCrosstalkSources_855437_9.stderr parsl.dataflow.dflow DEBUG: Task 4489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4489 has memoization hash 7fcfb39b926301bf249a1deaf76be665 parsl.dataflow.memoization INFO: Task 4489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4489 try 0 launched on executor multi with executor id 4490 parsl.dataflow.dflow INFO: Standard output for task 4489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b526bc0d-2076-41ea-a098-2da54dd9e2f0_isrForCrosstalkSources_855456_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b526bc0d-2076-41ea-a098-2da54dd9e2f0_isrForCrosstalkSources_855456_55.stderr parsl.dataflow.dflow DEBUG: Task 4490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4490 has memoization hash b455f56b5c44d895735cba68706ad74d parsl.dataflow.memoization INFO: Task 4490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4490 try 0 launched on executor multi with executor id 4491 parsl.dataflow.dflow INFO: Standard output for task 4490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d0627ec3-bc67-4c88-9d99-0572e83d0e02_isrForCrosstalkSources_855431_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d0627ec3-bc67-4c88-9d99-0572e83d0e02_isrForCrosstalkSources_855431_60.stderr parsl.dataflow.dflow DEBUG: Task 4491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4491 has memoization hash c469a13ad601641de162d6713e6ecd23 parsl.dataflow.memoization INFO: Task 4491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4491 try 0 launched on executor multi with executor id 4492 parsl.dataflow.dflow INFO: Standard output for task 4491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/651a31e2-3d61-43b4-bf90-5402d59ed4bd_isrForCrosstalkSources_855436_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/651a31e2-3d61-43b4-bf90-5402d59ed4bd_isrForCrosstalkSources_855436_24.stderr parsl.dataflow.dflow DEBUG: Task 4492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4492 has memoization hash ca06d4ecddebb639a21d95790c358a4c parsl.dataflow.memoization INFO: Task 4492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4492 try 0 launched on executor multi with executor id 4493 parsl.dataflow.dflow INFO: Standard output for task 4492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f31e918a-0ce4-497d-a6b2-3996cb4aa655_isrForCrosstalkSources_855390_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f31e918a-0ce4-497d-a6b2-3996cb4aa655_isrForCrosstalkSources_855390_59.stderr parsl.dataflow.dflow DEBUG: Task 4493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4493 has memoization hash 32fc916e6bd4bcb0b5bc21b926d31333 parsl.dataflow.memoization INFO: Task 4493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:52 proc_lsst.multi:170 [INFO] [multi] job_id 16964195 proc_lsst.multi INFO: [multi] job_id 16964195 2024-03-09 14:51:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-09 14:51:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964195 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964195 parsl.executors.status_handling DEBUG: Launched block 169 on executor multi with job ID 16964195 parsl.executors.status_handling INFO: Allocated block ID 170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-170 2024-03-09 14:51:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-09 14:51:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4493 try 0 launched on executor multi with executor id 4494 parsl.dataflow.dflow INFO: Standard output for task 4493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5e58c47e-a384-4e92-9c16-642d7444fdd5_isrForCrosstalkSources_855404_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5e58c47e-a384-4e92-9c16-642d7444fdd5_isrForCrosstalkSources_855404_30.stderr parsl.dataflow.dflow DEBUG: Task 4494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4494 has memoization hash 121b959ba17dca127d37e2e17495a2a1 parsl.dataflow.memoization INFO: Task 4494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4494 try 0 launched on executor multi with executor id 4495 parsl.dataflow.dflow INFO: Standard output for task 4494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/751de754-0798-4dc8-a72d-d053ed6d0426_isrForCrosstalkSources_855451_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/751de754-0798-4dc8-a72d-d053ed6d0426_isrForCrosstalkSources_855451_24.stderr parsl.dataflow.dflow DEBUG: Task 4495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4495 has memoization hash 98a593904d2dac1d43d712722d98fd31 parsl.dataflow.memoization INFO: Task 4495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4495 try 0 launched on executor multi with executor id 4496 parsl.dataflow.dflow INFO: Standard output for task 4495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/79776475-b895-4fe7-982b-bce81eacf342_isrForCrosstalkSources_855444_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/79776475-b895-4fe7-982b-bce81eacf342_isrForCrosstalkSources_855444_14.stderr parsl.dataflow.dflow DEBUG: Task 4496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4496 has memoization hash ee79ec3ae716b860d7ac812cbc0a37f4 parsl.dataflow.memoization INFO: Task 4496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4496 try 0 launched on executor multi with executor id 4497 parsl.dataflow.dflow INFO: Standard output for task 4496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/e463a854-12d1-4c2b-ac2e-d702ef48110b_isrForCrosstalkSources_855389_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/e463a854-12d1-4c2b-ac2e-d702ef48110b_isrForCrosstalkSources_855389_25.stderr parsl.dataflow.dflow DEBUG: Task 4497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4497 has memoization hash 443334c6f6b06fd2803e8ad09efb9145 parsl.dataflow.memoization INFO: Task 4497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4497 try 0 launched on executor multi with executor id 4498 parsl.dataflow.dflow INFO: Standard output for task 4497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/f8a8d2e5-1d9e-419a-9e93-23b5637d6593_isrForCrosstalkSources_855410_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/f8a8d2e5-1d9e-419a-9e93-23b5637d6593_isrForCrosstalkSources_855410_38.stderr parsl.dataflow.dflow DEBUG: Task 4498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4498 has memoization hash 8e2d64819c709d3dde24ce04076c18c5 parsl.dataflow.memoization INFO: Task 4498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4498 try 0 launched on executor multi with executor id 4499 parsl.dataflow.dflow INFO: Standard output for task 4498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ff71dee4-b287-4f71-838d-98d488dff6dc_isrForCrosstalkSources_855403_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ff71dee4-b287-4f71-838d-98d488dff6dc_isrForCrosstalkSources_855403_53.stderr parsl.dataflow.dflow DEBUG: Task 4499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4499 has memoization hash f3168b8a10b3af2212953e4b0c02c17f parsl.dataflow.memoization INFO: Task 4499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4499 try 0 launched on executor multi with executor id 4500 parsl.dataflow.dflow INFO: Standard output for task 4499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5f866e33-691d-4075-8401-b1f4d92d6ee5_isrForCrosstalkSources_855425_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5f866e33-691d-4075-8401-b1f4d92d6ee5_isrForCrosstalkSources_855425_45.stderr parsl.dataflow.dflow DEBUG: Task 4500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4500 has memoization hash 6639cf57c72399a8b282e6a723c9aec3 parsl.dataflow.memoization INFO: Task 4500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4500 try 0 launched on executor multi with executor id 4501 parsl.dataflow.dflow INFO: Standard output for task 4500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5b0ae83b-d399-4226-994e-93599832e162_isrForCrosstalkSources_855397_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5b0ae83b-d399-4226-994e-93599832e162_isrForCrosstalkSources_855397_50.stderr parsl.dataflow.dflow DEBUG: Task 4501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4501 has memoization hash 2f17cd3818cc40c74056f1550161175d parsl.dataflow.memoization INFO: Task 4501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4501 try 0 launched on executor multi with executor id 4502 parsl.dataflow.dflow INFO: Standard output for task 4501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/516ee66c-9a98-4a17-b211-9363a2da21de_isrForCrosstalkSources_855422_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/516ee66c-9a98-4a17-b211-9363a2da21de_isrForCrosstalkSources_855422_29.stderr parsl.dataflow.dflow DEBUG: Task 4502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4502 has memoization hash f71c83aab8be4df76ae7394a681999eb parsl.dataflow.memoization INFO: Task 4502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4502 try 0 launched on executor multi with executor id 4503 parsl.dataflow.dflow INFO: Standard output for task 4502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9222778-802d-4ee6-9ffb-bc914c3efb40_isrForCrosstalkSources_855449_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9222778-802d-4ee6-9ffb-bc914c3efb40_isrForCrosstalkSources_855449_5.stderr parsl.dataflow.dflow DEBUG: Task 4503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4503 has memoization hash 4f3e66002cd9b02e3afc905dde5f446a parsl.dataflow.memoization INFO: Task 4503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4503 try 0 launched on executor multi with executor id 4504 parsl.dataflow.dflow INFO: Standard output for task 4503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a42060d-bc4d-4263-9e2a-76fdaa105fa0_isrForCrosstalkSources_855447_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a42060d-bc4d-4263-9e2a-76fdaa105fa0_isrForCrosstalkSources_855447_3.stderr parsl.dataflow.dflow DEBUG: Task 4504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4504 has memoization hash f170fa50c2bfa063d80737f569bffc70 parsl.dataflow.memoization INFO: Task 4504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4504 try 0 launched on executor multi with executor id 4505 parsl.dataflow.dflow INFO: Standard output for task 4504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/53aaec3b-978e-4ad9-8161-92003f461bf6_isrForCrosstalkSources_855421_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/53aaec3b-978e-4ad9-8161-92003f461bf6_isrForCrosstalkSources_855421_33.stderr parsl.dataflow.dflow DEBUG: Task 4505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4505 has memoization hash 2546d159fea4188ed0b609a156252e6f parsl.dataflow.memoization INFO: Task 4505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4505 try 0 launched on executor multi with executor id 4506 parsl.dataflow.dflow INFO: Standard output for task 4505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6f65f311-e3a7-471e-9eb4-173f1bfe39c8_isrForCrosstalkSources_855458_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6f65f311-e3a7-471e-9eb4-173f1bfe39c8_isrForCrosstalkSources_855458_53.stderr parsl.dataflow.dflow DEBUG: Task 4506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4506 has memoization hash 23fbc4b1691fcd5641b35f4ddbf53baf parsl.dataflow.memoization INFO: Task 4506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4506 try 0 launched on executor multi with executor id 4507 parsl.dataflow.dflow INFO: Standard output for task 4506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f94fe58d-00dc-411a-9661-5c9cc574e9ce_isrForCrosstalkSources_855381_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f94fe58d-00dc-411a-9661-5c9cc574e9ce_isrForCrosstalkSources_855381_4.stderr parsl.dataflow.dflow DEBUG: Task 4507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4507 has memoization hash cf1918bbdcaa5e0e324dcb99704d53ca parsl.dataflow.memoization INFO: Task 4507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4507 try 0 launched on executor multi with executor id 4508 parsl.dataflow.dflow INFO: Standard output for task 4507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/179d0d87-ad03-4995-a241-0d4b99a52631_isrForCrosstalkSources_855410_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/179d0d87-ad03-4995-a241-0d4b99a52631_isrForCrosstalkSources_855410_35.stderr parsl.dataflow.dflow DEBUG: Task 4508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4508 has memoization hash 72274e0707f8d0125e206c881c04c1ca parsl.dataflow.memoization INFO: Task 4508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4508 try 0 launched on executor multi with executor id 4509 parsl.dataflow.dflow INFO: Standard output for task 4508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/14602b86-f19f-47ee-824f-a91ddab2b374_isrForCrosstalkSources_855413_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/14602b86-f19f-47ee-824f-a91ddab2b374_isrForCrosstalkSources_855413_6.stderr parsl.dataflow.dflow DEBUG: Task 4509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4509 has memoization hash 4cb067fa8cc359a6bc8d12e95faae19d parsl.dataflow.memoization INFO: Task 4509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4509 try 0 launched on executor multi with executor id 4510 parsl.dataflow.dflow INFO: Standard output for task 4509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a919adc5-8501-45d6-9ac8-f85bd3edc40f_isrForCrosstalkSources_855406_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a919adc5-8501-45d6-9ac8-f85bd3edc40f_isrForCrosstalkSources_855406_9.stderr parsl.dataflow.dflow DEBUG: Task 4510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4510 has memoization hash e91ad367b51ce750c1d94599baf0769a parsl.dataflow.memoization INFO: Task 4510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4510 try 0 launched on executor multi with executor id 4511 parsl.dataflow.dflow INFO: Standard output for task 4510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/348a20a7-d472-417a-b962-2a95dc0c6718_isrForCrosstalkSources_855429_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/348a20a7-d472-417a-b962-2a95dc0c6718_isrForCrosstalkSources_855429_13.stderr parsl.dataflow.dflow DEBUG: Task 4511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4511 has memoization hash 4ba46e55675eb14c2f499977fc3eaba0 parsl.dataflow.memoization INFO: Task 4511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4511 try 0 launched on executor multi with executor id 4512 parsl.dataflow.dflow INFO: Standard output for task 4511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4da6316c-ab9a-4bc6-9f26-4b9f13a594be_isrForCrosstalkSources_855404_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4da6316c-ab9a-4bc6-9f26-4b9f13a594be_isrForCrosstalkSources_855404_27.stderr parsl.dataflow.dflow DEBUG: Task 4512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4512 has memoization hash a1af4ee64534c1e0466cab25647b172e parsl.dataflow.memoization INFO: Task 4512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4512 try 0 launched on executor multi with executor id 4513 parsl.dataflow.dflow INFO: Standard output for task 4512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2802d345-7937-41b4-9bab-6a103ac4e55a_isrForCrosstalkSources_855403_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2802d345-7937-41b4-9bab-6a103ac4e55a_isrForCrosstalkSources_855403_42.stderr parsl.dataflow.dflow DEBUG: Task 4513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4513 has memoization hash fb88feb59c7855498a307856a4b25519 parsl.dataflow.memoization INFO: Task 4513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4513 try 0 launched on executor multi with executor id 4514 parsl.dataflow.dflow INFO: Standard output for task 4513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3a3b6ce6-1676-4123-b447-cb557157e9df_isrForCrosstalkSources_855445_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3a3b6ce6-1676-4123-b447-cb557157e9df_isrForCrosstalkSources_855445_2.stderr parsl.dataflow.dflow DEBUG: Task 4514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4514 has memoization hash 33769302d5bcc569245bd30190029bf8 parsl.dataflow.memoization INFO: Task 4514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4514 try 0 launched on executor multi with executor id 4515 parsl.dataflow.dflow INFO: Standard output for task 4514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7812401e-16fb-495d-9ac4-e4d93629a803_isrForCrosstalkSources_855437_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7812401e-16fb-495d-9ac4-e4d93629a803_isrForCrosstalkSources_855437_29.stderr parsl.dataflow.dflow DEBUG: Task 4515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4515 has memoization hash cced7f4caf350474ff0a83e3c2e12182 parsl.dataflow.memoization INFO: Task 4515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4515 try 0 launched on executor multi with executor id 4516 parsl.dataflow.dflow INFO: Standard output for task 4515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0a6ccec8-80ca-45a9-96f7-ee22c1bfc5bf_isrForCrosstalkSources_855424_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0a6ccec8-80ca-45a9-96f7-ee22c1bfc5bf_isrForCrosstalkSources_855424_24.stderr parsl.dataflow.dflow DEBUG: Task 4516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4516 has memoization hash 6719473c2d91b46ea7abf8e60dcbc912 parsl.dataflow.memoization INFO: Task 4516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4516 try 0 launched on executor multi with executor id 4517 parsl.dataflow.dflow INFO: Standard output for task 4516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84c5cb95-00bc-493c-babc-acd1fe86a908_isrForCrosstalkSources_855416_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84c5cb95-00bc-493c-babc-acd1fe86a908_isrForCrosstalkSources_855416_50.stderr parsl.dataflow.dflow DEBUG: Task 4517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4517 has memoization hash dc74a189023a036d8d591bf3df3dd458 parsl.dataflow.memoization INFO: Task 4517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4517 try 0 launched on executor multi with executor id 4518 parsl.dataflow.dflow INFO: Standard output for task 4517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/59a66a81-7668-46d6-9b6e-cae5f2d1bc48_isrForCrosstalkSources_855426_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/59a66a81-7668-46d6-9b6e-cae5f2d1bc48_isrForCrosstalkSources_855426_9.stderr parsl.dataflow.dflow DEBUG: Task 4518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4518 has memoization hash 6965d28e59a1e780dc33abe2693c0df6 parsl.dataflow.memoization INFO: Task 4518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5556012ada06c9e935a2a9f8dd843c64 with result from task 276 parsl.dataflow.dflow INFO: Parsl task 4518 try 0 launched on executor multi with executor id 4519 parsl.dataflow.dflow INFO: Standard output for task 4518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5d0c7eaa-a236-4c1d-855f-d77470889218_isrForCrosstalkSources_855381_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5d0c7eaa-a236-4c1d-855f-d77470889218_isrForCrosstalkSources_855381_28.stderr parsl.dataflow.dflow DEBUG: Task 4519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4519 has memoization hash ee2141935d1da50f05fe35fc029f0296 parsl.dataflow.memoization INFO: Task 4519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4519 try 0 launched on executor multi with executor id 4520 parsl.dataflow.dflow INFO: Standard output for task 4519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fa77dbad-ab6b-4026-9cc9-3f56d74a71b6_isrForCrosstalkSources_855440_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fa77dbad-ab6b-4026-9cc9-3f56d74a71b6_isrForCrosstalkSources_855440_7.stderr parsl.dataflow.dflow DEBUG: Task 4520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4520 has memoization hash 4d35fec0c0c6d0e2586a5bc098ddfe49 parsl.dataflow.memoization INFO: Task 4520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/fe631fb3-fa1e-4f2c-9b75-87af4ab60d78_isrForCrosstalkSources_855398_62.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/fe631fb3-fa1e-4f2c-9b75-87af4ab60d78_isrForCrosstalkSources_855398_62.stderr parsl.dataflow.dflow INFO: Parsl task 4520 try 0 launched on executor multi with executor id 4521 parsl.dataflow.dflow INFO: Standard output for task 4520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c8194955-1fca-4235-89cc-b87077ed6158_isrForCrosstalkSources_855419_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c8194955-1fca-4235-89cc-b87077ed6158_isrForCrosstalkSources_855419_32.stderr parsl.dataflow.dflow DEBUG: Task 4521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4521 has memoization hash b4df7cc4461cc99e89c6d800117ad0fa parsl.dataflow.memoization INFO: Task 4521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4521 try 0 launched on executor multi with executor id 4522 parsl.dataflow.dflow INFO: Standard output for task 4521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3768d226-8eaf-4db7-99f4-63b69989856d_isrForCrosstalkSources_855447_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3768d226-8eaf-4db7-99f4-63b69989856d_isrForCrosstalkSources_855447_38.stderr parsl.dataflow.dflow DEBUG: Task 4522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4522 has memoization hash 90d4316df10372f0e7fd5528c5f8d80c parsl.dataflow.memoization INFO: Task 4522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:53 proc_lsst.multi:170 [INFO] [multi] job_id 16964196 proc_lsst.multi INFO: [multi] job_id 16964196 2024-03-09 14:51:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-09 14:51:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964196 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964196 parsl.executors.status_handling DEBUG: Launched block 170 on executor multi with job ID 16964196 parsl.executors.status_handling INFO: Allocated block ID 171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-171 2024-03-09 14:51:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4522 try 0 launched on executor multi with executor id 4523 parsl.dataflow.dflow INFO: Standard output for task 4522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f6f4dadf-30c5-4d61-b644-342c3fc087ee_isrForCrosstalkSources_855446_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f6f4dadf-30c5-4d61-b644-342c3fc087ee_isrForCrosstalkSources_855446_10.stderr parsl.dataflow.dflow DEBUG: Task 4523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4523 has memoization hash 8afd943f2589d0776b3b087a148c8ed1 parsl.dataflow.memoization INFO: Task 4523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4523 try 0 launched on executor multi with executor id 4524 parsl.dataflow.dflow INFO: Standard output for task 4523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8bdf65ab-c4b5-42ec-81d2-36b1aef0dbb2_isrForCrosstalkSources_855456_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8bdf65ab-c4b5-42ec-81d2-36b1aef0dbb2_isrForCrosstalkSources_855456_53.stderr parsl.dataflow.dflow DEBUG: Task 4524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4524 has memoization hash 7ae54a4736aa2f1b7d0bfbab7e54ec14 parsl.dataflow.memoization INFO: Task 4524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4524 try 0 launched on executor multi with executor id 4525 parsl.dataflow.dflow INFO: Standard output for task 4524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/67713071-13a8-48b4-b25b-667b5c3f2d8d_isrForCrosstalkSources_855436_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/67713071-13a8-48b4-b25b-667b5c3f2d8d_isrForCrosstalkSources_855436_62.stderr parsl.dataflow.dflow DEBUG: Task 4525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4525 has memoization hash 86ec2e7690f4eb9f6ac4e21be99f5e9a parsl.dataflow.memoization INFO: Task 4525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4525 try 0 launched on executor multi with executor id 4526 parsl.dataflow.dflow INFO: Standard output for task 4525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d7d7d656-22e8-4eee-bfee-8ec2a9cc7bb0_isrForCrosstalkSources_855386_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d7d7d656-22e8-4eee-bfee-8ec2a9cc7bb0_isrForCrosstalkSources_855386_16.stderr parsl.dataflow.dflow DEBUG: Task 4526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4526 has memoization hash 6b62adf1f99ff6e74217e6c0985ac94a parsl.dataflow.memoization INFO: Task 4526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4526 try 0 launched on executor multi with executor id 4527 parsl.dataflow.dflow INFO: Standard output for task 4526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/72836117-ccac-4bca-80e6-9f3b222894f0_isrForCrosstalkSources_855404_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/72836117-ccac-4bca-80e6-9f3b222894f0_isrForCrosstalkSources_855404_42.stderr parsl.dataflow.dflow DEBUG: Task 4527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4527 has memoization hash 1160e647bb4383dc541a6a7c2f274165 parsl.dataflow.memoization INFO: Task 4527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aedd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4527 try 0 launched on executor multi with executor id 4528 parsl.dataflow.dflow INFO: Standard output for task 4527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1c2da685-0384-4fa5-a1cf-bfd224f47633_isrForCrosstalkSources_855452_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1c2da685-0384-4fa5-a1cf-bfd224f47633_isrForCrosstalkSources_855452_11.stderr parsl.dataflow.dflow DEBUG: Task 4528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4528 has memoization hash a8d0ff160453bc5566b7245d5e716c14 parsl.dataflow.memoization INFO: Task 4528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4528 try 0 launched on executor multi with executor id 4529 parsl.dataflow.dflow INFO: Standard output for task 4528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cc65b729-f3e0-4285-af6e-37f100185f09_isrForCrosstalkSources_855382_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cc65b729-f3e0-4285-af6e-37f100185f09_isrForCrosstalkSources_855382_40.stderr parsl.dataflow.dflow DEBUG: Task 4529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4529 has memoization hash 561e955efafb6e24d66fc9e3413dd0fd parsl.dataflow.memoization INFO: Task 4529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59a8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4529 try 0 launched on executor multi with executor id 4530 parsl.dataflow.dflow INFO: Standard output for task 4529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08707dd7-ee1e-4e6d-8247-50a0baf0fffc_isrForCrosstalkSources_855437_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08707dd7-ee1e-4e6d-8247-50a0baf0fffc_isrForCrosstalkSources_855437_35.stderr parsl.dataflow.dflow DEBUG: Task 4530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4530 has memoization hash b90e024d140e7d6f06bf41e8d57db268 parsl.dataflow.memoization INFO: Task 4530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4530 try 0 launched on executor multi with executor id 4531 parsl.dataflow.dflow INFO: Standard output for task 4530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f586b0f1-8748-4989-b208-52f361fe26e2_isrForCrosstalkSources_855384_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f586b0f1-8748-4989-b208-52f361fe26e2_isrForCrosstalkSources_855384_35.stderr parsl.dataflow.dflow DEBUG: Task 4531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4531 has memoization hash d534730b4818527d0da6ab56540ecbeb parsl.dataflow.memoization INFO: Task 4531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4531 try 0 launched on executor multi with executor id 4532 parsl.dataflow.dflow INFO: Standard output for task 4531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/acf2dd10-58aa-47ac-8050-ba913d6635df_isrForCrosstalkSources_855398_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/acf2dd10-58aa-47ac-8050-ba913d6635df_isrForCrosstalkSources_855398_12.stderr parsl.dataflow.dflow DEBUG: Task 4532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4532 has memoization hash a033e2643093bab08a252b8cf428b69b parsl.dataflow.memoization INFO: Task 4532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4532 try 0 launched on executor multi with executor id 4533 parsl.dataflow.dflow INFO: Standard output for task 4532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35a4c009-91cc-4f75-87dc-288c1501f7b5_isrForCrosstalkSources_855385_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35a4c009-91cc-4f75-87dc-288c1501f7b5_isrForCrosstalkSources_855385_20.stderr parsl.dataflow.dflow DEBUG: Task 4533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4533 has memoization hash a39d123b7b7de2a90973e4a68f5d4842 parsl.dataflow.memoization INFO: Task 4533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4533 try 0 launched on executor multi with executor id 4534 parsl.dataflow.dflow INFO: Standard output for task 4533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6bd9e5c1-843d-47a0-9a21-7ab0cc433650_isrForCrosstalkSources_855395_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6bd9e5c1-843d-47a0-9a21-7ab0cc433650_isrForCrosstalkSources_855395_38.stderr parsl.dataflow.dflow DEBUG: Task 4534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4534 has memoization hash ae814f112b628c2bdb223e5068f23b9a parsl.dataflow.memoization INFO: Task 4534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4534 try 0 launched on executor multi with executor id 4535 parsl.dataflow.dflow INFO: Standard output for task 4534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0e06a9af-7847-4a80-ba09-b01a22a29f69_isrForCrosstalkSources_855406_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0e06a9af-7847-4a80-ba09-b01a22a29f69_isrForCrosstalkSources_855406_47.stderr parsl.dataflow.dflow DEBUG: Task 4535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4535 has memoization hash 0fa7eaa837ab486c7a9306e70204ee2e parsl.dataflow.memoization INFO: Task 4535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4535 try 0 launched on executor multi with executor id 4536 parsl.dataflow.dflow INFO: Standard output for task 4535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/668db0d8-3fe9-4f64-b11b-d2960f02783b_isrForCrosstalkSources_855399_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/668db0d8-3fe9-4f64-b11b-d2960f02783b_isrForCrosstalkSources_855399_40.stderr parsl.dataflow.dflow DEBUG: Task 4536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4536 has memoization hash 566bdc673e44b9c85541bcf32e9dbdc4 parsl.dataflow.memoization INFO: Task 4536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4536 try 0 launched on executor multi with executor id 4537 parsl.dataflow.dflow INFO: Standard output for task 4536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e4f34cde-7b74-4d88-90c7-35aa8b2a8705_isrForCrosstalkSources_855421_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e4f34cde-7b74-4d88-90c7-35aa8b2a8705_isrForCrosstalkSources_855421_60.stderr parsl.dataflow.dflow DEBUG: Task 4537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4537 has memoization hash fd388507ef713280dac66d51f89728a2 parsl.dataflow.memoization INFO: Task 4537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4537 try 0 launched on executor multi with executor id 4538 parsl.dataflow.dflow INFO: Standard output for task 4537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/af094aaf-2b9e-4e81-86de-012b7359ff63_isrForCrosstalkSources_855404_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/af094aaf-2b9e-4e81-86de-012b7359ff63_isrForCrosstalkSources_855404_23.stderr parsl.dataflow.dflow DEBUG: Task 4538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4538 has memoization hash 4989dc53b0ec02dffa02a0e8467fd8cf parsl.dataflow.memoization INFO: Task 4538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4538 try 0 launched on executor multi with executor id 4539 parsl.dataflow.dflow INFO: Standard output for task 4538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a6ecba16-8701-4430-8234-99e4bb7cd1f0_isrForCrosstalkSources_855410_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a6ecba16-8701-4430-8234-99e4bb7cd1f0_isrForCrosstalkSources_855410_15.stderr parsl.dataflow.dflow DEBUG: Task 4539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4539 has memoization hash 26e2a0acc319c53965351ad7793f19fe parsl.dataflow.memoization INFO: Task 4539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4539 try 0 launched on executor multi with executor id 4540 parsl.dataflow.dflow INFO: Standard output for task 4539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b7eab635-c02e-4026-a973-6ace5dc7f23d_isrForCrosstalkSources_855432_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b7eab635-c02e-4026-a973-6ace5dc7f23d_isrForCrosstalkSources_855432_30.stderr parsl.dataflow.dflow DEBUG: Task 4540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4540 has memoization hash 57987db0a383302fcf6d20b58dcedd16 parsl.dataflow.memoization INFO: Task 4540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4540 try 0 launched on executor multi with executor id 4541 parsl.dataflow.dflow INFO: Standard output for task 4540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/49adf696-6f05-4fac-a402-54428a1c762a_isrForCrosstalkSources_855385_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/49adf696-6f05-4fac-a402-54428a1c762a_isrForCrosstalkSources_855385_32.stderr parsl.dataflow.dflow DEBUG: Task 4541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4541 has memoization hash a669646a1fb8fad12924e4519937d1b7 parsl.dataflow.memoization INFO: Task 4541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4541 try 0 launched on executor multi with executor id 4542 parsl.dataflow.dflow INFO: Standard output for task 4541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e4632eea-7cbc-4b66-9a21-2becfa4a983a_isrForCrosstalkSources_855402_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e4632eea-7cbc-4b66-9a21-2becfa4a983a_isrForCrosstalkSources_855402_20.stderr parsl.dataflow.dflow DEBUG: Task 4542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4542 has memoization hash a183ea301ad2e6cf7070cd9735bf2f44 parsl.dataflow.memoization INFO: Task 4542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4542 try 0 launched on executor multi with executor id 4543 parsl.dataflow.dflow INFO: Standard output for task 4542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e7e89990-288c-47c0-8f15-68057804b588_isrForCrosstalkSources_855420_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e7e89990-288c-47c0-8f15-68057804b588_isrForCrosstalkSources_855420_35.stderr parsl.dataflow.dflow DEBUG: Task 4543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4543 has memoization hash 19de98207408c169bfd6c2460790099a parsl.dataflow.memoization INFO: Task 4543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4543 try 0 launched on executor multi with executor id 4544 parsl.dataflow.dflow INFO: Standard output for task 4543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/39667fa6-7e0b-44d8-8a39-42e072df4d09_isrForCrosstalkSources_855399_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/39667fa6-7e0b-44d8-8a39-42e072df4d09_isrForCrosstalkSources_855399_34.stderr parsl.dataflow.dflow DEBUG: Task 4544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4544 has memoization hash c37a2a616daa733087d5abc9d71336bb parsl.dataflow.memoization INFO: Task 4544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59ab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4544 try 0 launched on executor multi with executor id 4545 parsl.dataflow.dflow INFO: Standard output for task 4544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8440e62b-4d88-46d2-b353-c08f9e2a3d2b_isrForCrosstalkSources_855404_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8440e62b-4d88-46d2-b353-c08f9e2a3d2b_isrForCrosstalkSources_855404_26.stderr parsl.dataflow.dflow DEBUG: Task 4545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4545 has memoization hash 82933277d4be4eb7b619a6800e3f3c65 parsl.dataflow.memoization INFO: Task 4545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4545 try 0 launched on executor multi with executor id 4546 parsl.dataflow.dflow INFO: Standard output for task 4545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/24bd0894-b463-4317-b8d8-dcd4bb2b5b30_isrForCrosstalkSources_855407_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/24bd0894-b463-4317-b8d8-dcd4bb2b5b30_isrForCrosstalkSources_855407_1.stderr parsl.dataflow.dflow DEBUG: Task 4546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4546 has memoization hash 341383a350c92269c52bf82b45ac6fcb parsl.dataflow.memoization INFO: Task 4546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4546 try 0 launched on executor multi with executor id 4547 parsl.dataflow.dflow INFO: Standard output for task 4546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1f0e8c8a-ad7b-41a1-941d-95072133e6e9_isrForCrosstalkSources_855452_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1f0e8c8a-ad7b-41a1-941d-95072133e6e9_isrForCrosstalkSources_855452_46.stderr parsl.dataflow.dflow DEBUG: Task 4547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4547 has memoization hash 3f6f9aacac7b16f9b12554eac26ba75b parsl.dataflow.memoization INFO: Task 4547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4547 try 0 launched on executor multi with executor id 4548 parsl.dataflow.dflow INFO: Standard output for task 4547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c111500c-792d-4f85-a59d-b26efc2777a4_isrForCrosstalkSources_855413_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c111500c-792d-4f85-a59d-b26efc2777a4_isrForCrosstalkSources_855413_1.stderr parsl.dataflow.dflow DEBUG: Task 4548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4548 has memoization hash bafe8396e4c6e29bfdb95812a6af11f2 parsl.dataflow.memoization INFO: Task 4548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4548 try 0 launched on executor multi with executor id 4549 parsl.dataflow.dflow INFO: Standard output for task 4548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b292ff7-d5db-4b92-9f28-23e6274f1752_isrForCrosstalkSources_855449_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b292ff7-d5db-4b92-9f28-23e6274f1752_isrForCrosstalkSources_855449_49.stderr parsl.dataflow.dflow DEBUG: Task 4549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4549 has memoization hash cb9d757b0161a9dbe044a17dc9b702c7 parsl.dataflow.memoization INFO: Task 4549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4549 try 0 launched on executor multi with executor id 4550 parsl.dataflow.dflow INFO: Standard output for task 4549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0acfad0e-7d3b-4780-829e-df3be8d63809_isrForCrosstalkSources_855439_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0acfad0e-7d3b-4780-829e-df3be8d63809_isrForCrosstalkSources_855439_40.stderr parsl.dataflow.dflow DEBUG: Task 4550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4550 has memoization hash 6e6666793521a42b11f545f18b9a5e7d parsl.dataflow.memoization INFO: Task 4550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4550 try 0 launched on executor multi with executor id 4551 parsl.dataflow.dflow INFO: Standard output for task 4550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0826e4c9-4a6e-4a27-9a1c-9ea3371ba2d3_isrForCrosstalkSources_855429_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0826e4c9-4a6e-4a27-9a1c-9ea3371ba2d3_isrForCrosstalkSources_855429_27.stderr parsl.dataflow.dflow DEBUG: Task 4551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4551 has memoization hash c33a243665de0410216f0a3123fd88f6 parsl.dataflow.memoization INFO: Task 4551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59abba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4551 try 0 launched on executor multi with executor id 4552 parsl.dataflow.dflow INFO: Standard output for task 4551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/44d5fc74-52c2-4afb-9952-1a36222f1c2a_isrForCrosstalkSources_855438_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/44d5fc74-52c2-4afb-9952-1a36222f1c2a_isrForCrosstalkSources_855438_25.stderr parsl.dataflow.dflow DEBUG: Task 4552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4552 has memoization hash ac0cdd6112f8001a737097aee0ba113c parsl.dataflow.memoization INFO: Task 4552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4552 try 0 launched on executor multi with executor id 4553 parsl.dataflow.dflow INFO: Standard output for task 4552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/37761017-bf10-4181-be73-646fb58367ff_isrForCrosstalkSources_855430_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/37761017-bf10-4181-be73-646fb58367ff_isrForCrosstalkSources_855430_24.stderr parsl.dataflow.dflow DEBUG: Task 4553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4553 has memoization hash 46a3d3bb6b4dc2b828c3f371278dbaf8 parsl.dataflow.memoization INFO: Task 4553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4553 try 0 launched on executor multi with executor id 4554 parsl.dataflow.dflow INFO: Standard output for task 4553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bc5f7542-5d00-436d-966d-fa413087588d_isrForCrosstalkSources_855417_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bc5f7542-5d00-436d-966d-fa413087588d_isrForCrosstalkSources_855417_33.stderr parsl.dataflow.dflow DEBUG: Task 4554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4554 has memoization hash 94bc628a431cbac3807f2e3810470791 parsl.dataflow.memoization INFO: Task 4554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4554 try 0 launched on executor multi with executor id 4555 parsl.dataflow.dflow INFO: Standard output for task 4554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0277f124-b1af-434f-8296-22dea8ef59b2_isrForCrosstalkSources_855424_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0277f124-b1af-434f-8296-22dea8ef59b2_isrForCrosstalkSources_855424_47.stderr parsl.dataflow.dflow DEBUG: Task 4555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4555 has memoization hash 3d04db9fc8869b30bb8b462a9db0c266 parsl.dataflow.memoization INFO: Task 4555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4555 try 0 launched on executor multi with executor id 4556 parsl.dataflow.dflow INFO: Standard output for task 4555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/9960c492-95cd-477f-8ca7-fb3a0e45381c_isrForCrosstalkSources_855412_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/9960c492-95cd-477f-8ca7-fb3a0e45381c_isrForCrosstalkSources_855412_60.stderr parsl.dataflow.dflow DEBUG: Task 4556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4556 has memoization hash 8402ddaa26625e7f4831d6f4919a8d19 parsl.dataflow.memoization INFO: Task 4556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4556 try 0 launched on executor multi with executor id 4557 parsl.dataflow.dflow INFO: Standard output for task 4556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0626d095-4c61-46ea-9594-b5a50908bcea_isrForCrosstalkSources_855426_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0626d095-4c61-46ea-9594-b5a50908bcea_isrForCrosstalkSources_855426_41.stderr parsl.dataflow.dflow DEBUG: Task 4557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4557 has memoization hash 590799232c51e858cf34ec25b505fbb4 parsl.dataflow.memoization INFO: Task 4557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4557 try 0 launched on executor multi with executor id 4558 parsl.dataflow.dflow INFO: Standard output for task 4557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/379573f2-1734-43a2-88c9-286e13715725_isrForCrosstalkSources_855424_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/379573f2-1734-43a2-88c9-286e13715725_isrForCrosstalkSources_855424_16.stderr parsl.dataflow.dflow DEBUG: Task 4558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4558 has memoization hash 8b97fc5b33b251c643c188b3ee52aae7 parsl.dataflow.memoization INFO: Task 4558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4558 try 0 launched on executor multi with executor id 4559 parsl.dataflow.dflow INFO: Standard output for task 4558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/bb90ec88-90bc-4fcf-972a-a5b13f2275a8_isrForCrosstalkSources_855418_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/bb90ec88-90bc-4fcf-972a-a5b13f2275a8_isrForCrosstalkSources_855418_11.stderr parsl.dataflow.dflow DEBUG: Task 4559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4559 has memoization hash 4aceb33b041c78dc8f07f0906ecaff21 parsl.dataflow.memoization INFO: Task 4559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4559 try 0 launched on executor multi with executor id 4560 parsl.dataflow.dflow INFO: Standard output for task 4559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/030d3933-905e-4548-aa19-54cc2b1dca0d_isrForCrosstalkSources_855422_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/030d3933-905e-4548-aa19-54cc2b1dca0d_isrForCrosstalkSources_855422_39.stderr parsl.dataflow.dflow DEBUG: Task 4560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4560 has memoization hash 3aa12507a52fe2184e3d6b5b633f1ccc parsl.dataflow.memoization INFO: Task 4560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4560 try 0 launched on executor multi with executor id 4561 parsl.dataflow.dflow INFO: Standard output for task 4560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/bbed5fad-0dc9-4390-aa69-7f331e08d7a4_isrForCrosstalkSources_855444_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/bbed5fad-0dc9-4390-aa69-7f331e08d7a4_isrForCrosstalkSources_855444_50.stderr parsl.dataflow.dflow DEBUG: Task 4561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4561 has memoization hash 7de3d82a53f43f212c820903d4247f1f parsl.dataflow.memoization INFO: Task 4561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4561 try 0 launched on executor multi with executor id 4562 parsl.dataflow.dflow INFO: Standard output for task 4561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/32284a33-f9ed-4b6e-ba12-72cf5ac718ff_isrForCrosstalkSources_855440_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/32284a33-f9ed-4b6e-ba12-72cf5ac718ff_isrForCrosstalkSources_855440_41.stderr parsl.dataflow.dflow DEBUG: Task 4562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4562 has memoization hash f169deb66cda3b9b0ef9ff5b16516c1d parsl.dataflow.memoization INFO: Task 4562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4562 try 0 launched on executor multi with executor id 4563 parsl.dataflow.dflow INFO: Standard output for task 4562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9e5c00be-0e54-49e5-a2bc-02558fb52530_isrForCrosstalkSources_855428_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9e5c00be-0e54-49e5-a2bc-02558fb52530_isrForCrosstalkSources_855428_48.stderr parsl.dataflow.dflow DEBUG: Task 4563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4563 has memoization hash 719a7fb3f7f6c3eee3d439533625a79c parsl.dataflow.memoization INFO: Task 4563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4563 try 0 launched on executor multi with executor id 4564 parsl.dataflow.dflow INFO: Standard output for task 4563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/98ce37d5-bc3b-441b-870f-0c2c8938d386_isrForCrosstalkSources_855455_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/98ce37d5-bc3b-441b-870f-0c2c8938d386_isrForCrosstalkSources_855455_38.stderr parsl.dataflow.dflow DEBUG: Task 4564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4564 has memoization hash 76d9bf575c30607b04e48e4db2762574 parsl.dataflow.memoization INFO: Task 4564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4564 try 0 launched on executor multi with executor id 4565 parsl.dataflow.dflow INFO: Standard output for task 4564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fb600c-7f77-4a27-95c8-6e993e49233a_isrForCrosstalkSources_855420_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fb600c-7f77-4a27-95c8-6e993e49233a_isrForCrosstalkSources_855420_1.stderr parsl.dataflow.dflow DEBUG: Task 4565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4565 has memoization hash bc8637a79b249bcafdc43e1ae51f10b0 parsl.dataflow.memoization INFO: Task 4565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4565 try 0 launched on executor multi with executor id 4566 parsl.dataflow.dflow INFO: Standard output for task 4565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/113adcb8-8758-44b6-8d50-7c1dcb98cee5_isrForCrosstalkSources_855434_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/113adcb8-8758-44b6-8d50-7c1dcb98cee5_isrForCrosstalkSources_855434_26.stderr parsl.dataflow.dflow DEBUG: Task 4566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4566 has memoization hash 274eef1d38c65c3e2ef370f46c8b9c91 parsl.dataflow.memoization INFO: Task 4566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4566 try 0 launched on executor multi with executor id 4567 parsl.dataflow.dflow INFO: Standard output for task 4566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/aee60548-94ec-40d1-9da9-2af87ecc6def_isrForCrosstalkSources_855433_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/aee60548-94ec-40d1-9da9-2af87ecc6def_isrForCrosstalkSources_855433_59.stderr parsl.dataflow.dflow DEBUG: Task 4567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4567 has memoization hash 2c7a6e6cfb0dbd457e0d82dc337ca0e0 parsl.dataflow.memoization INFO: Task 4567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4567 try 0 launched on executor multi with executor id 4568 parsl.dataflow.dflow INFO: Standard output for task 4567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2faa0fa5-11f1-485d-8e7f-9815c022b1b9_isrForCrosstalkSources_855429_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2faa0fa5-11f1-485d-8e7f-9815c022b1b9_isrForCrosstalkSources_855429_59.stderr parsl.dataflow.dflow DEBUG: Task 4568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4568 has memoization hash 6e2c2a1fb3f008204d3e76f3ea7e3096 parsl.dataflow.memoization INFO: Task 4568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4568 try 0 launched on executor multi with executor id 4569 parsl.dataflow.dflow INFO: Standard output for task 4568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b4237304-2fd2-4ec0-b2d3-52ed6b72fe86_isrForCrosstalkSources_855452_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b4237304-2fd2-4ec0-b2d3-52ed6b72fe86_isrForCrosstalkSources_855452_1.stderr parsl.dataflow.dflow DEBUG: Task 4569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4569 has memoization hash 811f3997172e63e589348368588a2a6a parsl.dataflow.memoization INFO: Task 4569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5c15815182e643ab4fe22224fc9d3dc with result from task 308 parsl.dataflow.dflow INFO: Parsl task 4569 try 0 launched on executor multi with executor id 4570 parsl.dataflow.dflow INFO: Standard output for task 4569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ea90c68e-217b-4241-9df2-ffffa0a9e273_isrForCrosstalkSources_855431_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ea90c68e-217b-4241-9df2-ffffa0a9e273_isrForCrosstalkSources_855431_38.stderr parsl.dataflow.dflow DEBUG: Task 4570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4570 has memoization hash 6de0d00c7a9705c72348f095c681060e parsl.dataflow.memoization INFO: Task 4570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4570 try 0 launched on executor multi with executor id 4571 parsl.dataflow.dflow INFO: Standard output for task 4570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f79f19b5-c771-417e-bd2a-154805cde88e_isrForCrosstalkSources_855401_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f79f19b5-c771-417e-bd2a-154805cde88e_isrForCrosstalkSources_855401_38.stderr parsl.dataflow.dflow DEBUG: Task 4571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4571 has memoization hash a9dea4fe47a49d08386f46ca27a18294 parsl.dataflow.memoization INFO: Task 4571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ee37b39a-d458-462d-8789-34cff3ddd614_isrForCrosstalkSources_855383_53.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ee37b39a-d458-462d-8789-34cff3ddd614_isrForCrosstalkSources_855383_53.stderr parsl.dataflow.dflow INFO: Parsl task 4571 try 0 launched on executor multi with executor id 4572 parsl.dataflow.dflow INFO: Standard output for task 4571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d934a907-d057-4883-85da-5835b5d73073_isrForCrosstalkSources_855450_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d934a907-d057-4883-85da-5835b5d73073_isrForCrosstalkSources_855450_40.stderr parsl.dataflow.dflow DEBUG: Task 4572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4572 has memoization hash a446b74e0e54d04307daca266f14297b parsl.dataflow.memoization INFO: Task 4572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4572 try 0 launched on executor multi with executor id 4573 parsl.dataflow.dflow INFO: Standard output for task 4572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e6b63f05-f1fe-470e-ba9b-03163efe3c26_isrForCrosstalkSources_855421_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e6b63f05-f1fe-470e-ba9b-03163efe3c26_isrForCrosstalkSources_855421_45.stderr parsl.dataflow.dflow DEBUG: Task 4573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4573 has memoization hash 9a1e22fe832e491aaecb395d809ea617 parsl.dataflow.memoization INFO: Task 4573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4573 try 0 launched on executor multi with executor id 4574 parsl.dataflow.dflow INFO: Standard output for task 4573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e6a6efd-8585-4cfc-bb57-bcaf5cda0ced_isrForCrosstalkSources_855408_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e6a6efd-8585-4cfc-bb57-bcaf5cda0ced_isrForCrosstalkSources_855408_55.stderr parsl.dataflow.dflow DEBUG: Task 4574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4574 has memoization hash 832278823b198a5bbeede070e514fda8 parsl.dataflow.memoization INFO: Task 4574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4574 try 0 launched on executor multi with executor id 4575 parsl.dataflow.dflow INFO: Standard output for task 4574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ba4b8835-06f4-41f7-b8ae-fb494f25e4eb_isrForCrosstalkSources_855457_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ba4b8835-06f4-41f7-b8ae-fb494f25e4eb_isrForCrosstalkSources_855457_17.stderr parsl.dataflow.dflow DEBUG: Task 4575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4575 has memoization hash fcf296bf69da38a32b6cc95e8b699693 parsl.dataflow.memoization INFO: Task 4575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4575 try 0 launched on executor multi with executor id 4576 parsl.dataflow.dflow INFO: Standard output for task 4575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d5e8829c-6597-4d19-9f54-ab2f147b8bd7_isrForCrosstalkSources_855442_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d5e8829c-6597-4d19-9f54-ab2f147b8bd7_isrForCrosstalkSources_855442_41.stderr parsl.dataflow.dflow DEBUG: Task 4576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4576 has memoization hash 5b0ef34f4e6ff0e27c492c770cdf01bf parsl.dataflow.memoization INFO: Task 4576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4576 try 0 launched on executor multi with executor id 4577 parsl.dataflow.dflow INFO: Standard output for task 4576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dde19a2a-80c6-49db-869f-a64b7c7b25c8_isrForCrosstalkSources_855405_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dde19a2a-80c6-49db-869f-a64b7c7b25c8_isrForCrosstalkSources_855405_1.stderr parsl.dataflow.dflow DEBUG: Task 4577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4577 has memoization hash c40bc3cbbd3e5ac0c4ba35cc5e223f48 parsl.dataflow.memoization INFO: Task 4577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:53 proc_lsst.multi:170 [INFO] [multi] job_id 16964197 proc_lsst.multi INFO: [multi] job_id 16964197 2024-03-09 14:51:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-09 14:51:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964197 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964197 parsl.executors.status_handling DEBUG: Launched block 171 on executor multi with job ID 16964197 parsl.executors.status_handling INFO: Allocated block ID 172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-172 2024-03-09 14:51:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-09 14:51:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4577 try 0 launched on executor multi with executor id 4578 parsl.dataflow.dflow INFO: Standard output for task 4577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9cba7650-d7a9-47a1-a2f1-271638f837b0_isrForCrosstalkSources_855457_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9cba7650-d7a9-47a1-a2f1-271638f837b0_isrForCrosstalkSources_855457_5.stderr parsl.dataflow.dflow DEBUG: Task 4578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4578 has memoization hash ecdf9acae82224ca82c845898981128d parsl.dataflow.memoization INFO: Task 4578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4578 try 0 launched on executor multi with executor id 4579 parsl.dataflow.dflow INFO: Standard output for task 4578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ab3ac9eb-2fab-4be5-b2e7-22cef37d159b_isrForCrosstalkSources_855412_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ab3ac9eb-2fab-4be5-b2e7-22cef37d159b_isrForCrosstalkSources_855412_16.stderr parsl.dataflow.dflow DEBUG: Task 4579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4579 has memoization hash 6b8e6e4aa44f3fe1561232a6c2f1d4eb parsl.dataflow.memoization INFO: Task 4579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4579 try 0 launched on executor multi with executor id 4580 parsl.dataflow.dflow INFO: Standard output for task 4579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ee3fba0f-cf0d-4b7c-b53c-f188c38598d3_isrForCrosstalkSources_855401_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ee3fba0f-cf0d-4b7c-b53c-f188c38598d3_isrForCrosstalkSources_855401_21.stderr parsl.dataflow.dflow DEBUG: Task 4580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4580 has memoization hash ac05b34b4e7ca23746a61363ea97dddd parsl.dataflow.memoization INFO: Task 4580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4580 try 0 launched on executor multi with executor id 4581 parsl.dataflow.dflow INFO: Standard output for task 4580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a9ca11e5-a1e2-4026-9de9-def1bd62739d_isrForCrosstalkSources_855426_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a9ca11e5-a1e2-4026-9de9-def1bd62739d_isrForCrosstalkSources_855426_22.stderr parsl.dataflow.dflow DEBUG: Task 4581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4581 has memoization hash f6833b4e3314ecc83b2b5ced631aa974 parsl.dataflow.memoization INFO: Task 4581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4581 try 0 launched on executor multi with executor id 4582 parsl.dataflow.dflow INFO: Standard output for task 4581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2dcdf1ef-1126-40a5-961a-15139ce9efc7_isrForCrosstalkSources_855388_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2dcdf1ef-1126-40a5-961a-15139ce9efc7_isrForCrosstalkSources_855388_14.stderr parsl.dataflow.dflow DEBUG: Task 4582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4582 has memoization hash dca02eaea0ed9c033dba61990f2e4358 parsl.dataflow.memoization INFO: Task 4582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4582 try 0 launched on executor multi with executor id 4583 parsl.dataflow.dflow INFO: Standard output for task 4582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/8be35f57-e603-408f-b79c-2bf2128c7064_isrForCrosstalkSources_855396_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/8be35f57-e603-408f-b79c-2bf2128c7064_isrForCrosstalkSources_855396_46.stderr parsl.dataflow.dflow DEBUG: Task 4583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4583 has memoization hash 6c4102dd1f660397321775cf3a1889a1 parsl.dataflow.memoization INFO: Task 4583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4583 try 0 launched on executor multi with executor id 4584 parsl.dataflow.dflow INFO: Standard output for task 4583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/66026946-f5ce-4685-9c1f-1a54f603c0cf_isrForCrosstalkSources_855450_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/66026946-f5ce-4685-9c1f-1a54f603c0cf_isrForCrosstalkSources_855450_24.stderr parsl.dataflow.dflow DEBUG: Task 4584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4584 has memoization hash eb8f12e16a41e8240c5d4b9a5a34e6a9 parsl.dataflow.memoization INFO: Task 4584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4584 try 0 launched on executor multi with executor id 4585 parsl.dataflow.dflow INFO: Standard output for task 4584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/235cad24-e8d5-4941-9b4c-aeef1a182b00_isrForCrosstalkSources_855448_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/235cad24-e8d5-4941-9b4c-aeef1a182b00_isrForCrosstalkSources_855448_13.stderr parsl.dataflow.dflow DEBUG: Task 4585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4585 has memoization hash 6492de2308efd3eba7df1ac06a793666 parsl.dataflow.memoization INFO: Task 4585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4585 try 0 launched on executor multi with executor id 4586 parsl.dataflow.dflow INFO: Standard output for task 4585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9e50d0de-0159-468a-8af2-7f6e5dda40a1_isrForCrosstalkSources_855381_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9e50d0de-0159-468a-8af2-7f6e5dda40a1_isrForCrosstalkSources_855381_53.stderr parsl.dataflow.dflow DEBUG: Task 4586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4586 has memoization hash 32683eefc8e950f6895275bbcf412148 parsl.dataflow.memoization INFO: Task 4586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4586 try 0 launched on executor multi with executor id 4587 parsl.dataflow.dflow INFO: Standard output for task 4586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6d8a47c-d860-4372-b2a0-a247d25e594f_isrForCrosstalkSources_855457_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6d8a47c-d860-4372-b2a0-a247d25e594f_isrForCrosstalkSources_855457_47.stderr parsl.dataflow.dflow DEBUG: Task 4587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4587 has memoization hash 018d393f37e2f3b8fcbd5bf5a96874ee parsl.dataflow.memoization INFO: Task 4587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4587 try 0 launched on executor multi with executor id 4588 parsl.dataflow.dflow INFO: Standard output for task 4587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7df9ab20-8abb-4bbc-88ca-e5e9bf69971c_isrForCrosstalkSources_855405_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7df9ab20-8abb-4bbc-88ca-e5e9bf69971c_isrForCrosstalkSources_855405_35.stderr parsl.dataflow.dflow DEBUG: Task 4588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4588 has memoization hash 47089ea0fc2661317aed181c58c647eb parsl.dataflow.memoization INFO: Task 4588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4588 try 0 launched on executor multi with executor id 4589 parsl.dataflow.dflow INFO: Standard output for task 4588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f6573a17-bce2-4877-b239-33d9c4e8312d_isrForCrosstalkSources_855397_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f6573a17-bce2-4877-b239-33d9c4e8312d_isrForCrosstalkSources_855397_18.stderr parsl.dataflow.dflow DEBUG: Task 4589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4589 has memoization hash 25ed212c8faac55504a6d70bac64904a parsl.dataflow.memoization INFO: Task 4589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4589 try 0 launched on executor multi with executor id 4590 parsl.dataflow.dflow INFO: Standard output for task 4589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d29510e4-00e2-4314-a90f-f5290dee5c53_isrForCrosstalkSources_855392_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d29510e4-00e2-4314-a90f-f5290dee5c53_isrForCrosstalkSources_855392_12.stderr parsl.dataflow.dflow DEBUG: Task 4590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4590 has memoization hash 4c153e1e26ee2cea935ed4900d4e22d5 parsl.dataflow.memoization INFO: Task 4590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4590 try 0 launched on executor multi with executor id 4591 parsl.dataflow.dflow INFO: Standard output for task 4590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/37ba2fc4-8e69-4286-a233-af04148960ea_isrForCrosstalkSources_855439_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/37ba2fc4-8e69-4286-a233-af04148960ea_isrForCrosstalkSources_855439_35.stderr parsl.dataflow.dflow DEBUG: Task 4591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4591 has memoization hash e91a6288ca890e755f3c6a5921428281 parsl.dataflow.memoization INFO: Task 4591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4591 try 0 launched on executor multi with executor id 4592 parsl.dataflow.dflow INFO: Standard output for task 4591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3c2f8fcb-f688-4e88-8490-3bdf2f0000e2_isrForCrosstalkSources_855450_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3c2f8fcb-f688-4e88-8490-3bdf2f0000e2_isrForCrosstalkSources_855450_49.stderr parsl.dataflow.dflow DEBUG: Task 4592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4592 has memoization hash e7cc1cf20e011fbb26f1a243f7347902 parsl.dataflow.memoization INFO: Task 4592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4592 try 0 launched on executor multi with executor id 4593 parsl.dataflow.dflow INFO: Standard output for task 4592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f3c75a99-3c8f-4ada-b9d2-a59af95e1e2f_isrForCrosstalkSources_855414_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f3c75a99-3c8f-4ada-b9d2-a59af95e1e2f_isrForCrosstalkSources_855414_60.stderr parsl.dataflow.dflow DEBUG: Task 4593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4593 has memoization hash 8a08da452a787ccb409fec99821e9357 parsl.dataflow.memoization INFO: Task 4593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4593 try 0 launched on executor multi with executor id 4594 parsl.dataflow.dflow INFO: Standard output for task 4593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f625ccb0-32cd-4074-9420-9459da26716d_isrForCrosstalkSources_855381_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f625ccb0-32cd-4074-9420-9459da26716d_isrForCrosstalkSources_855381_3.stderr parsl.dataflow.dflow DEBUG: Task 4594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4594 has memoization hash c222278222c936df8715a116a1aaf175 parsl.dataflow.memoization INFO: Task 4594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4594 try 0 launched on executor multi with executor id 4595 parsl.dataflow.dflow INFO: Standard output for task 4594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/09388ca4-1081-4c3e-97fb-dec1a284135a_isrForCrosstalkSources_855446_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/09388ca4-1081-4c3e-97fb-dec1a284135a_isrForCrosstalkSources_855446_55.stderr parsl.dataflow.dflow DEBUG: Task 4595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4595 has memoization hash ee28af9618978f1024fb4cfe27cd8bee parsl.dataflow.memoization INFO: Task 4595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4595 try 0 launched on executor multi with executor id 4596 parsl.dataflow.dflow INFO: Standard output for task 4595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/a11815a7-471e-4f3b-9788-099023807b4f_isrForCrosstalkSources_855423_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/a11815a7-471e-4f3b-9788-099023807b4f_isrForCrosstalkSources_855423_55.stderr parsl.dataflow.dflow DEBUG: Task 4596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4596 has memoization hash 1fbb6b06dd9efd5a4883430dd3a5419d parsl.dataflow.memoization INFO: Task 4596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4596 try 0 launched on executor multi with executor id 4597 parsl.dataflow.dflow INFO: Standard output for task 4596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/815191dc-9319-4679-abbf-21e04655c16e_isrForCrosstalkSources_855411_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/815191dc-9319-4679-abbf-21e04655c16e_isrForCrosstalkSources_855411_21.stderr parsl.dataflow.dflow DEBUG: Task 4597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4597 has memoization hash aca669477653a15802131742d1387e1f parsl.dataflow.memoization INFO: Task 4597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4597 try 0 launched on executor multi with executor id 4598 parsl.dataflow.dflow INFO: Standard output for task 4597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d3d29bad-8e46-429d-ba5a-89bbde93e043_isrForCrosstalkSources_855448_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d3d29bad-8e46-429d-ba5a-89bbde93e043_isrForCrosstalkSources_855448_59.stderr parsl.dataflow.dflow DEBUG: Task 4598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4598 has memoization hash 7d81909b01f2f140b8aaa00d9b1ffdee parsl.dataflow.memoization INFO: Task 4598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4598 try 0 launched on executor multi with executor id 4599 parsl.dataflow.dflow INFO: Standard output for task 4598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b6f0be86-f6fa-4db2-a681-0598e35e87ab_isrForCrosstalkSources_855448_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b6f0be86-f6fa-4db2-a681-0598e35e87ab_isrForCrosstalkSources_855448_61.stderr parsl.dataflow.dflow DEBUG: Task 4599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4599 has memoization hash 003415eaf564319dc97fd0926b3c345e parsl.dataflow.memoization INFO: Task 4599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 275f0f019c9793b4bcbc51256b505112 with result from task 535 parsl.dataflow.dflow INFO: Parsl task 4599 try 0 launched on executor multi with executor id 4600 parsl.dataflow.dflow INFO: Standard output for task 4599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a8dc423a-458f-4f46-86c5-17842194fb1b_isrForCrosstalkSources_855439_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a8dc423a-458f-4f46-86c5-17842194fb1b_isrForCrosstalkSources_855439_38.stderr parsl.dataflow.dflow DEBUG: Task 4600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4600 has memoization hash d3f664bbfe7bbddfaabf8ac537c8063a parsl.dataflow.memoization INFO: Task 4600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7913d145-472b-440f-ad83-3b92732aec8f_isrForCrosstalkSources_855431_42.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7913d145-472b-440f-ad83-3b92732aec8f_isrForCrosstalkSources_855431_42.stderr parsl.dataflow.dflow INFO: Parsl task 4600 try 0 launched on executor multi with executor id 4601 parsl.dataflow.dflow INFO: Standard output for task 4600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e0514520-baf0-4689-b876-dc5ee49540fe_isrForCrosstalkSources_855409_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e0514520-baf0-4689-b876-dc5ee49540fe_isrForCrosstalkSources_855409_36.stderr parsl.dataflow.dflow DEBUG: Task 4601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4601 has memoization hash 57a67d85ce41d6a34c89b112c8e4df83 parsl.dataflow.memoization INFO: Task 4601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4601 try 0 launched on executor multi with executor id 4602 parsl.dataflow.dflow INFO: Standard output for task 4601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f33898bf-c857-43b2-bfbc-c79e1383d7b2_isrForCrosstalkSources_855454_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f33898bf-c857-43b2-bfbc-c79e1383d7b2_isrForCrosstalkSources_855454_41.stderr parsl.dataflow.dflow DEBUG: Task 4602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4602 has memoization hash 3c4146eb9f4c48e3964e914f5115e912 parsl.dataflow.memoization INFO: Task 4602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4602 try 0 launched on executor multi with executor id 4603 parsl.dataflow.dflow INFO: Standard output for task 4602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/426439e4-f3be-4a78-a1fc-b7fe149a21b2_isrForCrosstalkSources_855429_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/426439e4-f3be-4a78-a1fc-b7fe149a21b2_isrForCrosstalkSources_855429_29.stderr parsl.dataflow.dflow DEBUG: Task 4603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4603 has memoization hash a6c2a35c3576786dfb048ac2c3d408b8 parsl.dataflow.memoization INFO: Task 4603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4603 try 0 launched on executor multi with executor id 4604 parsl.dataflow.dflow INFO: Standard output for task 4603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2ea50818-4b31-4e8e-88dd-c40b836a810b_isrForCrosstalkSources_855410_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2ea50818-4b31-4e8e-88dd-c40b836a810b_isrForCrosstalkSources_855410_59.stderr parsl.dataflow.dflow DEBUG: Task 4604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4604 has memoization hash 9c0ac105b21e186f6faa45d8f4c95e1f parsl.dataflow.memoization INFO: Task 4604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4604 try 0 launched on executor multi with executor id 4605 parsl.dataflow.dflow INFO: Standard output for task 4604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/84d76693-f913-486c-aea5-94f3e97fb20d_isrForCrosstalkSources_855456_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/84d76693-f913-486c-aea5-94f3e97fb20d_isrForCrosstalkSources_855456_2.stderr parsl.dataflow.dflow DEBUG: Task 4605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4605 has memoization hash ff94133735ad140e7258ca7e803e9823 parsl.dataflow.memoization INFO: Task 4605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4605 try 0 launched on executor multi with executor id 4606 parsl.dataflow.dflow INFO: Standard output for task 4605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/64f1ef32-217a-4789-8dce-f0d99cf5b930_isrForCrosstalkSources_855443_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/64f1ef32-217a-4789-8dce-f0d99cf5b930_isrForCrosstalkSources_855443_2.stderr parsl.dataflow.dflow DEBUG: Task 4606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4606 has memoization hash 44f41413fb052067139d10397234cc6d parsl.dataflow.memoization INFO: Task 4606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4606 try 0 launched on executor multi with executor id 4607 parsl.dataflow.dflow INFO: Standard output for task 4606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ab782550-12b4-4181-aed6-d419beac6967_isrForCrosstalkSources_855401_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ab782550-12b4-4181-aed6-d419beac6967_isrForCrosstalkSources_855401_44.stderr parsl.dataflow.dflow DEBUG: Task 4607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4607 has memoization hash 3ee0a4d1967794757f08d3fb8eac71b5 parsl.dataflow.memoization INFO: Task 4607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4607 try 0 launched on executor multi with executor id 4608 parsl.dataflow.dflow INFO: Standard output for task 4607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/90189218-1e20-4b45-8bd6-14a30a059b57_isrForCrosstalkSources_855452_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/90189218-1e20-4b45-8bd6-14a30a059b57_isrForCrosstalkSources_855452_17.stderr parsl.dataflow.dflow DEBUG: Task 4608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4608 has memoization hash 4d40a5637e45f8dbc98b5b1e3a5dd47c parsl.dataflow.memoization INFO: Task 4608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4608 try 0 launched on executor multi with executor id 4609 parsl.dataflow.dflow INFO: Standard output for task 4608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f336ec43-c5be-4f11-bc41-6082d7717cde_isrForCrosstalkSources_855384_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f336ec43-c5be-4f11-bc41-6082d7717cde_isrForCrosstalkSources_855384_60.stderr parsl.dataflow.dflow DEBUG: Task 4609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4609 has memoization hash 8ce3abb6595f1b833cf63a22722ee7e0 parsl.dataflow.memoization INFO: Task 4609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4609 try 0 launched on executor multi with executor id 4610 parsl.dataflow.dflow INFO: Standard output for task 4609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/07e56acd-6204-49a4-b65f-7fad71d4a883_isrForCrosstalkSources_855391_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/07e56acd-6204-49a4-b65f-7fad71d4a883_isrForCrosstalkSources_855391_33.stderr parsl.dataflow.dflow DEBUG: Task 4610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4610 has memoization hash 8c0dda359fefd35ff0ea03c3dbcd4546 parsl.dataflow.memoization INFO: Task 4610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4610 try 0 launched on executor multi with executor id 4611 parsl.dataflow.dflow INFO: Standard output for task 4610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/935e514f-846b-4bdb-aa8d-9246426916aa_isrForCrosstalkSources_855432_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/935e514f-846b-4bdb-aa8d-9246426916aa_isrForCrosstalkSources_855432_24.stderr parsl.dataflow.dflow DEBUG: Task 4611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4611 has memoization hash 8ce6db4f8dd2d0c06d0b3b34af3cc0ed parsl.dataflow.memoization INFO: Task 4611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4611 try 0 launched on executor multi with executor id 4612 parsl.dataflow.dflow INFO: Standard output for task 4611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0ffa60ed-0105-414e-baf0-9859a3ceeb3a_isrForCrosstalkSources_855443_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0ffa60ed-0105-414e-baf0-9859a3ceeb3a_isrForCrosstalkSources_855443_45.stderr parsl.dataflow.dflow DEBUG: Task 4612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4612 has memoization hash 0792eb3f721c9f30c11354da17b39369 parsl.dataflow.memoization INFO: Task 4612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4612 try 0 launched on executor multi with executor id 4613 parsl.dataflow.dflow INFO: Standard output for task 4612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8cc66167-8d38-4f72-8410-f14e5f0479fd_isrForCrosstalkSources_855425_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8cc66167-8d38-4f72-8410-f14e5f0479fd_isrForCrosstalkSources_855425_33.stderr parsl.dataflow.dflow DEBUG: Task 4613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4613 has memoization hash e610a29b01d93373eb8697fc406e5555 parsl.dataflow.memoization INFO: Task 4613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4613 try 0 launched on executor multi with executor id 4614 parsl.dataflow.dflow INFO: Standard output for task 4613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d206a2f5-4eb4-48f6-802f-8fe1bb812dd3_isrForCrosstalkSources_855459_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d206a2f5-4eb4-48f6-802f-8fe1bb812dd3_isrForCrosstalkSources_855459_29.stderr parsl.dataflow.dflow DEBUG: Task 4614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4614 has memoization hash 9508e3ea3b3813af562ccfa61bd2d9f9 parsl.dataflow.memoization INFO: Task 4614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4614 try 0 launched on executor multi with executor id 4615 parsl.dataflow.dflow INFO: Standard output for task 4614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/82b7d3ab-c68a-4e2c-9997-b8e8d2d2843f_isrForCrosstalkSources_855432_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/82b7d3ab-c68a-4e2c-9997-b8e8d2d2843f_isrForCrosstalkSources_855432_42.stderr parsl.dataflow.dflow DEBUG: Task 4615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4615 has memoization hash 5aaec4b2003f64eaf82672c6f84e55c5 parsl.dataflow.memoization INFO: Task 4615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4615 try 0 launched on executor multi with executor id 4616 parsl.dataflow.dflow INFO: Standard output for task 4615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/3183bc89-790e-4c26-b92f-017d11f29485_isrForCrosstalkSources_855399_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/3183bc89-790e-4c26-b92f-017d11f29485_isrForCrosstalkSources_855399_30.stderr parsl.dataflow.dflow DEBUG: Task 4616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4616 has memoization hash dbdae70e44ee0a0950070932399b3d6f parsl.dataflow.memoization INFO: Task 4616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c417feee61a8a54b46db0ca16bcb43dd with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fca1425d-1f51-4d0c-b1ac-c194939d25a2_isrForCrosstalkSources_855423_40.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fca1425d-1f51-4d0c-b1ac-c194939d25a2_isrForCrosstalkSources_855423_40.stderr parsl.dataflow.dflow INFO: Parsl task 4616 try 0 launched on executor multi with executor id 4617 parsl.dataflow.dflow INFO: Standard output for task 4616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67687c00-209b-43e5-af49-847c4f4ed688_isrForCrosstalkSources_855382_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67687c00-209b-43e5-af49-847c4f4ed688_isrForCrosstalkSources_855382_59.stderr parsl.dataflow.dflow DEBUG: Task 4617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4617 has memoization hash 572dbc9f5ae840a0cb20a698b92ddc1e parsl.dataflow.memoization INFO: Task 4617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4617 try 0 launched on executor multi with executor id 4618 parsl.dataflow.dflow INFO: Standard output for task 4617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0205d0b1-0773-484c-8fb7-245438dfb393_isrForCrosstalkSources_855441_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0205d0b1-0773-484c-8fb7-245438dfb393_isrForCrosstalkSources_855441_55.stderr parsl.dataflow.dflow DEBUG: Task 4618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4618 has memoization hash 756becf97054e186a63ba338ab9e03d6 parsl.dataflow.memoization INFO: Task 4618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4618 try 0 launched on executor multi with executor id 4619 parsl.dataflow.dflow INFO: Standard output for task 4618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/338482d3-5572-431e-a120-690be1155d19_isrForCrosstalkSources_855413_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/338482d3-5572-431e-a120-690be1155d19_isrForCrosstalkSources_855413_11.stderr parsl.dataflow.dflow DEBUG: Task 4619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4619 has memoization hash 2c4c2ce0c4ca4fd89db7fc8de66c364d parsl.dataflow.memoization INFO: Task 4619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:54 proc_lsst.multi:170 [INFO] [multi] job_id 16964198 proc_lsst.multi INFO: [multi] job_id 16964198 2024-03-09 14:51:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-09 14:51:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964198 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964198 parsl.executors.status_handling DEBUG: Launched block 172 on executor multi with job ID 16964198 parsl.executors.status_handling INFO: Allocated block ID 173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-173 2024-03-09 14:51:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4619 try 0 launched on executor multi with executor id 4620 parsl.dataflow.dflow INFO: Standard output for task 4619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cd7d818a-aa5f-48b0-bb4e-669077c6b061_isrForCrosstalkSources_855456_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cd7d818a-aa5f-48b0-bb4e-669077c6b061_isrForCrosstalkSources_855456_11.stderr parsl.dataflow.dflow DEBUG: Task 4620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4620 has memoization hash d0440fa4c91c5a30df6dde7a17dabb2d parsl.dataflow.memoization INFO: Task 4620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4620 try 0 launched on executor multi with executor id 4621 parsl.dataflow.dflow INFO: Standard output for task 4620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee332a93-8319-4f0d-84f9-656c2bae7d78_isrForCrosstalkSources_855443_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee332a93-8319-4f0d-84f9-656c2bae7d78_isrForCrosstalkSources_855443_3.stderr parsl.dataflow.dflow DEBUG: Task 4621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4621 has memoization hash 957d0bfcdd1f0b4491c75ac90c75b320 parsl.dataflow.memoization INFO: Task 4621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4621 try 0 launched on executor multi with executor id 4622 parsl.dataflow.dflow INFO: Standard output for task 4621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d6ae350-ab05-4998-b357-95a5359d2943_isrForCrosstalkSources_855439_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d6ae350-ab05-4998-b357-95a5359d2943_isrForCrosstalkSources_855439_49.stderr parsl.dataflow.dflow DEBUG: Task 4622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4622 has memoization hash e58bf96ed78ccc27cdae1818512ced05 parsl.dataflow.memoization INFO: Task 4622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4622 try 0 launched on executor multi with executor id 4623 parsl.dataflow.dflow INFO: Standard output for task 4622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2427d7dd-ac23-4277-b6ca-ad97cbb3625c_isrForCrosstalkSources_855417_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2427d7dd-ac23-4277-b6ca-ad97cbb3625c_isrForCrosstalkSources_855417_7.stderr parsl.dataflow.dflow DEBUG: Task 4623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4623 has memoization hash a33ef817168cc366348dbcfbb4711319 parsl.dataflow.memoization INFO: Task 4623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4623 try 0 launched on executor multi with executor id 4624 parsl.dataflow.dflow INFO: Standard output for task 4623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0a5f2a0e-7a87-4ebc-a204-4606ff1c8040_isrForCrosstalkSources_855385_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0a5f2a0e-7a87-4ebc-a204-4606ff1c8040_isrForCrosstalkSources_855385_11.stderr parsl.dataflow.dflow DEBUG: Task 4624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4624 has memoization hash 5773373abdef39125401192637742c8f parsl.dataflow.memoization INFO: Task 4624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4624 try 0 launched on executor multi with executor id 4625 parsl.dataflow.dflow INFO: Standard output for task 4624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f5a3b77a-168e-4a5c-bfaf-dccb4b5e230e_isrForCrosstalkSources_855419_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f5a3b77a-168e-4a5c-bfaf-dccb4b5e230e_isrForCrosstalkSources_855419_43.stderr parsl.dataflow.dflow DEBUG: Task 4625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4625 has memoization hash a409c3e07c3257e55b8bdb8988222c65 parsl.dataflow.memoization INFO: Task 4625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4625 try 0 launched on executor multi with executor id 4626 parsl.dataflow.dflow INFO: Standard output for task 4625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6ee943e9-49f9-42c3-a5a5-16c14afe5ffd_isrForCrosstalkSources_855432_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6ee943e9-49f9-42c3-a5a5-16c14afe5ffd_isrForCrosstalkSources_855432_14.stderr parsl.dataflow.dflow DEBUG: Task 4626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4626 has memoization hash 02829b740d8c26e5a7c8eb60882a1aff parsl.dataflow.memoization INFO: Task 4626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4626 try 0 launched on executor multi with executor id 4627 parsl.dataflow.dflow INFO: Standard output for task 4626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/07686f91-0a62-4690-b9c5-54fb8da241be_isrForCrosstalkSources_855426_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/07686f91-0a62-4690-b9c5-54fb8da241be_isrForCrosstalkSources_855426_35.stderr parsl.dataflow.dflow DEBUG: Task 4627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4627 has memoization hash b22ec8d2c2a90a69b92390f4d003f24f parsl.dataflow.memoization INFO: Task 4627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4627 try 0 launched on executor multi with executor id 4628 parsl.dataflow.dflow INFO: Standard output for task 4627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/180e1f38-9be3-4c1d-afe0-bfdb079b955f_isrForCrosstalkSources_855452_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/180e1f38-9be3-4c1d-afe0-bfdb079b955f_isrForCrosstalkSources_855452_37.stderr parsl.dataflow.dflow DEBUG: Task 4628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4628 has memoization hash 72334edd11872489336b903b4fb29f05 parsl.dataflow.memoization INFO: Task 4628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4628 try 0 launched on executor multi with executor id 4629 parsl.dataflow.dflow INFO: Standard output for task 4628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a20fa69e-5cfb-4b8f-9565-2ad4861e19b3_isrForCrosstalkSources_855385_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a20fa69e-5cfb-4b8f-9565-2ad4861e19b3_isrForCrosstalkSources_855385_23.stderr parsl.dataflow.dflow DEBUG: Task 4629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4629 has memoization hash 851a09ff74b36d32f826451a2f2b1e32 parsl.dataflow.memoization INFO: Task 4629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4629 try 0 launched on executor multi with executor id 4630 parsl.dataflow.dflow INFO: Standard output for task 4629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d40a58-ca35-4ebc-8b00-a3274f998057_isrForCrosstalkSources_855448_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d40a58-ca35-4ebc-8b00-a3274f998057_isrForCrosstalkSources_855448_41.stderr parsl.dataflow.dflow DEBUG: Task 4630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4630 has memoization hash 990738aa77db087b9639088d2da2164a parsl.dataflow.memoization INFO: Task 4630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4630 try 0 launched on executor multi with executor id 4631 parsl.dataflow.dflow INFO: Standard output for task 4630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a7eaac77-fa83-4774-a94b-f246c79cba97_isrForCrosstalkSources_855441_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a7eaac77-fa83-4774-a94b-f246c79cba97_isrForCrosstalkSources_855441_41.stderr parsl.dataflow.dflow DEBUG: Task 4631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4631 has memoization hash 98b23605be244eb24a60cf695ad62f09 parsl.dataflow.memoization INFO: Task 4631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4631 try 0 launched on executor multi with executor id 4632 parsl.dataflow.dflow INFO: Standard output for task 4631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ce66ad49-af30-494f-8f75-5e097fd5abb7_isrForCrosstalkSources_855459_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ce66ad49-af30-494f-8f75-5e097fd5abb7_isrForCrosstalkSources_855459_41.stderr parsl.dataflow.dflow DEBUG: Task 4632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4632 has memoization hash 72266f80bc57d1e3e87e0364e453a449 parsl.dataflow.memoization INFO: Task 4632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4632 try 0 launched on executor multi with executor id 4633 parsl.dataflow.dflow INFO: Standard output for task 4632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/334be8ba-3ab3-4e6e-8944-793c746f82ef_isrForCrosstalkSources_855396_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/334be8ba-3ab3-4e6e-8944-793c746f82ef_isrForCrosstalkSources_855396_19.stderr parsl.dataflow.dflow DEBUG: Task 4633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4633 has memoization hash 08bba0a81540cde44183154e24325e79 parsl.dataflow.memoization INFO: Task 4633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4633 try 0 launched on executor multi with executor id 4634 parsl.dataflow.dflow INFO: Standard output for task 4633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e3bca7a3-d3eb-4763-938c-8ebaf1e9bb22_isrForCrosstalkSources_855420_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e3bca7a3-d3eb-4763-938c-8ebaf1e9bb22_isrForCrosstalkSources_855420_20.stderr parsl.dataflow.dflow DEBUG: Task 4634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4634 has memoization hash ec11524aabe0f0c076f0242ac94a9bd4 parsl.dataflow.memoization INFO: Task 4634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4634 try 0 launched on executor multi with executor id 4635 parsl.dataflow.dflow INFO: Standard output for task 4634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6bda1318-5235-4922-8d6c-e3d68d72ad49_isrForCrosstalkSources_855385_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6bda1318-5235-4922-8d6c-e3d68d72ad49_isrForCrosstalkSources_855385_35.stderr parsl.dataflow.dflow DEBUG: Task 4635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4635 has memoization hash 5b8a0c2a953b96a7c69533db18973293 parsl.dataflow.memoization INFO: Task 4635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4635 try 0 launched on executor multi with executor id 4636 parsl.dataflow.dflow INFO: Standard output for task 4635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14b0a0c1-1714-4f62-b60a-fd1209b87b62_isrForCrosstalkSources_855442_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14b0a0c1-1714-4f62-b60a-fd1209b87b62_isrForCrosstalkSources_855442_54.stderr parsl.dataflow.dflow DEBUG: Task 4636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4636 has memoization hash e96482ee10b2cafde164768ab0fbd9c4 parsl.dataflow.memoization INFO: Task 4636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4636 try 0 launched on executor multi with executor id 4637 parsl.dataflow.dflow INFO: Standard output for task 4636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/e72966fd-a5cf-4376-befa-a9ec756f5a92_isrForCrosstalkSources_855427_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/e72966fd-a5cf-4376-befa-a9ec756f5a92_isrForCrosstalkSources_855427_30.stderr parsl.dataflow.dflow DEBUG: Task 4637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4637 has memoization hash cc3817bf7bd57df66197ece90d240e5a parsl.dataflow.memoization INFO: Task 4637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4637 try 0 launched on executor multi with executor id 4638 parsl.dataflow.dflow INFO: Standard output for task 4637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/1137f112-6e4b-4578-8758-cda3dde6cabb_isrForCrosstalkSources_855402_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/1137f112-6e4b-4578-8758-cda3dde6cabb_isrForCrosstalkSources_855402_23.stderr parsl.dataflow.dflow DEBUG: Task 4638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4638 has memoization hash c470fc817af0021a172e729c1994cf98 parsl.dataflow.memoization INFO: Task 4638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4638 try 0 launched on executor multi with executor id 4639 parsl.dataflow.dflow INFO: Standard output for task 4638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/0fc6b1a5-1c0a-4c29-8547-19709c2e1535_isrForCrosstalkSources_855379_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/0fc6b1a5-1c0a-4c29-8547-19709c2e1535_isrForCrosstalkSources_855379_33.stderr parsl.dataflow.dflow DEBUG: Task 4639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4639 has memoization hash bc62c25786846383baa7265c1e77640d parsl.dataflow.memoization INFO: Task 4639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4639 try 0 launched on executor multi with executor id 4640 parsl.dataflow.dflow INFO: Standard output for task 4639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/efe796fa-a243-411e-8c6d-18861b7a6d58_isrForCrosstalkSources_855413_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/efe796fa-a243-411e-8c6d-18861b7a6d58_isrForCrosstalkSources_855413_30.stderr parsl.dataflow.dflow DEBUG: Task 4640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4640 has memoization hash 6a50ebbc46be66966d43e2a5e5f0a232 parsl.dataflow.memoization INFO: Task 4640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4640 try 0 launched on executor multi with executor id 4641 parsl.dataflow.dflow INFO: Standard output for task 4640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d153609-df4e-48f0-b222-45336088aee0_isrForCrosstalkSources_855410_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d153609-df4e-48f0-b222-45336088aee0_isrForCrosstalkSources_855410_54.stderr parsl.dataflow.dflow DEBUG: Task 4641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4641 has memoization hash 8d7a14d7f4b214c1a6993bc16fafa506 parsl.dataflow.memoization INFO: Task 4641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4641 try 0 launched on executor multi with executor id 4642 parsl.dataflow.dflow INFO: Standard output for task 4641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0279d00e-146b-4758-8de1-72df7d54bbca_isrForCrosstalkSources_855403_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0279d00e-146b-4758-8de1-72df7d54bbca_isrForCrosstalkSources_855403_20.stderr parsl.dataflow.dflow DEBUG: Task 4642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4642 has memoization hash de7beb446cb57f9f6cbd9f4d1d42f46b parsl.dataflow.memoization INFO: Task 4642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4642 try 0 launched on executor multi with executor id 4643 parsl.dataflow.dflow INFO: Standard output for task 4642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5f10b2d-c0fd-4fb9-9f32-d8c56e79789e_isrForCrosstalkSources_855456_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5f10b2d-c0fd-4fb9-9f32-d8c56e79789e_isrForCrosstalkSources_855456_32.stderr parsl.dataflow.dflow DEBUG: Task 4643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4643 has memoization hash 27994ecf67b20d0b1d57ba4a4d6011df parsl.dataflow.memoization INFO: Task 4643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa8041d001863ec21d30cc6832fa082 with result from task 609 parsl.dataflow.dflow INFO: Parsl task 4643 try 0 launched on executor multi with executor id 4644 parsl.dataflow.dflow INFO: Standard output for task 4643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9ed0745b-960a-49f1-bfa1-4a4e258326fc_isrForCrosstalkSources_855426_2.stdout parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Standard error for task 4643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9ed0745b-960a-49f1-bfa1-4a4e258326fc_isrForCrosstalkSources_855426_2.stderr parsl.dataflow.dflow DEBUG: Task 4644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Task 4644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/06dd2db3-a8d6-4d93-8341-cc9175bca26b_isrForCrosstalkSources_855436_17.stdout parsl.dataflow.dflow DEBUG: Task 4644 set to pending state with AppFuture: parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/06dd2db3-a8d6-4d93-8341-cc9175bca26b_isrForCrosstalkSources_855436_17.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4644 has memoization hash e25b7f6c7bde9055fd9170a6bf0d645e parsl.dataflow.memoization INFO: Task 4644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:54 proc_lsst.multi:170 [INFO] [multi] job_id 16964199 proc_lsst.multi INFO: [multi] job_id 16964199 2024-03-09 14:51:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-09 14:51:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964199 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964199 parsl.executors.status_handling DEBUG: Launched block 173 on executor multi with job ID 16964199 parsl.executors.status_handling INFO: Allocated block ID 174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-174 2024-03-09 14:51:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4644 try 0 launched on executor multi with executor id 4645 parsl.dataflow.dflow INFO: Standard output for task 4644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/04002a52-aff6-420d-bbf0-090cb89fe646_isrForCrosstalkSources_855389_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/04002a52-aff6-420d-bbf0-090cb89fe646_isrForCrosstalkSources_855389_59.stderr parsl.dataflow.dflow DEBUG: Task 4645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4645 has memoization hash dc10ff9d1bf5dae397dcb29c4fad8ff2 parsl.dataflow.memoization INFO: Task 4645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a59aa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4645 try 0 launched on executor multi with executor id 4646 parsl.dataflow.dflow INFO: Standard output for task 4645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4fefa4b9-503d-4fe0-a8fb-f9510222779e_isrForCrosstalkSources_855430_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4fefa4b9-503d-4fe0-a8fb-f9510222779e_isrForCrosstalkSources_855430_30.stderr parsl.dataflow.dflow DEBUG: Task 4646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4646 has memoization hash c90332f424d0d3952dd930fce5bce2d8 parsl.dataflow.memoization INFO: Task 4646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4646 try 0 launched on executor multi with executor id 4647 parsl.dataflow.dflow INFO: Standard output for task 4646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/959b09ba-ebd3-46bc-9eed-04450cd2c0b1_isrForCrosstalkSources_855425_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/959b09ba-ebd3-46bc-9eed-04450cd2c0b1_isrForCrosstalkSources_855425_56.stderr parsl.dataflow.dflow DEBUG: Task 4647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4647 has memoization hash a3c88110cb8cdd37fc9c22597a119b23 parsl.dataflow.memoization INFO: Task 4647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4647 try 0 launched on executor multi with executor id 4648 parsl.dataflow.dflow INFO: Standard output for task 4647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c91ed46f-ec78-4234-9e6a-83771758e4e5_isrForCrosstalkSources_855401_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c91ed46f-ec78-4234-9e6a-83771758e4e5_isrForCrosstalkSources_855401_9.stderr parsl.dataflow.dflow DEBUG: Task 4648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4648 has memoization hash ce69af9c3306809a3acfb82c4a3fa1a1 parsl.dataflow.memoization INFO: Task 4648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4648 try 0 launched on executor multi with executor id 4649 parsl.dataflow.dflow INFO: Standard output for task 4648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8c13d640-3618-4fa8-a9af-a0fec33cf956_isrForCrosstalkSources_855447_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8c13d640-3618-4fa8-a9af-a0fec33cf956_isrForCrosstalkSources_855447_40.stderr parsl.dataflow.dflow DEBUG: Task 4649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4649 has memoization hash 843e3bc1ac9d07e8afd145c304793011 parsl.dataflow.memoization INFO: Task 4649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4649 try 0 launched on executor multi with executor id 4650 parsl.dataflow.dflow INFO: Standard output for task 4649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cb0f553a-f21a-43d0-b3c1-712134abd80e_isrForCrosstalkSources_855449_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cb0f553a-f21a-43d0-b3c1-712134abd80e_isrForCrosstalkSources_855449_9.stderr parsl.dataflow.dflow DEBUG: Task 4650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4650 has memoization hash 566717c6387a6461dcf6f45845128623 parsl.dataflow.memoization INFO: Task 4650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4650 try 0 launched on executor multi with executor id 4651 parsl.dataflow.dflow INFO: Standard output for task 4650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/80a659b8-934b-4bd4-aaea-d74688cb1802_isrForCrosstalkSources_855414_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/80a659b8-934b-4bd4-aaea-d74688cb1802_isrForCrosstalkSources_855414_18.stderr parsl.dataflow.dflow DEBUG: Task 4651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4651 has memoization hash 87105f69ed65f92017f25db392b26b3e parsl.dataflow.memoization INFO: Task 4651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4651 try 0 launched on executor multi with executor id 4652 parsl.dataflow.dflow INFO: Standard output for task 4651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ae37a3a5-583b-462d-b576-d04e345f4532_isrForCrosstalkSources_855440_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ae37a3a5-583b-462d-b576-d04e345f4532_isrForCrosstalkSources_855440_25.stderr parsl.dataflow.dflow DEBUG: Task 4652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4652 has memoization hash 48bcf6396fdd9d287c93d6d204048dea parsl.dataflow.memoization INFO: Task 4652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4652 try 0 launched on executor multi with executor id 4653 parsl.dataflow.dflow INFO: Standard output for task 4652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9e4056c3-822e-495b-b8c0-d3f482cdabc8_isrForCrosstalkSources_855400_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9e4056c3-822e-495b-b8c0-d3f482cdabc8_isrForCrosstalkSources_855400_48.stderr parsl.dataflow.dflow DEBUG: Task 4653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4653 has memoization hash 6f78b3885fd77cc39dae4ccd41217cbb parsl.dataflow.memoization INFO: Task 4653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4653 try 0 launched on executor multi with executor id 4654 parsl.dataflow.dflow INFO: Standard output for task 4653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e4569980-a41c-43dd-8a46-b9aa216357af_isrForCrosstalkSources_855431_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e4569980-a41c-43dd-8a46-b9aa216357af_isrForCrosstalkSources_855431_44.stderr parsl.dataflow.dflow DEBUG: Task 4654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4654 has memoization hash 068b77cfb910eefa38904c42631398b5 parsl.dataflow.memoization INFO: Task 4654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4654 try 0 launched on executor multi with executor id 4655 parsl.dataflow.dflow INFO: Standard output for task 4654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/29de2ba7-8342-4625-bb4d-5f50226f459d_isrForCrosstalkSources_855429_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/29de2ba7-8342-4625-bb4d-5f50226f459d_isrForCrosstalkSources_855429_15.stderr parsl.dataflow.dflow DEBUG: Task 4655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4655 has memoization hash 163bc6f5e6455ca896d9a5dec54f74ef parsl.dataflow.memoization INFO: Task 4655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e4a0c2981ef175a359766114ad4260 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9805e960-a540-4d4e-8109-9e2e792823c3_isrForCrosstalkSources_855409_49.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9805e960-a540-4d4e-8109-9e2e792823c3_isrForCrosstalkSources_855409_49.stderr parsl.dataflow.dflow INFO: Parsl task 4655 try 0 launched on executor multi with executor id 4656 parsl.dataflow.dflow INFO: Standard output for task 4655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/268e6567-ff5b-499d-9a8b-56e65bdea9e0_isrForCrosstalkSources_855412_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/268e6567-ff5b-499d-9a8b-56e65bdea9e0_isrForCrosstalkSources_855412_32.stderr parsl.dataflow.dflow DEBUG: Task 4656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4656 has memoization hash 348335a860e48a0f24a8ffd73fa671e7 parsl.dataflow.memoization INFO: Task 4656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4656 try 0 launched on executor multi with executor id 4657 parsl.dataflow.dflow INFO: Standard output for task 4656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c9fa249a-c0db-4e3a-875d-295fcfbb84d0_isrForCrosstalkSources_855412_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c9fa249a-c0db-4e3a-875d-295fcfbb84d0_isrForCrosstalkSources_855412_27.stderr parsl.dataflow.dflow DEBUG: Task 4657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4657 has memoization hash 28ffec2de7ab61506bea05c1a57f828b parsl.dataflow.memoization INFO: Task 4657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4657 try 0 launched on executor multi with executor id 4658 parsl.dataflow.dflow INFO: Standard output for task 4657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4445a8fd-2e1c-49a7-97e4-3ae46aa6a991_isrForCrosstalkSources_855400_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4445a8fd-2e1c-49a7-97e4-3ae46aa6a991_isrForCrosstalkSources_855400_31.stderr parsl.dataflow.dflow DEBUG: Task 4658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4658 has memoization hash 2d79111b04aa296ab87eae85d7771a1b parsl.dataflow.memoization INFO: Task 4658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4658 try 0 launched on executor multi with executor id 4659 parsl.dataflow.dflow INFO: Standard output for task 4658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7e26897a-50cd-4b6a-8b0a-1b2973bb9b78_isrForCrosstalkSources_855392_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7e26897a-50cd-4b6a-8b0a-1b2973bb9b78_isrForCrosstalkSources_855392_17.stderr parsl.dataflow.dflow DEBUG: Task 4659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4659 has memoization hash 6d52d954aa458317a40ad6ed5cde087f parsl.dataflow.memoization INFO: Task 4659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4659 try 0 launched on executor multi with executor id 4660 parsl.dataflow.dflow INFO: Standard output for task 4659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/75121019-7b23-4568-bd6b-9ca86b344b89_isrForCrosstalkSources_855389_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/75121019-7b23-4568-bd6b-9ca86b344b89_isrForCrosstalkSources_855389_1.stderr parsl.dataflow.dflow DEBUG: Task 4660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4660 has memoization hash 700db3b095075dc70152241c738f149d parsl.dataflow.memoization INFO: Task 4660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4660 try 0 launched on executor multi with executor id 4661 parsl.dataflow.dflow INFO: Standard output for task 4660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/bc6c7ae3-a516-4680-bffa-594acf1042c5_isrForCrosstalkSources_855422_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/bc6c7ae3-a516-4680-bffa-594acf1042c5_isrForCrosstalkSources_855422_46.stderr parsl.dataflow.dflow DEBUG: Task 4661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4661 has memoization hash 738f355ac844611e864f7a05fc8b6b90 parsl.dataflow.memoization INFO: Task 4661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4661 try 0 launched on executor multi with executor id 4662 parsl.dataflow.dflow INFO: Standard output for task 4661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d1f6e0bf-b155-4070-b0bb-d06251b9f348_isrForCrosstalkSources_855436_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d1f6e0bf-b155-4070-b0bb-d06251b9f348_isrForCrosstalkSources_855436_20.stderr parsl.dataflow.dflow DEBUG: Task 4662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4662 has memoization hash 934ff2a8a88348c828273307979e4c94 parsl.dataflow.memoization INFO: Task 4662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4662 try 0 launched on executor multi with executor id 4663 parsl.dataflow.dflow INFO: Standard output for task 4662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4f780375-ff6d-4384-9152-c721eb455acb_isrForCrosstalkSources_855383_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4f780375-ff6d-4384-9152-c721eb455acb_isrForCrosstalkSources_855383_19.stderr parsl.dataflow.dflow DEBUG: Task 4663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4663 has memoization hash 42c5460f44342d51fe100a2296d9a13e parsl.dataflow.memoization INFO: Task 4663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4663 try 0 launched on executor multi with executor id 4664 parsl.dataflow.dflow INFO: Standard output for task 4663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3cc05831-1bbb-49db-b63f-24f003d7281f_isrForCrosstalkSources_855453_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3cc05831-1bbb-49db-b63f-24f003d7281f_isrForCrosstalkSources_855453_53.stderr parsl.dataflow.dflow DEBUG: Task 4664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4664 has memoization hash 9fd9043adfb24cfabe7717920de7c1a6 parsl.dataflow.memoization INFO: Task 4664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4664 try 0 launched on executor multi with executor id 4665 parsl.dataflow.dflow INFO: Standard output for task 4664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a1769384-4dd2-423b-94e5-1902a6bce515_isrForCrosstalkSources_855408_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a1769384-4dd2-423b-94e5-1902a6bce515_isrForCrosstalkSources_855408_13.stderr parsl.dataflow.dflow DEBUG: Task 4665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4665 has memoization hash ccb3c36644a90ec66195aa82339cb1ce parsl.dataflow.memoization INFO: Task 4665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4665 try 0 launched on executor multi with executor id 4666 parsl.dataflow.dflow INFO: Standard output for task 4665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0fdeae7d-ce3d-4a99-9bda-8b8c90e8224c_isrForCrosstalkSources_855407_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0fdeae7d-ce3d-4a99-9bda-8b8c90e8224c_isrForCrosstalkSources_855407_22.stderr parsl.dataflow.dflow DEBUG: Task 4666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4666 has memoization hash e36c3a2ee6767e5c81c78dc782411b80 parsl.dataflow.memoization INFO: Task 4666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a5aee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4666 try 0 launched on executor multi with executor id 4667 parsl.dataflow.dflow INFO: Standard output for task 4666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/da224c52-22bc-4ad5-9efb-88c836e82046_isrForCrosstalkSources_855444_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/da224c52-22bc-4ad5-9efb-88c836e82046_isrForCrosstalkSources_855444_21.stderr parsl.dataflow.dflow DEBUG: Task 4667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4667 has memoization hash 1098b1864046ae68961bd888e2d26c3c parsl.dataflow.memoization INFO: Task 4667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4667 try 0 launched on executor multi with executor id 4668 parsl.dataflow.dflow INFO: Standard output for task 4667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/38c64183-2ca3-413c-88f0-c563c5e46d3e_isrForCrosstalkSources_855385_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/38c64183-2ca3-413c-88f0-c563c5e46d3e_isrForCrosstalkSources_855385_62.stderr parsl.dataflow.dflow DEBUG: Task 4668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4668 has memoization hash 713fd18b3aec2dfe2c9c46c14530dc99 parsl.dataflow.memoization INFO: Task 4668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4668 try 0 launched on executor multi with executor id 4669 parsl.dataflow.dflow INFO: Standard output for task 4668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/70c124d7-b15e-41ba-8de7-73232c81e948_isrForCrosstalkSources_855458_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/70c124d7-b15e-41ba-8de7-73232c81e948_isrForCrosstalkSources_855458_10.stderr parsl.dataflow.dflow DEBUG: Task 4669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4669 has memoization hash 519848c36dc000d75cf31f977afb4fb5 parsl.dataflow.memoization INFO: Task 4669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4669 try 0 launched on executor multi with executor id 4670 parsl.dataflow.dflow INFO: Standard output for task 4669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/51b70413-5885-46b3-80cf-52fd8b9e418a_isrForCrosstalkSources_855416_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/51b70413-5885-46b3-80cf-52fd8b9e418a_isrForCrosstalkSources_855416_1.stderr parsl.dataflow.dflow DEBUG: Task 4670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4670 has memoization hash 824173e64606a6bf2376c89e2a04dbbf parsl.dataflow.memoization INFO: Task 4670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4670 try 0 launched on executor multi with executor id 4671 parsl.dataflow.dflow INFO: Standard output for task 4670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b49ea62-ec8f-42d9-b837-05bda88f605b_isrForCrosstalkSources_855406_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b49ea62-ec8f-42d9-b837-05bda88f605b_isrForCrosstalkSources_855406_10.stderr parsl.dataflow.dflow DEBUG: Task 4671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4671 has memoization hash 6a20b69142b1ebbe74af839c1c35c63b parsl.dataflow.memoization INFO: Task 4671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4671 try 0 launched on executor multi with executor id 4672 parsl.dataflow.dflow INFO: Standard output for task 4671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fac24a84-df82-473a-92c4-36082f90bd29_isrForCrosstalkSources_855418_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fac24a84-df82-473a-92c4-36082f90bd29_isrForCrosstalkSources_855418_30.stderr parsl.dataflow.dflow DEBUG: Task 4672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4672 has memoization hash 41e7808c0846ba92362d673a854853e0 parsl.dataflow.memoization INFO: Task 4672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4672 try 0 launched on executor multi with executor id 4673 parsl.dataflow.dflow INFO: Standard output for task 4672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7f0832db-fedb-48e7-a1d4-c6260f30afe0_isrForCrosstalkSources_855431_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7f0832db-fedb-48e7-a1d4-c6260f30afe0_isrForCrosstalkSources_855431_29.stderr parsl.dataflow.dflow DEBUG: Task 4673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4673 has memoization hash 71fea54d348a3499914f74b2f8b09912 parsl.dataflow.memoization INFO: Task 4673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:54 proc_lsst.multi:170 [INFO] [multi] job_id 16964200 proc_lsst.multi INFO: [multi] job_id 16964200 2024-03-09 14:51:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-09 14:51:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964200 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964200 parsl.executors.status_handling DEBUG: Launched block 174 on executor multi with job ID 16964200 parsl.executors.status_handling INFO: Allocated block ID 175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-175 2024-03-09 14:51:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4673 try 0 launched on executor multi with executor id 4674 parsl.dataflow.dflow INFO: Standard output for task 4673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/4bb17698-05cf-43f0-9241-51e655f3d7b8_isrForCrosstalkSources_855414_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/4bb17698-05cf-43f0-9241-51e655f3d7b8_isrForCrosstalkSources_855414_62.stderr parsl.dataflow.dflow DEBUG: Task 4674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4674 has memoization hash 21dfe75dd009ce88f222ec4e06531059 parsl.dataflow.memoization INFO: Task 4674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9795d11263019faed06103ebbdabeb92 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c8c0d428-6360-485e-95b9-6e1037704bef_isrForCrosstalkSources_855415_35.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c8c0d428-6360-485e-95b9-6e1037704bef_isrForCrosstalkSources_855415_35.stderr parsl.dataflow.dflow INFO: Parsl task 4674 try 0 launched on executor multi with executor id 4675 parsl.dataflow.dflow INFO: Standard output for task 4674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/8d70b013-3289-4cc8-a630-cf0d960f90eb_isrForCrosstalkSources_855418_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/8d70b013-3289-4cc8-a630-cf0d960f90eb_isrForCrosstalkSources_855418_19.stderr parsl.dataflow.dflow DEBUG: Task 4675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4675 has memoization hash 10739b68fffa8a97a43b1ee80dd902a5 parsl.dataflow.memoization INFO: Task 4675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4675 try 0 launched on executor multi with executor id 4676 parsl.dataflow.dflow INFO: Standard output for task 4675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/05feaf69-a778-40e2-90ef-abcfb13f7046_isrForCrosstalkSources_855441_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/05feaf69-a778-40e2-90ef-abcfb13f7046_isrForCrosstalkSources_855441_33.stderr parsl.dataflow.dflow DEBUG: Task 4676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4676 has memoization hash 5ec62c3beabb39e17b8e4389cfd55bcb parsl.dataflow.memoization INFO: Task 4676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4676 try 0 launched on executor multi with executor id 4677 parsl.dataflow.dflow INFO: Standard output for task 4676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c3f6120c-5dde-4a57-83e4-75d208022470_isrForCrosstalkSources_855397_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c3f6120c-5dde-4a57-83e4-75d208022470_isrForCrosstalkSources_855397_19.stderr parsl.dataflow.dflow DEBUG: Task 4677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4677 has memoization hash 9c164d70eb331a162b792e5e05d8b499 parsl.dataflow.memoization INFO: Task 4677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4677 try 0 launched on executor multi with executor id 4678 parsl.dataflow.dflow INFO: Standard output for task 4677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f9601865-ee64-41a9-a6d5-05aaf00c69d7_isrForCrosstalkSources_855437_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f9601865-ee64-41a9-a6d5-05aaf00c69d7_isrForCrosstalkSources_855437_42.stderr parsl.dataflow.dflow DEBUG: Task 4678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4678 has memoization hash 1fc3a7ff3428cea6eb51c32ff29af3d7 parsl.dataflow.memoization INFO: Task 4678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4678 try 0 launched on executor multi with executor id 4679 parsl.dataflow.dflow INFO: Standard output for task 4678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6a635d25-faa5-42aa-b2d8-767caa22e1f3_isrForCrosstalkSources_855395_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6a635d25-faa5-42aa-b2d8-767caa22e1f3_isrForCrosstalkSources_855395_20.stderr parsl.dataflow.dflow DEBUG: Task 4679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4679 has memoization hash cc5d4eaa377e27162cd7f94b6212f7dd parsl.dataflow.memoization INFO: Task 4679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4679 try 0 launched on executor multi with executor id 4680 parsl.dataflow.dflow INFO: Standard output for task 4679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/bf2d83a2-61af-4383-ac9c-fe2f00645b17_isrForCrosstalkSources_855414_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/bf2d83a2-61af-4383-ac9c-fe2f00645b17_isrForCrosstalkSources_855414_6.stderr parsl.dataflow.dflow DEBUG: Task 4680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4680 has memoization hash a8b99d1d0d338c5b5cbddc930be8741c parsl.dataflow.memoization INFO: Task 4680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4680 try 0 launched on executor multi with executor id 4681 parsl.dataflow.dflow INFO: Standard output for task 4680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7c386a6f-33fc-49ab-b50c-c9c7ddb929f1_isrForCrosstalkSources_855419_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7c386a6f-33fc-49ab-b50c-c9c7ddb929f1_isrForCrosstalkSources_855419_12.stderr parsl.dataflow.dflow DEBUG: Task 4681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4681 has memoization hash 210cba202884e5872e527633f9ebd76f parsl.dataflow.memoization INFO: Task 4681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4681 try 0 launched on executor multi with executor id 4682 parsl.dataflow.dflow INFO: Standard output for task 4681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/832e078f-9688-4230-b477-bf3980fbe5c8_isrForCrosstalkSources_855411_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/832e078f-9688-4230-b477-bf3980fbe5c8_isrForCrosstalkSources_855411_50.stderr parsl.dataflow.dflow DEBUG: Task 4682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4682 has memoization hash 3efde83b38e30fd4eb46be695ff21ae2 parsl.dataflow.memoization INFO: Task 4682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4682 try 0 launched on executor multi with executor id 4683 parsl.dataflow.dflow INFO: Standard output for task 4682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9df66c54-108d-411f-b6c6-26c2ad2d6e4a_isrForCrosstalkSources_855406_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9df66c54-108d-411f-b6c6-26c2ad2d6e4a_isrForCrosstalkSources_855406_3.stderr parsl.dataflow.dflow DEBUG: Task 4683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4683 has memoization hash db06785e93761b238c6be04a0c0d1eea parsl.dataflow.memoization INFO: Task 4683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4683 try 0 launched on executor multi with executor id 4684 parsl.dataflow.dflow INFO: Standard output for task 4683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ce579ecf-41dd-428d-827f-8003b201e2d3_isrForCrosstalkSources_855388_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ce579ecf-41dd-428d-827f-8003b201e2d3_isrForCrosstalkSources_855388_27.stderr parsl.dataflow.dflow DEBUG: Task 4684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4684 has memoization hash cd1936792f40e6f35066a1f452f1f3fd parsl.dataflow.memoization INFO: Task 4684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4684 try 0 launched on executor multi with executor id 4685 parsl.dataflow.dflow INFO: Standard output for task 4684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/252c70bf-fb90-4147-9d04-5f35b051ed2d_isrForCrosstalkSources_855422_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/252c70bf-fb90-4147-9d04-5f35b051ed2d_isrForCrosstalkSources_855422_41.stderr parsl.dataflow.dflow DEBUG: Task 4685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4685 has memoization hash a80e872923dcc81da2a1d57c181bbb5e parsl.dataflow.memoization INFO: Task 4685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4685 try 0 launched on executor multi with executor id 4686 parsl.dataflow.dflow INFO: Standard output for task 4685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/27f8924c-c861-4c9e-8cba-7e9a70671276_isrForCrosstalkSources_855399_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/27f8924c-c861-4c9e-8cba-7e9a70671276_isrForCrosstalkSources_855399_45.stderr parsl.dataflow.dflow DEBUG: Task 4686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4686 has memoization hash 095434fb08c9bd69a8391f4cc3940d1f parsl.dataflow.memoization INFO: Task 4686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4686 try 0 launched on executor multi with executor id 4687 parsl.dataflow.dflow INFO: Standard output for task 4686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3ebb8cf8-2a25-47e4-bc5f-b7b0a187b845_isrForCrosstalkSources_855379_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3ebb8cf8-2a25-47e4-bc5f-b7b0a187b845_isrForCrosstalkSources_855379_21.stderr parsl.dataflow.dflow DEBUG: Task 4687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4687 has memoization hash afaf81e87a48332954d176d688bd7a82 parsl.dataflow.memoization INFO: Task 4687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4687 try 0 launched on executor multi with executor id 4688 parsl.dataflow.dflow INFO: Standard output for task 4687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e983a939-0e9a-4b2f-8938-3ce3ee439dc6_isrForCrosstalkSources_855392_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e983a939-0e9a-4b2f-8938-3ce3ee439dc6_isrForCrosstalkSources_855392_20.stderr parsl.dataflow.dflow DEBUG: Task 4688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4688 has memoization hash 8b9ac0f2526934d6c612c352f8488c82 parsl.dataflow.memoization INFO: Task 4688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4688 try 0 launched on executor multi with executor id 4689 parsl.dataflow.dflow INFO: Standard output for task 4688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/df1f1c32-02ad-444b-90e1-bdfff70ad4f0_isrForCrosstalkSources_855442_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/df1f1c32-02ad-444b-90e1-bdfff70ad4f0_isrForCrosstalkSources_855442_5.stderr parsl.dataflow.dflow DEBUG: Task 4689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4689 has memoization hash 711b75bea022e7bba7923898427210a1 parsl.dataflow.memoization INFO: Task 4689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4689 try 0 launched on executor multi with executor id 4690 parsl.dataflow.dflow INFO: Standard output for task 4689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9a60d519-a5ba-489c-b2fa-6a41e8b4b288_isrForCrosstalkSources_855399_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9a60d519-a5ba-489c-b2fa-6a41e8b4b288_isrForCrosstalkSources_855399_4.stderr parsl.dataflow.dflow DEBUG: Task 4690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4690 has memoization hash 279b71074a4073947518f4ce76d66743 parsl.dataflow.memoization INFO: Task 4690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4690 try 0 launched on executor multi with executor id 4691 parsl.dataflow.dflow INFO: Standard output for task 4690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6eabccf3-00f7-4b0e-8327-202e56e19f07_isrForCrosstalkSources_855424_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6eabccf3-00f7-4b0e-8327-202e56e19f07_isrForCrosstalkSources_855424_13.stderr parsl.dataflow.dflow DEBUG: Task 4691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4691 has memoization hash b6611c9f4bca197c1a5e7b69751000bf parsl.dataflow.memoization INFO: Task 4691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4691 try 0 launched on executor multi with executor id 4692 parsl.dataflow.dflow INFO: Standard output for task 4691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d5bb1139-187f-489e-a274-0b13fd6f326f_isrForCrosstalkSources_855429_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d5bb1139-187f-489e-a274-0b13fd6f326f_isrForCrosstalkSources_855429_5.stderr parsl.dataflow.dflow DEBUG: Task 4692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4692 has memoization hash a9ea3d661ba2e3eb6d1c3004f068a20e parsl.dataflow.memoization INFO: Task 4692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4692 try 0 launched on executor multi with executor id 4693 parsl.dataflow.dflow INFO: Standard output for task 4692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a08f2727-8580-404e-b6ce-35d4a5d974fd_isrForCrosstalkSources_855455_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a08f2727-8580-404e-b6ce-35d4a5d974fd_isrForCrosstalkSources_855455_44.stderr parsl.dataflow.dflow DEBUG: Task 4693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4693 has memoization hash fd1269f9e7958dac2caa61394eb495e4 parsl.dataflow.memoization INFO: Task 4693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4693 try 0 launched on executor multi with executor id 4694 parsl.dataflow.dflow INFO: Standard output for task 4693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/f012c92a-07d3-4a5d-9843-b37998372471_isrForCrosstalkSources_855447_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/f012c92a-07d3-4a5d-9843-b37998372471_isrForCrosstalkSources_855447_24.stderr parsl.dataflow.dflow DEBUG: Task 4694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4694 has memoization hash 89e2fad4f0913ac00216c8c7a28c3a85 parsl.dataflow.memoization INFO: Task 4694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4694 try 0 launched on executor multi with executor id 4695 parsl.dataflow.dflow INFO: Standard output for task 4694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e10254b9-20e0-4c9e-adac-6d9885639274_isrForCrosstalkSources_855451_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e10254b9-20e0-4c9e-adac-6d9885639274_isrForCrosstalkSources_855451_47.stderr parsl.dataflow.dflow DEBUG: Task 4695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4695 has memoization hash fe8516bfe0df0e3f53262abbbeb9a6d8 parsl.dataflow.memoization INFO: Task 4695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4695 try 0 launched on executor multi with executor id 4696 parsl.dataflow.dflow INFO: Standard output for task 4695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/678baf93-a650-4796-9b98-65a77b533ad6_isrForCrosstalkSources_855417_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/678baf93-a650-4796-9b98-65a77b533ad6_isrForCrosstalkSources_855417_46.stderr parsl.dataflow.dflow DEBUG: Task 4696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4696 has memoization hash 7591941f4130ee554637e9a958c78e69 parsl.dataflow.memoization INFO: Task 4696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4696 try 0 launched on executor multi with executor id 4697 parsl.dataflow.dflow INFO: Standard output for task 4696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/241b26a0-31bf-4f37-bedf-7c5e2d857ddf_isrForCrosstalkSources_855428_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/241b26a0-31bf-4f37-bedf-7c5e2d857ddf_isrForCrosstalkSources_855428_42.stderr parsl.dataflow.dflow DEBUG: Task 4697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4697 has memoization hash 9a79a9220cbd1627cf70a6a8fe0a6126 parsl.dataflow.memoization INFO: Task 4697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4697 try 0 launched on executor multi with executor id 4698 parsl.dataflow.dflow INFO: Standard output for task 4697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/33d58ec9-dd3e-4270-ac86-82dab582e41e_isrForCrosstalkSources_855389_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/33d58ec9-dd3e-4270-ac86-82dab582e41e_isrForCrosstalkSources_855389_9.stderr parsl.dataflow.dflow DEBUG: Task 4698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4698 has memoization hash f58bc25b8a729ea9d616530715976db5 parsl.dataflow.memoization INFO: Task 4698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4698 try 0 launched on executor multi with executor id 4699 parsl.dataflow.dflow INFO: Standard output for task 4698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/750ec061-4251-4a56-897d-9a53fdbea4e0_isrForCrosstalkSources_855391_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/750ec061-4251-4a56-897d-9a53fdbea4e0_isrForCrosstalkSources_855391_16.stderr parsl.dataflow.dflow DEBUG: Task 4699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4699 has memoization hash 1a4e5206519553b3f13498f605c299b8 parsl.dataflow.memoization INFO: Task 4699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4699 try 0 launched on executor multi with executor id 4700 parsl.dataflow.dflow INFO: Standard output for task 4699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/6b3aeefd-e049-49eb-9779-86b8ac5ea840_isrForCrosstalkSources_855414_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/6b3aeefd-e049-49eb-9779-86b8ac5ea840_isrForCrosstalkSources_855414_48.stderr parsl.dataflow.dflow DEBUG: Task 4700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4700 has memoization hash d3e8a589bfb7c6256753fd0677c7296f parsl.dataflow.memoization INFO: Task 4700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4700 try 0 launched on executor multi with executor id 4701 parsl.dataflow.dflow INFO: Standard output for task 4700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/be1784b3-a219-4766-adce-83ee325bbc56_isrForCrosstalkSources_855427_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/be1784b3-a219-4766-adce-83ee325bbc56_isrForCrosstalkSources_855427_56.stderr parsl.dataflow.dflow DEBUG: Task 4701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4701 submitted for App isrForCrosstalkSources, not waiting on any dependency 2024-03-09 14:51:54 proc_lsst.multi:170 [INFO] [multi] job_id 16964201 parsl.dataflow.dflow DEBUG: Task 4701 set to pending state with AppFuture: proc_lsst.multi INFO: [multi] job_id 16964201 2024-03-09 14:51:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 120 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-09 14:51:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964201 parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964201 parsl.dataflow.memoization DEBUG: Task 4701 has memoization hash 8588b252159f0fa4b71bf7a81bf60f24 parsl.executors.status_handling DEBUG: Launched block 175 on executor multi with job ID 16964201 parsl.dataflow.memoization INFO: Task 4701 had no result in cache parsl.executors.status_handling INFO: Allocated block ID 176 parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-176 2024-03-09 14:51:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-09 14:51:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4701 try 0 launched on executor multi with executor id 4702 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 4701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/83c220ad-b89f-4d43-9453-05ea87660d85_isrForCrosstalkSources_855414_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/83c220ad-b89f-4d43-9453-05ea87660d85_isrForCrosstalkSources_855414_38.stderr parsl.dataflow.dflow DEBUG: Task 4702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4702 has memoization hash e37d68effe300bfca1741ae4c0338470 parsl.dataflow.memoization INFO: Task 4702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 345ac789903185b79d5895eb3ea837d2 with result from task 612 parsl.dataflow.dflow INFO: Parsl task 4702 try 0 launched on executor multi with executor id 4703 parsl.dataflow.dflow INFO: Standard output for task 4702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/998ccdad-ec3c-4b01-8ea3-56b7d6ab9f95_isrForCrosstalkSources_855420_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/998ccdad-ec3c-4b01-8ea3-56b7d6ab9f95_isrForCrosstalkSources_855420_27.stderr parsl.dataflow.dflow DEBUG: Task 4703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4703 has memoization hash 7ab97a28696e78cc4e1ae84406e304ee parsl.dataflow.memoization INFO: Task 4703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/00935590-d862-4d3a-a8ec-302c01f4a54a_isrForCrosstalkSources_855424_14.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/00935590-d862-4d3a-a8ec-302c01f4a54a_isrForCrosstalkSources_855424_14.stderr parsl.dataflow.dflow INFO: Parsl task 4703 try 0 launched on executor multi with executor id 4704 parsl.dataflow.dflow INFO: Standard output for task 4703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ef237ae4-174b-432e-92ce-7b7369325e1a_isrForCrosstalkSources_855396_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ef237ae4-174b-432e-92ce-7b7369325e1a_isrForCrosstalkSources_855396_12.stderr parsl.dataflow.dflow DEBUG: Task 4704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4704 has memoization hash 9180a7a3829cbce28b8eb106a8551a3e parsl.dataflow.memoization INFO: Task 4704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4704 try 0 launched on executor multi with executor id 4705 parsl.dataflow.dflow INFO: Standard output for task 4704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3ee6dcf3-deaf-4b11-8eb2-58e6ea911f34_isrForCrosstalkSources_855385_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3ee6dcf3-deaf-4b11-8eb2-58e6ea911f34_isrForCrosstalkSources_855385_36.stderr parsl.dataflow.dflow DEBUG: Task 4705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4705 has memoization hash ca1f3001553fdf11c34d4b214999de89 parsl.dataflow.memoization INFO: Task 4705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4705 try 0 launched on executor multi with executor id 4706 parsl.dataflow.dflow INFO: Standard output for task 4705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/160489f9-4340-470e-ba27-d8e1d6183f58_isrForCrosstalkSources_855401_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/160489f9-4340-470e-ba27-d8e1d6183f58_isrForCrosstalkSources_855401_18.stderr parsl.dataflow.dflow DEBUG: Task 4706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4706 has memoization hash 5658383138b306ee739d08664a1d5a89 parsl.dataflow.memoization INFO: Task 4706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3c23f4cb4a96bc45bb8dcf6d2e19bb7 with result from task 575 parsl.dataflow.dflow INFO: Parsl task 4706 try 0 launched on executor multi with executor id 4707 parsl.dataflow.dflow INFO: Standard output for task 4706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/36735455-6f54-4a30-b95c-f0dcf7d12650_isrForCrosstalkSources_855444_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/36735455-6f54-4a30-b95c-f0dcf7d12650_isrForCrosstalkSources_855444_38.stderr parsl.dataflow.dflow DEBUG: Task 4707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4707 has memoization hash 5d38ea6953937860df991baa162b90f9 parsl.dataflow.memoization INFO: Task 4707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4707 try 0 launched on executor multi with executor id 4708 parsl.dataflow.dflow INFO: Standard output for task 4707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/99dcd15e-00ea-46b2-b55a-0f1154404228_isrForCrosstalkSources_855380_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/99dcd15e-00ea-46b2-b55a-0f1154404228_isrForCrosstalkSources_855380_27.stderr parsl.dataflow.dflow DEBUG: Task 4708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4708 has memoization hash fe7fc3789a73ad7941b5bff63b8ed8d6 parsl.dataflow.memoization INFO: Task 4708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4708 try 0 launched on executor multi with executor id 4709 parsl.dataflow.dflow INFO: Standard output for task 4708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/4b39c764-a09d-4d15-b392-858bc6668f2a_isrForCrosstalkSources_855448_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/4b39c764-a09d-4d15-b392-858bc6668f2a_isrForCrosstalkSources_855448_24.stderr parsl.dataflow.dflow DEBUG: Task 4709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4709 has memoization hash 24d10f82be4b4a65f65529cec60a05bc parsl.dataflow.memoization INFO: Task 4709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2758b3f1-fbf4-4e80-b1ce-e7fd30b59a37_isrForCrosstalkSources_855417_31.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2758b3f1-fbf4-4e80-b1ce-e7fd30b59a37_isrForCrosstalkSources_855417_31.stderr parsl.dataflow.dflow INFO: Parsl task 4709 try 0 launched on executor multi with executor id 4710 parsl.dataflow.dflow INFO: Standard output for task 4709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1e4bf6b-e35c-4c9a-9f8e-ed07582d12e6_isrForCrosstalkSources_855458_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1e4bf6b-e35c-4c9a-9f8e-ed07582d12e6_isrForCrosstalkSources_855458_61.stderr parsl.dataflow.dflow DEBUG: Task 4710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4710 has memoization hash 408c517bf3ad36efdae441b14f718685 parsl.dataflow.memoization INFO: Task 4710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4710 try 0 launched on executor multi with executor id 4711 parsl.dataflow.dflow INFO: Standard output for task 4710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5c5fab54-8b75-4a97-af9f-f38c0ddb2317_isrForCrosstalkSources_855418_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5c5fab54-8b75-4a97-af9f-f38c0ddb2317_isrForCrosstalkSources_855418_40.stderr parsl.dataflow.dflow DEBUG: Task 4711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4711 has memoization hash 0898bcd693bf6d665dd1626edabbfd7d parsl.dataflow.memoization INFO: Task 4711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4711 try 0 launched on executor multi with executor id 4712 parsl.dataflow.dflow INFO: Standard output for task 4711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/424d5b6f-d797-4de7-b7c7-a95899f2889e_isrForCrosstalkSources_855387_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/424d5b6f-d797-4de7-b7c7-a95899f2889e_isrForCrosstalkSources_855387_38.stderr parsl.dataflow.dflow DEBUG: Task 4712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4712 has memoization hash 05809c3af84b2a68caccea5a319bc63f parsl.dataflow.memoization INFO: Task 4712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4712 try 0 launched on executor multi with executor id 4713 parsl.dataflow.dflow INFO: Standard output for task 4712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/72c02f0c-19de-4b6e-9e5b-33d5064a9c73_isrForCrosstalkSources_855415_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/72c02f0c-19de-4b6e-9e5b-33d5064a9c73_isrForCrosstalkSources_855415_18.stderr parsl.dataflow.dflow DEBUG: Task 4713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4713 has memoization hash bd3f3e42cc1bfa3eda3b41ec1f937476 parsl.dataflow.memoization INFO: Task 4713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4713 try 0 launched on executor multi with executor id 4714 parsl.dataflow.dflow INFO: Standard output for task 4713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/2964a1dc-13ff-44d8-8716-28a9fcc8d138_isrForCrosstalkSources_855405_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/2964a1dc-13ff-44d8-8716-28a9fcc8d138_isrForCrosstalkSources_855405_30.stderr parsl.dataflow.dflow DEBUG: Task 4714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4714 has memoization hash c5d602de3b39b3d3afd8b146b7f821a6 parsl.dataflow.memoization INFO: Task 4714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4714 try 0 launched on executor multi with executor id 4715 parsl.dataflow.dflow INFO: Standard output for task 4714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/76f4bc9c-0fa9-456c-8cb7-363ffd29cc94_isrForCrosstalkSources_855451_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/76f4bc9c-0fa9-456c-8cb7-363ffd29cc94_isrForCrosstalkSources_855451_59.stderr parsl.dataflow.dflow DEBUG: Task 4715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4715 has memoization hash 52dc2c7e4e70507aaa4afa1beb31d24a parsl.dataflow.memoization INFO: Task 4715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4715 try 0 launched on executor multi with executor id 4716 parsl.dataflow.dflow INFO: Standard output for task 4715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cc600854-fb9a-41ea-b53b-fb6d96aa689c_isrForCrosstalkSources_855387_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cc600854-fb9a-41ea-b53b-fb6d96aa689c_isrForCrosstalkSources_855387_36.stderr parsl.dataflow.dflow DEBUG: Task 4716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4716 has memoization hash 0857047cbe66430d32ebf92ab4e80322 parsl.dataflow.memoization INFO: Task 4716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4716 try 0 launched on executor multi with executor id 4717 parsl.dataflow.dflow INFO: Standard output for task 4716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/94d6c420-ca6f-4157-a234-0ff9caf76d45_isrForCrosstalkSources_855414_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/94d6c420-ca6f-4157-a234-0ff9caf76d45_isrForCrosstalkSources_855414_35.stderr parsl.dataflow.dflow DEBUG: Task 4717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4717 has memoization hash efc2607beb0df2558976cab77351ed5e parsl.dataflow.memoization INFO: Task 4717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4717 try 0 launched on executor multi with executor id 4718 parsl.dataflow.dflow INFO: Standard output for task 4717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/061c5f0c-4b92-4eb8-81ec-073f530c0263_isrForCrosstalkSources_855454_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/061c5f0c-4b92-4eb8-81ec-073f530c0263_isrForCrosstalkSources_855454_17.stderr parsl.dataflow.dflow DEBUG: Task 4718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4718 has memoization hash 8589f9bc2e153d25f3c3f90defe64fea parsl.dataflow.memoization INFO: Task 4718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4718 try 0 launched on executor multi with executor id 4719 parsl.dataflow.dflow INFO: Standard output for task 4718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4cfb2e70-fdf5-469b-9a15-1a4cefda433b_isrForCrosstalkSources_855415_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4cfb2e70-fdf5-469b-9a15-1a4cefda433b_isrForCrosstalkSources_855415_15.stderr parsl.dataflow.dflow DEBUG: Task 4719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4719 has memoization hash 6749b33d49e166fdec8e87dd64b17c7e parsl.dataflow.memoization INFO: Task 4719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4719 try 0 launched on executor multi with executor id 4720 parsl.dataflow.dflow INFO: Standard output for task 4719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ffba4a8e-a2c3-4d99-a407-ea00caab0b1f_isrForCrosstalkSources_855416_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ffba4a8e-a2c3-4d99-a407-ea00caab0b1f_isrForCrosstalkSources_855416_26.stderr parsl.dataflow.dflow DEBUG: Task 4720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4720 has memoization hash 119ab10641d3affe705011285a660f2f parsl.dataflow.memoization INFO: Task 4720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4720 try 0 launched on executor multi with executor id 4721 parsl.dataflow.dflow INFO: Standard output for task 4720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0d4198be-f81e-4bfc-9157-073163a29853_isrForCrosstalkSources_855437_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0d4198be-f81e-4bfc-9157-073163a29853_isrForCrosstalkSources_855437_47.stderr parsl.dataflow.dflow DEBUG: Task 4721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4721 has memoization hash ced4423349061975548a5484a831fb83 parsl.dataflow.memoization INFO: Task 4721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4721 try 0 launched on executor multi with executor id 4722 parsl.dataflow.dflow INFO: Standard output for task 4721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe41ad-7979-4797-a922-0da316fb3b73_isrForCrosstalkSources_855423_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe41ad-7979-4797-a922-0da316fb3b73_isrForCrosstalkSources_855423_36.stderr parsl.dataflow.dflow DEBUG: Task 4722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4722 has memoization hash 4aa0f4f4389b08d2486fd38ee59ecaf0 parsl.dataflow.memoization INFO: Task 4722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4722 try 0 launched on executor multi with executor id 4723 parsl.dataflow.dflow INFO: Standard output for task 4722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8848a791-b2b5-489c-b0b8-76c0b62db33a_isrForCrosstalkSources_855419_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8848a791-b2b5-489c-b0b8-76c0b62db33a_isrForCrosstalkSources_855419_8.stderr parsl.dataflow.dflow DEBUG: Task 4723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4723 has memoization hash 909d3957f564c310d0833fd16d1b8f2f parsl.dataflow.memoization INFO: Task 4723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4723 try 0 launched on executor multi with executor id 4724 parsl.dataflow.dflow INFO: Standard output for task 4723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/153a7c8d-9dfb-4d39-8a7b-240db1ca0e73_isrForCrosstalkSources_855397_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/153a7c8d-9dfb-4d39-8a7b-240db1ca0e73_isrForCrosstalkSources_855397_13.stderr parsl.dataflow.dflow DEBUG: Task 4724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4724 has memoization hash 0519538a01375299f64b9a8e0591fd6b parsl.dataflow.memoization INFO: Task 4724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4724 try 0 launched on executor multi with executor id 4725 parsl.dataflow.dflow INFO: Standard output for task 4724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/66b01b7d-8924-45e6-9e4e-c5884a25c90a_isrForCrosstalkSources_855392_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/66b01b7d-8924-45e6-9e4e-c5884a25c90a_isrForCrosstalkSources_855392_59.stderr parsl.dataflow.dflow DEBUG: Task 4725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4725 has memoization hash c6b44739438ca7592027a4c9fab2dc4b parsl.dataflow.memoization INFO: Task 4725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4725 try 0 launched on executor multi with executor id 4726 parsl.dataflow.dflow INFO: Standard output for task 4725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7e4cd012-abff-414e-9fa9-d66907d7edc9_isrForCrosstalkSources_855449_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7e4cd012-abff-414e-9fa9-d66907d7edc9_isrForCrosstalkSources_855449_62.stderr parsl.dataflow.dflow DEBUG: Task 4726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4726 has memoization hash 513448cd69ec377ca720db42c1e78010 parsl.dataflow.memoization INFO: Task 4726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4726 try 0 launched on executor multi with executor id 4727 parsl.dataflow.dflow INFO: Standard output for task 4726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f0f73b3-c3a3-4219-9859-f4018dd2e78a_isrForCrosstalkSources_855387_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f0f73b3-c3a3-4219-9859-f4018dd2e78a_isrForCrosstalkSources_855387_17.stderr parsl.dataflow.dflow DEBUG: Task 4727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4727 has memoization hash 179e8b2e6bc99bab6b726560fead9366 parsl.dataflow.memoization INFO: Task 4727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4727 try 0 launched on executor multi with executor id 4728 parsl.dataflow.dflow INFO: Standard output for task 4727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/974c4d6a-06f5-4c8f-b592-4cf1c8a913ba_isrForCrosstalkSources_855381_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/974c4d6a-06f5-4c8f-b592-4cf1c8a913ba_isrForCrosstalkSources_855381_50.stderr parsl.dataflow.dflow DEBUG: Task 4728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4728 has memoization hash 5652b263882c947646fd1a2905349256 parsl.dataflow.memoization INFO: Task 4728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4728 try 0 launched on executor multi with executor id 4729 parsl.dataflow.dflow INFO: Standard output for task 4728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bbefc63f-bd27-46e4-865b-9015114ec10e_isrForCrosstalkSources_855450_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bbefc63f-bd27-46e4-865b-9015114ec10e_isrForCrosstalkSources_855450_58.stderr parsl.dataflow.dflow DEBUG: Task 4729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4729 has memoization hash 28cbba24a5085b68982b79bef5b14852 parsl.dataflow.memoization INFO: Task 4729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4729 try 0 launched on executor multi with executor id 4730 parsl.dataflow.dflow INFO: Standard output for task 4729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6175362e-402f-441f-a610-4c3ddecfd751_isrForCrosstalkSources_855424_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6175362e-402f-441f-a610-4c3ddecfd751_isrForCrosstalkSources_855424_10.stderr parsl.dataflow.dflow DEBUG: Task 4730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4730 has memoization hash e595a7259cb7ea33a9f4c23efd412887 parsl.dataflow.memoization INFO: Task 4730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4730 try 0 launched on executor multi with executor id 4731 parsl.dataflow.dflow INFO: Standard output for task 4730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7feff4b2-cd21-4385-9076-3c8e8b22dc30_isrForCrosstalkSources_855441_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7feff4b2-cd21-4385-9076-3c8e8b22dc30_isrForCrosstalkSources_855441_7.stderr 2024-03-09 14:51:55 proc_lsst.multi:170 [INFO] [multi] job_id 16964202 parsl.dataflow.dflow DEBUG: Task 4731 will be sent to executor multi proc_lsst.multi INFO: [multi] job_id 16964202 2024-03-09 14:51:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 121 parsl.dataflow.dflow DEBUG: Adding output dependencies proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-09 14:51:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow INFO: Task 4731 submitted for App isrForCrosstalkSources, not waiting on any dependency proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964202 parsl.dataflow.dflow DEBUG: Task 4731 set to pending state with AppFuture: proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964202 parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.executors.status_handling DEBUG: Launched block 176 on executor multi with job ID 16964202 parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.executors.status_handling INFO: Allocated block ID 177 parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-177 2024-03-09 14:51:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 parsl.dataflow.memoization DEBUG: Task 4731 has memoization hash e301c03348a621ad6988ecad42dcd2ec proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.memoization INFO: Task 4731 had no result in cache proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4731 try 0 launched on executor multi with executor id 4732 parsl.dataflow.dflow INFO: Standard output for task 4731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2ec98e4e-2a68-4d50-834f-7a2cf96a8dee_isrForCrosstalkSources_855453_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2ec98e4e-2a68-4d50-834f-7a2cf96a8dee_isrForCrosstalkSources_855453_35.stderr parsl.dataflow.dflow DEBUG: Task 4732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4732 has memoization hash 38f9abf2ba349480c4b320c9e6b816f9 parsl.dataflow.memoization INFO: Task 4732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4732 try 0 launched on executor multi with executor id 4733 parsl.dataflow.dflow INFO: Standard output for task 4732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/9b781c53-dc37-4758-b61d-4c27c00c3d93_isrForCrosstalkSources_855425_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/9b781c53-dc37-4758-b61d-4c27c00c3d93_isrForCrosstalkSources_855425_14.stderr parsl.dataflow.dflow DEBUG: Task 4733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4733 has memoization hash f6b89012b40dc5bbfd422e355fc99d15 parsl.dataflow.memoization INFO: Task 4733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4733 try 0 launched on executor multi with executor id 4734 parsl.dataflow.dflow INFO: Standard output for task 4733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/71281858-46c9-4f4a-a7ad-5b599c44fe0e_isrForCrosstalkSources_855449_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/71281858-46c9-4f4a-a7ad-5b599c44fe0e_isrForCrosstalkSources_855449_2.stderr parsl.dataflow.dflow DEBUG: Task 4734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4734 has memoization hash e3648c43c40e5b1457fa803a912cbf30 parsl.dataflow.memoization INFO: Task 4734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4734 try 0 launched on executor multi with executor id 4735 parsl.dataflow.dflow INFO: Standard output for task 4734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/98be16d3-675e-42c3-8dfa-42bbac476242_isrForCrosstalkSources_855423_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/98be16d3-675e-42c3-8dfa-42bbac476242_isrForCrosstalkSources_855423_19.stderr parsl.dataflow.dflow DEBUG: Task 4735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4735 has memoization hash d963d6dfd36e835fb30c5313a43f1a2e parsl.dataflow.memoization INFO: Task 4735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4735 try 0 launched on executor multi with executor id 4736 parsl.dataflow.dflow INFO: Standard output for task 4735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/24a12a1a-52ff-4537-a7d6-de91c4674ebf_isrForCrosstalkSources_855451_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/24a12a1a-52ff-4537-a7d6-de91c4674ebf_isrForCrosstalkSources_855451_5.stderr parsl.dataflow.dflow DEBUG: Task 4736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4736 has memoization hash 6f44f8def71617ae6af5264c6fb04735 parsl.dataflow.memoization INFO: Task 4736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4736 try 0 launched on executor multi with executor id 4737 parsl.dataflow.dflow INFO: Standard output for task 4736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/96ea71d6-fa6c-4c5c-9f4a-a02f77771581_isrForCrosstalkSources_855459_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/96ea71d6-fa6c-4c5c-9f4a-a02f77771581_isrForCrosstalkSources_855459_14.stderr parsl.dataflow.dflow DEBUG: Task 4737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4737 has memoization hash c23fa846e1db051242930ebf3c0af11a parsl.dataflow.memoization INFO: Task 4737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4737 try 0 launched on executor multi with executor id 4738 parsl.dataflow.dflow INFO: Standard output for task 4737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/cc35d258-e342-493a-aed0-50b178452e89_isrForCrosstalkSources_855426_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/cc35d258-e342-493a-aed0-50b178452e89_isrForCrosstalkSources_855426_4.stderr parsl.dataflow.dflow DEBUG: Task 4738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4738 has memoization hash cacc6f1781f75fd46b86e85ef94c759a parsl.dataflow.memoization INFO: Task 4738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4738 try 0 launched on executor multi with executor id 4739 parsl.dataflow.dflow INFO: Standard output for task 4738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6b3240dd-080e-45a5-b6ab-3af53c3e4b71_isrForCrosstalkSources_855397_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6b3240dd-080e-45a5-b6ab-3af53c3e4b71_isrForCrosstalkSources_855397_12.stderr parsl.dataflow.dflow DEBUG: Task 4739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4739 has memoization hash 2ca0403ee34655d8895339fd446eb9a9 parsl.dataflow.memoization INFO: Task 4739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f6c17d05d3da1742b7f9f78cce69a73 with result from task 595 parsl.dataflow.dflow INFO: Parsl task 4739 try 0 launched on executor multi with executor id 4740 parsl.dataflow.dflow INFO: Standard output for task 4739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/bb357eef-6291-4336-9cd3-c1a57be7ded7_isrForCrosstalkSources_855407_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/bb357eef-6291-4336-9cd3-c1a57be7ded7_isrForCrosstalkSources_855407_24.stderr parsl.dataflow.dflow DEBUG: Task 4740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4740 has memoization hash 65ccd3675103178f015c9817ac3a784a parsl.dataflow.memoization INFO: Task 4740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a239cd6f-148b-4436-bba7-afaaa3b4ece8_isrForCrosstalkSources_855398_14.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a239cd6f-148b-4436-bba7-afaaa3b4ece8_isrForCrosstalkSources_855398_14.stderr parsl.dataflow.dflow INFO: Parsl task 4740 try 0 launched on executor multi with executor id 4741 parsl.dataflow.dflow INFO: Standard output for task 4740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d28547bb-440c-4296-89ee-164a13a6e3d9_isrForCrosstalkSources_855385_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d28547bb-440c-4296-89ee-164a13a6e3d9_isrForCrosstalkSources_855385_51.stderr parsl.dataflow.dflow DEBUG: Task 4741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4741 has memoization hash 8a16a0cccd2336cb49b5a9a0a3b85b00 parsl.dataflow.memoization INFO: Task 4741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4741 try 0 launched on executor multi with executor id 4742 parsl.dataflow.dflow INFO: Standard output for task 4741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/49ad6525-c57f-46b2-a3a5-16dbe8c0c31a_isrForCrosstalkSources_855432_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/49ad6525-c57f-46b2-a3a5-16dbe8c0c31a_isrForCrosstalkSources_855432_52.stderr parsl.dataflow.dflow DEBUG: Task 4742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4742 has memoization hash aedf4e2e68bbf0c9517b49db4b4159b0 parsl.dataflow.memoization INFO: Task 4742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4742 try 0 launched on executor multi with executor id 4743 parsl.dataflow.dflow INFO: Standard output for task 4742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bbc282f9-9322-4753-9f90-548c70e13dfe_isrForCrosstalkSources_855424_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bbc282f9-9322-4753-9f90-548c70e13dfe_isrForCrosstalkSources_855424_45.stderr parsl.dataflow.dflow DEBUG: Task 4743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4743 has memoization hash ebd821ac029ec0c7c60c3c38ae22fe94 parsl.dataflow.memoization INFO: Task 4743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4743 try 0 launched on executor multi with executor id 4744 parsl.dataflow.dflow INFO: Standard output for task 4743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/eeb9b7b9-edc5-419f-8f93-dc8188d06b93_isrForCrosstalkSources_855448_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/eeb9b7b9-edc5-419f-8f93-dc8188d06b93_isrForCrosstalkSources_855448_62.stderr parsl.dataflow.dflow DEBUG: Task 4744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4744 has memoization hash cf889f6e6c241ce35e38213629d19bfb parsl.dataflow.memoization INFO: Task 4744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4744 try 0 launched on executor multi with executor id 4745 parsl.dataflow.dflow INFO: Standard output for task 4744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ae224175-2962-4759-8e52-201bd73c3dd6_isrForCrosstalkSources_855420_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ae224175-2962-4759-8e52-201bd73c3dd6_isrForCrosstalkSources_855420_25.stderr parsl.dataflow.dflow DEBUG: Task 4745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4745 has memoization hash f5ca98115cb55560165088b2722b33ad parsl.dataflow.memoization INFO: Task 4745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4745 try 0 launched on executor multi with executor id 4746 parsl.dataflow.dflow INFO: Standard output for task 4745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f86193d9-17b6-4af9-bb84-40ce67dba0c6_isrForCrosstalkSources_855425_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f86193d9-17b6-4af9-bb84-40ce67dba0c6_isrForCrosstalkSources_855425_38.stderr parsl.dataflow.dflow DEBUG: Task 4746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4746 has memoization hash 69910a116cc026d513deb96faf41d341 parsl.dataflow.memoization INFO: Task 4746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4746 try 0 launched on executor multi with executor id 4747 parsl.dataflow.dflow INFO: Standard output for task 4746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7b3b8cd8-fd4f-488d-bd6f-d426f756b448_isrForCrosstalkSources_855455_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7b3b8cd8-fd4f-488d-bd6f-d426f756b448_isrForCrosstalkSources_855455_46.stderr parsl.dataflow.dflow DEBUG: Task 4747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4747 has memoization hash 55a04075c9c8b01d06bda124f371ec22 parsl.dataflow.memoization INFO: Task 4747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4747 try 0 launched on executor multi with executor id 4748 parsl.dataflow.dflow INFO: Standard output for task 4747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/655c6307-501a-464c-b168-dee4bf2bc661_isrForCrosstalkSources_855417_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/655c6307-501a-464c-b168-dee4bf2bc661_isrForCrosstalkSources_855417_17.stderr parsl.dataflow.dflow DEBUG: Task 4748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4748 has memoization hash 5049b50e1542d5c93cb0a9a461cd2891 parsl.dataflow.memoization INFO: Task 4748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4748 try 0 launched on executor multi with executor id 4749 parsl.dataflow.dflow INFO: Standard output for task 4748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/284c6dcb-de7a-47f6-94c9-cb50dbb4392d_isrForCrosstalkSources_855429_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/284c6dcb-de7a-47f6-94c9-cb50dbb4392d_isrForCrosstalkSources_855429_53.stderr parsl.dataflow.dflow DEBUG: Task 4749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4749 has memoization hash b6714796f0a3a4214903f9169769e629 parsl.dataflow.memoization INFO: Task 4749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4749 try 0 launched on executor multi with executor id 4750 parsl.dataflow.dflow INFO: Standard output for task 4749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e1c0b867-453c-414f-862d-6bc7e09eee42_isrForCrosstalkSources_855382_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e1c0b867-453c-414f-862d-6bc7e09eee42_isrForCrosstalkSources_855382_58.stderr parsl.dataflow.dflow DEBUG: Task 4750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4750 has memoization hash 7cb604df777c37411411a96ab4a48778 parsl.dataflow.memoization INFO: Task 4750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4750 try 0 launched on executor multi with executor id 4751 parsl.dataflow.dflow INFO: Standard output for task 4750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/47ea28dd-9287-4e6e-8996-b493fd7e0580_isrForCrosstalkSources_855431_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/47ea28dd-9287-4e6e-8996-b493fd7e0580_isrForCrosstalkSources_855431_25.stderr parsl.dataflow.dflow DEBUG: Task 4751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4751 has memoization hash 1f2a5e97f6ced35db25b48a2b52045f1 parsl.dataflow.memoization INFO: Task 4751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4751 try 0 launched on executor multi with executor id 4752 parsl.dataflow.dflow INFO: Standard output for task 4751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/78fafebb-433b-47ef-9245-188cbf2a3976_isrForCrosstalkSources_855440_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/78fafebb-433b-47ef-9245-188cbf2a3976_isrForCrosstalkSources_855440_1.stderr parsl.dataflow.dflow DEBUG: Task 4752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4752 has memoization hash 5c700a1bb8ca2a86533667491652b772 parsl.dataflow.memoization INFO: Task 4752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58daf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4752 try 0 launched on executor multi with executor id 4753 parsl.dataflow.dflow INFO: Standard output for task 4752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/afedd47d-5de8-4ea4-95c1-f778caf483d6_isrForCrosstalkSources_855428_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/afedd47d-5de8-4ea4-95c1-f778caf483d6_isrForCrosstalkSources_855428_55.stderr parsl.dataflow.dflow DEBUG: Task 4753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4753 has memoization hash 9fe431d64d33e4b8dd740e23afe0b5f0 parsl.dataflow.memoization INFO: Task 4753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4753 try 0 launched on executor multi with executor id 4754 parsl.dataflow.dflow INFO: Standard output for task 4753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2216a23d-4240-4de7-9ac3-766f242e89e2_isrForCrosstalkSources_855404_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2216a23d-4240-4de7-9ac3-766f242e89e2_isrForCrosstalkSources_855404_61.stderr parsl.dataflow.dflow DEBUG: Task 4754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4754 has memoization hash a94265a49c39522d77c458ed68dc96b1 parsl.dataflow.memoization INFO: Task 4754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4754 try 0 launched on executor multi with executor id 4755 parsl.dataflow.dflow INFO: Standard output for task 4754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2f03f5ea-ef85-4e4e-8f1e-7a3405bf86c2_isrForCrosstalkSources_855414_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2f03f5ea-ef85-4e4e-8f1e-7a3405bf86c2_isrForCrosstalkSources_855414_29.stderr parsl.dataflow.dflow DEBUG: Task 4755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4755 has memoization hash 7d2f05f603a9842f97e5d1b35f39c178 parsl.dataflow.memoization INFO: Task 4755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4755 try 0 launched on executor multi with executor id 4756 parsl.dataflow.dflow INFO: Standard output for task 4755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f7cbe2fb-fc8b-4193-9207-26dd8416ec6b_isrForCrosstalkSources_855383_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f7cbe2fb-fc8b-4193-9207-26dd8416ec6b_isrForCrosstalkSources_855383_34.stderr parsl.dataflow.dflow DEBUG: Task 4756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4756 has memoization hash 19d9f80e2d9a16a0374a8a5763dfe8f6 parsl.dataflow.memoization INFO: Task 4756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4756 try 0 launched on executor multi with executor id 4757 parsl.dataflow.dflow INFO: Standard output for task 4756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/77ca3153-1aa2-4a7d-a9f8-b4dcc73f253f_isrForCrosstalkSources_855423_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/77ca3153-1aa2-4a7d-a9f8-b4dcc73f253f_isrForCrosstalkSources_855423_57.stderr parsl.dataflow.dflow DEBUG: Task 4757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4757 has memoization hash 16f916f4c8b7527307b919117812bcd9 parsl.dataflow.memoization INFO: Task 4757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4757 try 0 launched on executor multi with executor id 4758 parsl.dataflow.dflow INFO: Standard output for task 4757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/64f01096-c70b-44cc-8b90-36afd6ced2d2_isrForCrosstalkSources_855441_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/64f01096-c70b-44cc-8b90-36afd6ced2d2_isrForCrosstalkSources_855441_26.stderr parsl.dataflow.dflow DEBUG: Task 4758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4758 has memoization hash f017becdd21d5be72e6ebf0aca3d21d2 parsl.dataflow.memoization INFO: Task 4758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4758 try 0 launched on executor multi with executor id 4759 parsl.dataflow.dflow INFO: Standard output for task 4758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/31bbaf32-8e2e-4757-9ced-f89713e33e87_isrForCrosstalkSources_855394_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/31bbaf32-8e2e-4757-9ced-f89713e33e87_isrForCrosstalkSources_855394_10.stderr parsl.dataflow.dflow DEBUG: Task 4759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4759 has memoization hash 509780587eb74d1c07b88df572cba669 parsl.dataflow.memoization INFO: Task 4759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4759 try 0 launched on executor multi with executor id 4760 parsl.dataflow.dflow INFO: Standard output for task 4759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/21dae5d6-2ffc-4960-b0fe-f310dbaa2e4d_isrForCrosstalkSources_855407_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/21dae5d6-2ffc-4960-b0fe-f310dbaa2e4d_isrForCrosstalkSources_855407_48.stderr parsl.dataflow.dflow DEBUG: Task 4760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4760 has memoization hash d7fd97dd2c43fc2a694a10e6aeb8a9da parsl.dataflow.memoization INFO: Task 4760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4760 try 0 launched on executor multi with executor id 4761 parsl.dataflow.dflow INFO: Standard output for task 4760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e29968fd-342f-4607-aa7c-4eb5fe4e48e5_isrForCrosstalkSources_855403_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e29968fd-342f-4607-aa7c-4eb5fe4e48e5_isrForCrosstalkSources_855403_41.stderr parsl.dataflow.dflow DEBUG: Task 4761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4761 has memoization hash 237921cae171f846d16e2e2cbd5163c3 parsl.dataflow.memoization INFO: Task 4761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4761 try 0 launched on executor multi with executor id 4762 parsl.dataflow.dflow INFO: Standard output for task 4761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2614a38f-42fa-4f0b-926f-a36f8f9c80fd_isrForCrosstalkSources_855414_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2614a38f-42fa-4f0b-926f-a36f8f9c80fd_isrForCrosstalkSources_855414_19.stderr parsl.dataflow.dflow DEBUG: Task 4762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4762 has memoization hash c8aa9f3ca56d93ed6da29792cf986891 parsl.dataflow.memoization INFO: Task 4762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4762 try 0 launched on executor multi with executor id 4763 parsl.dataflow.dflow INFO: Standard output for task 4762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/14375599-9e48-4ef1-b8cc-58c7e4cb01d5_isrForCrosstalkSources_855437_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/14375599-9e48-4ef1-b8cc-58c7e4cb01d5_isrForCrosstalkSources_855437_25.stderr parsl.dataflow.dflow DEBUG: Task 4763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4763 has memoization hash 880456f5a5f8da328f88d923489de5b6 parsl.dataflow.memoization INFO: Task 4763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4763 try 0 launched on executor multi with executor id 4764 parsl.dataflow.dflow INFO: Standard output for task 4763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/190a900f-d531-41c7-9c42-8a8c95676b2a_isrForCrosstalkSources_855409_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/190a900f-d531-41c7-9c42-8a8c95676b2a_isrForCrosstalkSources_855409_57.stderr parsl.dataflow.dflow DEBUG: Task 4764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4764 has memoization hash ee337b398482d6e3f3a13e9c0faf724b parsl.dataflow.memoization INFO: Task 4764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58daa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:55 proc_lsst.multi:170 [INFO] [multi] job_id 16964203 proc_lsst.multi INFO: [multi] job_id 16964203 2024-03-09 14:51:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-09 14:51:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964203 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964203 parsl.executors.status_handling DEBUG: Launched block 177 on executor multi with job ID 16964203 parsl.executors.status_handling INFO: Allocated block ID 178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-178 2024-03-09 14:51:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-09 14:51:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4764 try 0 launched on executor multi with executor id 4765 parsl.dataflow.dflow INFO: Standard output for task 4764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/829c0872-38a7-472f-a9bb-14983b603fd7_isrForCrosstalkSources_855412_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/829c0872-38a7-472f-a9bb-14983b603fd7_isrForCrosstalkSources_855412_22.stderr parsl.dataflow.dflow DEBUG: Task 4765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4765 has memoization hash 21f75313ce1884eb12489aae5f45a58e parsl.dataflow.memoization INFO: Task 4765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58daac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4765 try 0 launched on executor multi with executor id 4766 parsl.dataflow.dflow INFO: Standard output for task 4765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d2a07bff-6cf7-4f86-8ee9-4dbc41e9fd11_isrForCrosstalkSources_855438_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d2a07bff-6cf7-4f86-8ee9-4dbc41e9fd11_isrForCrosstalkSources_855438_40.stderr parsl.dataflow.dflow DEBUG: Task 4766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4766 has memoization hash 086113f440b1229179421d78db0a3f2d parsl.dataflow.memoization INFO: Task 4766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58da5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4766 try 0 launched on executor multi with executor id 4767 parsl.dataflow.dflow INFO: Standard output for task 4766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/af74e4a4-452e-408e-be5c-e29a770bf4db_isrForCrosstalkSources_855416_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/af74e4a4-452e-408e-be5c-e29a770bf4db_isrForCrosstalkSources_855416_45.stderr parsl.dataflow.dflow DEBUG: Task 4767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4767 has memoization hash 39f5eabefbeef134be4803360d84ae2b parsl.dataflow.memoization INFO: Task 4767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58dab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4767 try 0 launched on executor multi with executor id 4768 parsl.dataflow.dflow INFO: Standard output for task 4767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e2ef4678-8fc9-4441-8244-460e41055738_isrForCrosstalkSources_855396_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e2ef4678-8fc9-4441-8244-460e41055738_isrForCrosstalkSources_855396_54.stderr parsl.dataflow.dflow DEBUG: Task 4768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4768 has memoization hash b8c9e397fdffe78747591937837d8eaa parsl.dataflow.memoization INFO: Task 4768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4768 try 0 launched on executor multi with executor id 4769 parsl.dataflow.dflow INFO: Standard output for task 4768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/337463f6-4feb-40c1-89f0-e48dad5ef36b_isrForCrosstalkSources_855453_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/337463f6-4feb-40c1-89f0-e48dad5ef36b_isrForCrosstalkSources_855453_31.stderr parsl.dataflow.dflow DEBUG: Task 4769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4769 has memoization hash 7fbef40c8985788c3b32b57830d13ed8 parsl.dataflow.memoization INFO: Task 4769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4769 try 0 launched on executor multi with executor id 4770 parsl.dataflow.dflow INFO: Standard output for task 4769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a515206-5286-4be0-8911-e471a79f9a75_isrForCrosstalkSources_855425_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a515206-5286-4be0-8911-e471a79f9a75_isrForCrosstalkSources_855425_6.stderr parsl.dataflow.dflow DEBUG: Task 4770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4770 has memoization hash 4f800fc662b48177271f5d7e2a470eba parsl.dataflow.memoization INFO: Task 4770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4770 try 0 launched on executor multi with executor id 4771 parsl.dataflow.dflow INFO: Standard output for task 4770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/d15c433e-86db-4d45-8b55-c274e1450f2b_isrForCrosstalkSources_855391_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/d15c433e-86db-4d45-8b55-c274e1450f2b_isrForCrosstalkSources_855391_53.stderr parsl.dataflow.dflow DEBUG: Task 4771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4771 has memoization hash 91b6846cd91764eb07d21d4abded05db parsl.dataflow.memoization INFO: Task 4771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4771 try 0 launched on executor multi with executor id 4772 parsl.dataflow.dflow INFO: Standard output for task 4771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/49c4f3a5-0e13-4773-b61a-6fb8f35cbdde_isrForCrosstalkSources_855446_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/49c4f3a5-0e13-4773-b61a-6fb8f35cbdde_isrForCrosstalkSources_855446_62.stderr parsl.dataflow.dflow DEBUG: Task 4772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4772 has memoization hash 8068983328a65df6415785f15aa4e0a5 parsl.dataflow.memoization INFO: Task 4772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4772 try 0 launched on executor multi with executor id 4773 parsl.dataflow.dflow INFO: Standard output for task 4772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/0ed030bb-90df-4ca2-9ca4-b9d3abe1147f_isrForCrosstalkSources_855398_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/0ed030bb-90df-4ca2-9ca4-b9d3abe1147f_isrForCrosstalkSources_855398_35.stderr parsl.dataflow.dflow DEBUG: Task 4773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4773 has memoization hash 533dda52002e33bc2ec9cd148a411bcd parsl.dataflow.memoization INFO: Task 4773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4773 try 0 launched on executor multi with executor id 4774 parsl.dataflow.dflow INFO: Standard output for task 4773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/39aebcfd-8410-4a65-8a80-789c2d54a661_isrForCrosstalkSources_855409_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/39aebcfd-8410-4a65-8a80-789c2d54a661_isrForCrosstalkSources_855409_29.stderr parsl.dataflow.dflow DEBUG: Task 4774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4774 has memoization hash 594ecc981aecf1cb4799c0de10e41077 parsl.dataflow.memoization INFO: Task 4774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4774 try 0 launched on executor multi with executor id 4775 parsl.dataflow.dflow INFO: Standard output for task 4774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3ffe4e7e-8e93-4281-b77b-13104e7d6586_isrForCrosstalkSources_855426_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3ffe4e7e-8e93-4281-b77b-13104e7d6586_isrForCrosstalkSources_855426_37.stderr parsl.dataflow.dflow DEBUG: Task 4775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4775 has memoization hash b4788c282838384e9f49480c360e04a6 parsl.dataflow.memoization INFO: Task 4775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4775 try 0 launched on executor multi with executor id 4776 parsl.dataflow.dflow INFO: Standard output for task 4775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/1356a133-82b3-461a-9130-f408f60ac8cf_isrForCrosstalkSources_855416_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/1356a133-82b3-461a-9130-f408f60ac8cf_isrForCrosstalkSources_855416_52.stderr parsl.dataflow.dflow DEBUG: Task 4776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4776 has memoization hash db711e9f7e027f538c2b771c01add49f parsl.dataflow.memoization INFO: Task 4776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4776 try 0 launched on executor multi with executor id 4777 parsl.dataflow.dflow INFO: Standard output for task 4776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/eb0c52a0-7724-4bc1-b44f-ac1559c51b69_isrForCrosstalkSources_855426_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/eb0c52a0-7724-4bc1-b44f-ac1559c51b69_isrForCrosstalkSources_855426_19.stderr parsl.dataflow.dflow DEBUG: Task 4777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4777 has memoization hash 4e0aee951340e2c9cdda9ec9585440bf parsl.dataflow.memoization INFO: Task 4777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4777 try 0 launched on executor multi with executor id 4778 parsl.dataflow.dflow INFO: Standard output for task 4777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bf47f90c-68d8-4d9f-b7ae-e1cb250e0c8b_isrForCrosstalkSources_855436_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bf47f90c-68d8-4d9f-b7ae-e1cb250e0c8b_isrForCrosstalkSources_855436_47.stderr parsl.dataflow.dflow DEBUG: Task 4778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4778 has memoization hash 5f1b430abc879973b106aa92f8b97460 parsl.dataflow.memoization INFO: Task 4778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4778 try 0 launched on executor multi with executor id 4779 parsl.dataflow.dflow INFO: Standard output for task 4778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e351af5e-9fde-465c-8134-0d855436c69f_isrForCrosstalkSources_855444_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e351af5e-9fde-465c-8134-0d855436c69f_isrForCrosstalkSources_855444_60.stderr parsl.dataflow.dflow DEBUG: Task 4779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4779 has memoization hash 23d75ba69a1ba38ab439064bdd18290f parsl.dataflow.memoization INFO: Task 4779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4779 try 0 launched on executor multi with executor id 4780 parsl.dataflow.dflow INFO: Standard output for task 4779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cddf2464-9061-4127-8335-dec59197f070_isrForCrosstalkSources_855446_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cddf2464-9061-4127-8335-dec59197f070_isrForCrosstalkSources_855446_60.stderr parsl.dataflow.dflow DEBUG: Task 4780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4780 has memoization hash b3be9829bfd54a6110a37149bb979778 parsl.dataflow.memoization INFO: Task 4780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4780 try 0 launched on executor multi with executor id 4781 parsl.dataflow.dflow INFO: Standard output for task 4780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9bdb76d8-abad-49fa-9ef0-ef3164f7a9ea_isrForCrosstalkSources_855401_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9bdb76d8-abad-49fa-9ef0-ef3164f7a9ea_isrForCrosstalkSources_855401_10.stderr parsl.dataflow.dflow DEBUG: Task 4781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4781 has memoization hash d2f08a4ab4c711e40c867ada88dc6f55 parsl.dataflow.memoization INFO: Task 4781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4781 try 0 launched on executor multi with executor id 4782 2024-03-09 14:51:55 proc_lsst.multi:170 [INFO] [multi] job_id 16964204 parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Standard output for task 4781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06b807fe-d709-4da4-bf0d-335d1f2e8cc1_isrForCrosstalkSources_855454_48.stdout proc_lsst.multi INFO: [multi] job_id 16964204 2024-03-09 14:51:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-09 14:51:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964204 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964204 parsl.dataflow.memoization DEBUG: Storing app cache entry 00e85844c5cf35a547868294f08d7d8d with result from task 654 parsl.executors.status_handling DEBUG: Launched block 178 on executor multi with job ID 16964204 parsl.executors.status_handling INFO: Allocated block ID 179 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Standard error for task 4781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06b807fe-d709-4da4-bf0d-335d1f2e8cc1_isrForCrosstalkSources_855454_48.stderr parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-179 2024-03-09 14:51:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow DEBUG: Task 4782 will be sent to executor multi proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/b54448d4-a009-4a1d-ab67-45ca4a123f42_isrForCrosstalkSources_855443_26.stdout parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/b54448d4-a009-4a1d-ab67-45ca4a123f42_isrForCrosstalkSources_855443_26.stderr parsl.dataflow.dflow INFO: Task 4782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e77648d7efc196c1768f2e520147e56 with result from task 577 parsl.dataflow.dflow DEBUG: Task 4782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4782 has memoization hash 17b3fbd4df7d99e7715effbbe4ddcdc8 parsl.dataflow.memoization INFO: Task 4782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b51f22e4-cb69-4e7c-bc7c-0ee032951a35_isrForCrosstalkSources_855423_9.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b51f22e4-cb69-4e7c-bc7c-0ee032951a35_isrForCrosstalkSources_855423_9.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4782 try 0 launched on executor multi with executor id 4783 parsl.dataflow.dflow INFO: Standard output for task 4782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/660cca55-23e4-4498-91d0-983b941e550b_isrForCrosstalkSources_855446_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/660cca55-23e4-4498-91d0-983b941e550b_isrForCrosstalkSources_855446_28.stderr parsl.dataflow.dflow DEBUG: Task 4783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4783 has memoization hash 95968c009789943876fe8c9e1df356cf parsl.dataflow.memoization INFO: Task 4783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58daca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4783 try 0 launched on executor multi with executor id 4784 parsl.dataflow.dflow INFO: Standard output for task 4783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/18b5910c-f001-4f5c-9740-eff1ce85ff04_isrForCrosstalkSources_855410_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/18b5910c-f001-4f5c-9740-eff1ce85ff04_isrForCrosstalkSources_855410_20.stderr parsl.dataflow.dflow DEBUG: Task 4784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4784 has memoization hash 0453f75f0cc2fd416801858cd1c9173e parsl.dataflow.memoization INFO: Task 4784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58db9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4784 try 0 launched on executor multi with executor id 4785 parsl.dataflow.dflow INFO: Standard output for task 4784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/bc527e5f-79be-44af-9eea-a487d36bd447_isrForCrosstalkSources_855429_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/bc527e5f-79be-44af-9eea-a487d36bd447_isrForCrosstalkSources_855429_1.stderr parsl.dataflow.dflow DEBUG: Task 4785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4785 has memoization hash 08e7cde3bda05d1ed8cd58057d840811 parsl.dataflow.memoization INFO: Task 4785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4785 try 0 launched on executor multi with executor id 4786 parsl.dataflow.dflow INFO: Standard output for task 4785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7b05a3d4-2fcf-4203-ac27-b8a0153769b8_isrForCrosstalkSources_855443_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7b05a3d4-2fcf-4203-ac27-b8a0153769b8_isrForCrosstalkSources_855443_47.stderr parsl.dataflow.dflow DEBUG: Task 4786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4786 has memoization hash 0b07457dee8279eaa237aea1c6b4144b parsl.dataflow.memoization INFO: Task 4786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4786 try 0 launched on executor multi with executor id 4787 parsl.dataflow.dflow INFO: Standard output for task 4786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bad94c10-60d9-43a1-a51b-839155a8dcd0_isrForCrosstalkSources_855397_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bad94c10-60d9-43a1-a51b-839155a8dcd0_isrForCrosstalkSources_855397_55.stderr parsl.dataflow.dflow DEBUG: Task 4787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4787 has memoization hash ab28fc9e691678b14c015adfd01a7618 parsl.dataflow.memoization INFO: Task 4787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4787 try 0 launched on executor multi with executor id 4788 parsl.dataflow.dflow INFO: Standard output for task 4787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/02e1f6b7-3a17-4ea1-86fe-bd7eebae0504_isrForCrosstalkSources_855420_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/02e1f6b7-3a17-4ea1-86fe-bd7eebae0504_isrForCrosstalkSources_855420_62.stderr parsl.dataflow.dflow DEBUG: Task 4788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4788 has memoization hash 57ae48299d8cb85501ad01a3ac67aa40 parsl.dataflow.memoization INFO: Task 4788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4788 try 0 launched on executor multi with executor id 4789 parsl.dataflow.dflow INFO: Standard output for task 4788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87a19571-fbfe-4e0f-ad6a-6196b898897c_isrForCrosstalkSources_855391_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87a19571-fbfe-4e0f-ad6a-6196b898897c_isrForCrosstalkSources_855391_18.stderr parsl.dataflow.dflow DEBUG: Task 4789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4789 has memoization hash b662ba134eb836b0493f9ec13a5792d2 parsl.dataflow.memoization INFO: Task 4789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4789 try 0 launched on executor multi with executor id 4790 parsl.dataflow.dflow INFO: Standard output for task 4789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/33a3a14c-ba4e-414a-90bb-e8b410ddefbe_isrForCrosstalkSources_855392_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/33a3a14c-ba4e-414a-90bb-e8b410ddefbe_isrForCrosstalkSources_855392_48.stderr parsl.dataflow.dflow DEBUG: Task 4790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4790 has memoization hash 57963a6cb5f7f0ad789b08cb3c34e09a parsl.dataflow.memoization INFO: Task 4790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4790 try 0 launched on executor multi with executor id 4791 parsl.dataflow.dflow INFO: Standard output for task 4790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a920e287-17be-448e-b52d-617bf6e7d586_isrForCrosstalkSources_855415_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a920e287-17be-448e-b52d-617bf6e7d586_isrForCrosstalkSources_855415_49.stderr parsl.dataflow.dflow DEBUG: Task 4791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4791 has memoization hash 5c9ca24eb0669998e637c6bb25a47b37 parsl.dataflow.memoization INFO: Task 4791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4791 try 0 launched on executor multi with executor id 4792 parsl.dataflow.dflow INFO: Standard output for task 4791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9163333f-4a2e-486f-ab49-7ef3c150368e_isrForCrosstalkSources_855432_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9163333f-4a2e-486f-ab49-7ef3c150368e_isrForCrosstalkSources_855432_44.stderr parsl.dataflow.dflow DEBUG: Task 4792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4792 has memoization hash 1aab04ff3ab1bff0677969fe12b39031 parsl.dataflow.memoization INFO: Task 4792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4792 try 0 launched on executor multi with executor id 4793 parsl.dataflow.dflow INFO: Standard output for task 4792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/56e3ed84-ac76-407f-a777-b1dd0b6ff621_isrForCrosstalkSources_855385_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/56e3ed84-ac76-407f-a777-b1dd0b6ff621_isrForCrosstalkSources_855385_9.stderr parsl.dataflow.dflow DEBUG: Task 4793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4793 has memoization hash db34a857d5531ceff790df0f7d656d5b parsl.dataflow.memoization INFO: Task 4793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Parsl task 4793 try 0 launched on executor multi with executor id 4794 parsl.dataflow.memoization DEBUG: Storing app cache entry 498ae510255bcca785a9e75b96ea4027 with result from task 593 parsl.dataflow.dflow INFO: Standard output for task 4793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3fe92d09-98e7-4039-b317-65458893f857_isrForCrosstalkSources_855452_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3fe92d09-98e7-4039-b317-65458893f857_isrForCrosstalkSources_855452_56.stderr parsl.dataflow.dflow DEBUG: Task 4794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow DEBUG: Task 4794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/40254421-1fda-4b34-bc87-59b6fe7634fc_isrForCrosstalkSources_855449_20.stdout parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/40254421-1fda-4b34-bc87-59b6fe7634fc_isrForCrosstalkSources_855449_20.stderr parsl.dataflow.memoization DEBUG: Task 4794 has memoization hash 66f8bd1a2a03e1bf11d3760b63a8a21b parsl.dataflow.memoization INFO: Task 4794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4794 try 0 launched on executor multi with executor id 4795 parsl.dataflow.dflow INFO: Standard output for task 4794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1f589b30-fe65-40c2-a14d-ee947edaf419_isrForCrosstalkSources_855437_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1f589b30-fe65-40c2-a14d-ee947edaf419_isrForCrosstalkSources_855437_38.stderr parsl.dataflow.dflow DEBUG: Task 4795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4795 has memoization hash a93487ee39940123ebfbd54f3a030372 parsl.dataflow.memoization INFO: Task 4795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4795 try 0 launched on executor multi with executor id 4796 parsl.dataflow.dflow INFO: Standard output for task 4795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/34c2406b-43ac-4d48-b26c-3fa0c42c8ff1_isrForCrosstalkSources_855420_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/34c2406b-43ac-4d48-b26c-3fa0c42c8ff1_isrForCrosstalkSources_855420_4.stderr parsl.dataflow.dflow DEBUG: Task 4796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4796 has memoization hash 0b2aa7acd2dbd2b030b506f506433b33 parsl.dataflow.memoization INFO: Task 4796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4796 try 0 launched on executor multi with executor id 4797 parsl.dataflow.dflow INFO: Standard output for task 4796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9bf9aeb3-a653-485b-85fd-640220de3745_isrForCrosstalkSources_855409_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9bf9aeb3-a653-485b-85fd-640220de3745_isrForCrosstalkSources_855409_7.stderr parsl.dataflow.dflow DEBUG: Task 4797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4797 has memoization hash eb9bdf3f429e4eb6ead5cda636bb1130 parsl.dataflow.memoization INFO: Task 4797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4797 try 0 launched on executor multi with executor id 4798 parsl.dataflow.dflow INFO: Standard output for task 4797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a2578705-575d-462c-9156-06b6682148c1_isrForCrosstalkSources_855411_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a2578705-575d-462c-9156-06b6682148c1_isrForCrosstalkSources_855411_29.stderr parsl.dataflow.dflow DEBUG: Task 4798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4798 has memoization hash ebf6dbf694abb5dfff69134a3dd64726 parsl.dataflow.memoization INFO: Task 4798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4798 try 0 launched on executor multi with executor id 4799 parsl.dataflow.dflow INFO: Standard output for task 4798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/be6a3ad9-fa38-4a51-9d50-3f172a7e2798_isrForCrosstalkSources_855382_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/be6a3ad9-fa38-4a51-9d50-3f172a7e2798_isrForCrosstalkSources_855382_11.stderr parsl.dataflow.dflow DEBUG: Task 4799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4799 has memoization hash ff38acfd1f56a18d2ea6dde2fe2b37c5 parsl.dataflow.memoization INFO: Task 4799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4799 try 0 launched on executor multi with executor id 4800 parsl.dataflow.dflow INFO: Standard output for task 4799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/268ebce8-6fd0-4ddc-9620-d90ef3550b6a_isrForCrosstalkSources_855379_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/268ebce8-6fd0-4ddc-9620-d90ef3550b6a_isrForCrosstalkSources_855379_61.stderr parsl.dataflow.dflow DEBUG: Task 4800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4800 has memoization hash 1cec6506f101e5f8b29f492272deb269 parsl.dataflow.memoization INFO: Task 4800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4800 try 0 launched on executor multi with executor id 4801 parsl.dataflow.dflow INFO: Standard output for task 4800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/36363978-ae72-4c81-9799-9567cb9e964b_isrForCrosstalkSources_855404_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/36363978-ae72-4c81-9799-9567cb9e964b_isrForCrosstalkSources_855404_11.stderr parsl.dataflow.dflow DEBUG: Task 4801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4801 has memoization hash 45fd60354a6b3cc070445793cf477b75 parsl.dataflow.memoization INFO: Task 4801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4801 try 0 launched on executor multi with executor id 4802 parsl.dataflow.dflow INFO: Standard output for task 4801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/928a2314-0805-4b14-ae43-21cdbcd6a19f_isrForCrosstalkSources_855385_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/928a2314-0805-4b14-ae43-21cdbcd6a19f_isrForCrosstalkSources_855385_48.stderr parsl.dataflow.dflow DEBUG: Task 4802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4802 has memoization hash 5b87e87bf9c8892cd53ff27daf398483 parsl.dataflow.memoization INFO: Task 4802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4802 try 0 launched on executor multi with executor id 4803 parsl.dataflow.dflow INFO: Standard output for task 4802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e84c2e37-784e-492b-ac6a-8b6e55c030d8_isrForCrosstalkSources_855413_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e84c2e37-784e-492b-ac6a-8b6e55c030d8_isrForCrosstalkSources_855413_38.stderr parsl.dataflow.dflow DEBUG: Task 4803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4803 has memoization hash 71c3b3629b269d7c6ca85f8d680a4913 parsl.dataflow.memoization INFO: Task 4803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4803 try 0 launched on executor multi with executor id 4804 parsl.dataflow.dflow INFO: Standard output for task 4803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/14df86af-5a16-4424-a589-05448afa5946_isrForCrosstalkSources_855432_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/14df86af-5a16-4424-a589-05448afa5946_isrForCrosstalkSources_855432_5.stderr parsl.dataflow.dflow DEBUG: Task 4804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4804 has memoization hash 54f5b4c96e629d7579a1d4abd0b6ac53 parsl.dataflow.memoization INFO: Task 4804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4804 try 0 launched on executor multi with executor id 4805 parsl.dataflow.dflow INFO: Standard output for task 4804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/904dc1d1-2aaa-47ab-8482-d78fafe7bcf6_isrForCrosstalkSources_855424_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/904dc1d1-2aaa-47ab-8482-d78fafe7bcf6_isrForCrosstalkSources_855424_28.stderr parsl.dataflow.dflow DEBUG: Task 4805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4805 has memoization hash 3aed7fb909af191b17606e05dd78803a parsl.dataflow.memoization INFO: Task 4805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4805 try 0 launched on executor multi with executor id 4806 parsl.dataflow.dflow INFO: Standard output for task 4805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b97a20-c821-4512-8428-15f4a60c51cf_isrForCrosstalkSources_855416_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b97a20-c821-4512-8428-15f4a60c51cf_isrForCrosstalkSources_855416_36.stderr parsl.dataflow.dflow DEBUG: Task 4806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4806 has memoization hash d22a48a021f8aee082b35d28f7ab612d parsl.dataflow.memoization INFO: Task 4806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4806 try 0 launched on executor multi with executor id 4807 parsl.dataflow.dflow INFO: Standard output for task 4806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8a77af0a-d7a2-42e7-9296-90f0ac83e32b_isrForCrosstalkSources_855402_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8a77af0a-d7a2-42e7-9296-90f0ac83e32b_isrForCrosstalkSources_855402_43.stderr parsl.dataflow.dflow DEBUG: Task 4807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4807 has memoization hash a3e7c0be2452d220a169a6639e7f5607 parsl.dataflow.memoization INFO: Task 4807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4807 try 0 launched on executor multi with executor id 4808 parsl.dataflow.dflow INFO: Standard output for task 4807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8039ae8c-8188-48fa-8a44-82503d533298_isrForCrosstalkSources_855397_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8039ae8c-8188-48fa-8a44-82503d533298_isrForCrosstalkSources_855397_23.stderr parsl.dataflow.dflow DEBUG: Task 4808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4808 has memoization hash 2ae20abf0768e6d64cbeb8c9d5127b83 parsl.dataflow.memoization INFO: Task 4808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4808 try 0 launched on executor multi with executor id 4809 parsl.dataflow.dflow INFO: Standard output for task 4808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7057e2b2-f1bf-4b38-886f-536a148da0c1_isrForCrosstalkSources_855443_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7057e2b2-f1bf-4b38-886f-536a148da0c1_isrForCrosstalkSources_855443_54.stderr parsl.dataflow.dflow DEBUG: Task 4809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4809 has memoization hash 7ec84ef5ae3ca5673d2ef7fb8e9460ae parsl.dataflow.memoization INFO: Task 4809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4809 try 0 launched on executor multi with executor id 4810 parsl.dataflow.dflow INFO: Standard output for task 4809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f240ea4d-5bc7-4265-9c92-2b2a5916e739_isrForCrosstalkSources_855429_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f240ea4d-5bc7-4265-9c92-2b2a5916e739_isrForCrosstalkSources_855429_36.stderr parsl.dataflow.dflow DEBUG: Task 4810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4810 has memoization hash da2c67c15507e96d115908fbe3b10b9e parsl.dataflow.memoization INFO: Task 4810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4810 try 0 launched on executor multi with executor id 4811 parsl.dataflow.dflow INFO: Standard output for task 4810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2d5a2dd8-c4a3-4555-8c17-ac8ccd474ff6_isrForCrosstalkSources_855434_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2d5a2dd8-c4a3-4555-8c17-ac8ccd474ff6_isrForCrosstalkSources_855434_33.stderr parsl.dataflow.dflow DEBUG: Task 4811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4811 has memoization hash d999aacb55c85f2fbcbee516c2dabbb1 parsl.dataflow.memoization INFO: Task 4811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4811 try 0 launched on executor multi with executor id 4812 parsl.dataflow.dflow INFO: Standard output for task 4811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2b0472c0-6964-4def-b744-dfce1b7d8b2a_isrForCrosstalkSources_855411_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2b0472c0-6964-4def-b744-dfce1b7d8b2a_isrForCrosstalkSources_855411_46.stderr parsl.dataflow.dflow DEBUG: Task 4812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4812 has memoization hash fcd0102e7afe1db46e8b74456bab2b71 parsl.dataflow.memoization INFO: Task 4812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4812 try 0 launched on executor multi with executor id 4813 parsl.dataflow.dflow INFO: Standard output for task 4812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9eb8efc5-954d-4bbb-a3d9-6b76be3c9caf_isrForCrosstalkSources_855411_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9eb8efc5-954d-4bbb-a3d9-6b76be3c9caf_isrForCrosstalkSources_855411_36.stderr parsl.dataflow.dflow DEBUG: Task 4813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4813 has memoization hash e11532200b6ee04cc5dbc1516aef5d32 parsl.dataflow.memoization INFO: Task 4813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4813 try 0 launched on executor multi with executor id 4814 parsl.dataflow.dflow INFO: Standard output for task 4813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/b1dfd0a1-d28d-4283-8970-50b9c4b1e66a_isrForCrosstalkSources_855405_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/b1dfd0a1-d28d-4283-8970-50b9c4b1e66a_isrForCrosstalkSources_855405_49.stderr parsl.dataflow.dflow DEBUG: Task 4814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4814 has memoization hash b91e2db83e4324378f8d9722464a5965 parsl.dataflow.memoization INFO: Task 4814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964205 proc_lsst.multi INFO: [multi] job_id 16964205 2024-03-09 14:51:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-09 14:51:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964205 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964205 parsl.executors.status_handling DEBUG: Launched block 179 on executor multi with job ID 16964205 parsl.executors.status_handling INFO: Allocated block ID 180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-180 2024-03-09 14:51:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4814 try 0 launched on executor multi with executor id 4815 parsl.dataflow.dflow INFO: Standard output for task 4814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/fa33bbda-8026-4520-8f3d-8d536b745175_isrForCrosstalkSources_855389_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/fa33bbda-8026-4520-8f3d-8d536b745175_isrForCrosstalkSources_855389_45.stderr parsl.dataflow.dflow DEBUG: Task 4815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4815 has memoization hash a17958819d9bdb6581201030e4709d29 parsl.dataflow.memoization INFO: Task 4815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4815 try 0 launched on executor multi with executor id 4816 parsl.dataflow.dflow INFO: Standard output for task 4815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5d0a8f94-d95f-408b-957e-6912c5fbf651_isrForCrosstalkSources_855435_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5d0a8f94-d95f-408b-957e-6912c5fbf651_isrForCrosstalkSources_855435_39.stderr parsl.dataflow.dflow DEBUG: Task 4816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4816 has memoization hash ca172671459280deedadcbaa795144ad parsl.dataflow.memoization INFO: Task 4816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a583e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4816 try 0 launched on executor multi with executor id 4817 parsl.dataflow.dflow INFO: Standard output for task 4816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e1d8431d-b5eb-424a-81c9-ac9f0cfb6839_isrForCrosstalkSources_855418_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e1d8431d-b5eb-424a-81c9-ac9f0cfb6839_isrForCrosstalkSources_855418_56.stderr parsl.dataflow.dflow DEBUG: Task 4817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4817 has memoization hash 7fd21c7cfd7343ff346ac3a35ed96aec parsl.dataflow.memoization INFO: Task 4817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4817 try 0 launched on executor multi with executor id 4818 parsl.dataflow.dflow INFO: Standard output for task 4817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/26b4dc17-608f-41c2-8f7e-efa103706caa_isrForCrosstalkSources_855446_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/26b4dc17-608f-41c2-8f7e-efa103706caa_isrForCrosstalkSources_855446_36.stderr parsl.dataflow.dflow DEBUG: Task 4818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4818 has memoization hash 20d33a325893a01914a4f59cf1cf9f34 parsl.dataflow.memoization INFO: Task 4818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4818 try 0 launched on executor multi with executor id 4819 parsl.dataflow.dflow INFO: Standard output for task 4818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b37a09ef-adb9-4dfe-9720-3c1e15440aba_isrForCrosstalkSources_855425_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b37a09ef-adb9-4dfe-9720-3c1e15440aba_isrForCrosstalkSources_855425_28.stderr parsl.dataflow.dflow DEBUG: Task 4819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4819 has memoization hash 3117ce37b022bad5476e7f937647df78 parsl.dataflow.memoization INFO: Task 4819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e721485c1bb9d42e2ea10a7b54190eeb with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3bb88089-4b46-438a-8ddf-9c2d64207218_isrForCrosstalkSources_855420_44.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3bb88089-4b46-438a-8ddf-9c2d64207218_isrForCrosstalkSources_855420_44.stderr parsl.dataflow.dflow INFO: Parsl task 4819 try 0 launched on executor multi with executor id 4820 parsl.dataflow.dflow INFO: Standard output for task 4819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/147226d3-087e-42fb-9435-96360186e2fe_isrForCrosstalkSources_855395_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/147226d3-087e-42fb-9435-96360186e2fe_isrForCrosstalkSources_855395_10.stderr parsl.dataflow.dflow DEBUG: Task 4820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4820 has memoization hash 646fbd1cb24371a6cc315b602b1c1c02 parsl.dataflow.memoization INFO: Task 4820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4820 try 0 launched on executor multi with executor id 4821 parsl.dataflow.dflow INFO: Standard output for task 4820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/bb6b3c68-78e9-4060-b099-c671f27b77af_isrForCrosstalkSources_855413_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/bb6b3c68-78e9-4060-b099-c671f27b77af_isrForCrosstalkSources_855413_41.stderr parsl.dataflow.dflow DEBUG: Task 4821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4821 has memoization hash 073a145876e944b413aaec4a57ecd9e9 parsl.dataflow.memoization INFO: Task 4821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4821 try 0 launched on executor multi with executor id 4822 parsl.dataflow.dflow INFO: Standard output for task 4821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a1ba808c-4de6-4676-a279-b49deebe4717_isrForCrosstalkSources_855455_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a1ba808c-4de6-4676-a279-b49deebe4717_isrForCrosstalkSources_855455_59.stderr parsl.dataflow.dflow DEBUG: Task 4822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4822 has memoization hash dcf4b88593c4a9a0d54423d3cfa6343c parsl.dataflow.memoization INFO: Task 4822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4822 try 0 launched on executor multi with executor id 4823 parsl.dataflow.dflow INFO: Standard output for task 4822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/34e76d03-47e6-400f-bc34-b7123bd07a6a_isrForCrosstalkSources_855438_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/34e76d03-47e6-400f-bc34-b7123bd07a6a_isrForCrosstalkSources_855438_48.stderr parsl.dataflow.dflow DEBUG: Task 4823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4823 has memoization hash 83ce9b82fb9be83284b77d2acfa22c33 parsl.dataflow.memoization INFO: Task 4823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4823 try 0 launched on executor multi with executor id 4824 parsl.dataflow.dflow INFO: Standard output for task 4823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5726f0f1-13c1-431e-884b-824331e161f5_isrForCrosstalkSources_855405_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5726f0f1-13c1-431e-884b-824331e161f5_isrForCrosstalkSources_855405_51.stderr parsl.dataflow.dflow DEBUG: Task 4824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4824 has memoization hash de208e6739d76d32be06e0f22d87760b parsl.dataflow.memoization INFO: Task 4824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4824 try 0 launched on executor multi with executor id 4825 parsl.dataflow.dflow INFO: Standard output for task 4824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/43875a40-dbe6-4d86-9627-3ad999ec8ebf_isrForCrosstalkSources_855399_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/43875a40-dbe6-4d86-9627-3ad999ec8ebf_isrForCrosstalkSources_855399_3.stderr parsl.dataflow.dflow DEBUG: Task 4825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4825 has memoization hash 7454db621d0bea1b8bd994a468752556 parsl.dataflow.memoization INFO: Task 4825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4825 try 0 launched on executor multi with executor id 4826 parsl.dataflow.dflow INFO: Standard output for task 4825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2ce9003c-676a-4620-b262-112e74ec49f8_isrForCrosstalkSources_855450_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2ce9003c-676a-4620-b262-112e74ec49f8_isrForCrosstalkSources_855450_46.stderr parsl.dataflow.dflow DEBUG: Task 4826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4826 has memoization hash c82385ea35c3b6f571092f3db0635181 parsl.dataflow.memoization INFO: Task 4826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4826 try 0 launched on executor multi with executor id 4827 parsl.dataflow.dflow INFO: Standard output for task 4826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/27dc8b4f-d9ff-4bec-b111-fc496e4a1b9b_isrForCrosstalkSources_855417_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/27dc8b4f-d9ff-4bec-b111-fc496e4a1b9b_isrForCrosstalkSources_855417_30.stderr parsl.dataflow.dflow DEBUG: Task 4827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4827 has memoization hash 1b5bbdbaee867783e411b5915529f86b parsl.dataflow.memoization INFO: Task 4827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4827 try 0 launched on executor multi with executor id 4828 parsl.dataflow.dflow INFO: Standard output for task 4827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c940364b-3037-4412-b7d1-441e72cb5a46_isrForCrosstalkSources_855452_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c940364b-3037-4412-b7d1-441e72cb5a46_isrForCrosstalkSources_855452_54.stderr parsl.dataflow.dflow DEBUG: Task 4828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4828 has memoization hash 1c044122cca16b7873aede63523d5625 parsl.dataflow.memoization INFO: Task 4828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4828 try 0 launched on executor multi with executor id 4829 parsl.dataflow.dflow INFO: Standard output for task 4828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a812ead1-b7e3-4e0a-aa1d-d64367628a9a_isrForCrosstalkSources_855439_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a812ead1-b7e3-4e0a-aa1d-d64367628a9a_isrForCrosstalkSources_855439_15.stderr parsl.dataflow.dflow DEBUG: Task 4829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4829 has memoization hash 9dcb17d7d18681ba7b3dbdc5b3e33c2f parsl.dataflow.memoization INFO: Task 4829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4829 try 0 launched on executor multi with executor id 4830 parsl.dataflow.dflow INFO: Standard output for task 4829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c70c0653-6907-4bdc-aa3c-512a23d59c07_isrForCrosstalkSources_855394_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c70c0653-6907-4bdc-aa3c-512a23d59c07_isrForCrosstalkSources_855394_38.stderr parsl.dataflow.dflow DEBUG: Task 4830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4830 has memoization hash feeb96b3e1474a900248ceddcdafe018 parsl.dataflow.memoization INFO: Task 4830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4830 try 0 launched on executor multi with executor id 4831 parsl.dataflow.dflow INFO: Standard output for task 4830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a943de96-93a7-40c5-87b7-7df9bc49f4d1_isrForCrosstalkSources_855421_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a943de96-93a7-40c5-87b7-7df9bc49f4d1_isrForCrosstalkSources_855421_57.stderr parsl.dataflow.dflow DEBUG: Task 4831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4831 has memoization hash b767d980bafbc144e45d0dbe19418529 parsl.dataflow.memoization INFO: Task 4831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4831 try 0 launched on executor multi with executor id 4832 parsl.dataflow.dflow INFO: Standard output for task 4831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/875b51c6-17fb-47c3-bed4-77b7185e3cea_isrForCrosstalkSources_855386_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/875b51c6-17fb-47c3-bed4-77b7185e3cea_isrForCrosstalkSources_855386_9.stderr parsl.dataflow.dflow DEBUG: Task 4832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4832 has memoization hash 094370421f092208d20396d773d74766 parsl.dataflow.memoization INFO: Task 4832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4832 try 0 launched on executor multi with executor id 4833 parsl.dataflow.dflow INFO: Standard output for task 4832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e3f8cd83-2128-4d5e-a6e3-4655ccf68ffc_isrForCrosstalkSources_855405_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e3f8cd83-2128-4d5e-a6e3-4655ccf68ffc_isrForCrosstalkSources_855405_13.stderr parsl.dataflow.dflow DEBUG: Task 4833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4833 has memoization hash a9464d26e3752c17cb1e89a2f0e91a8e parsl.dataflow.memoization INFO: Task 4833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4833 try 0 launched on executor multi with executor id 4834 parsl.dataflow.dflow INFO: Standard output for task 4833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3002315-b602-4730-992d-4ba28eaac179_isrForCrosstalkSources_855431_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3002315-b602-4730-992d-4ba28eaac179_isrForCrosstalkSources_855431_27.stderr parsl.dataflow.dflow DEBUG: Task 4834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4834 has memoization hash 9c3721f53112c290377f0f81813dd28d parsl.dataflow.memoization INFO: Task 4834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4834 try 0 launched on executor multi with executor id 4835 parsl.dataflow.dflow INFO: Standard output for task 4834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2ed6c27f-5d05-401d-9cc4-7163af8e7a3b_isrForCrosstalkSources_855457_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2ed6c27f-5d05-401d-9cc4-7163af8e7a3b_isrForCrosstalkSources_855457_49.stderr parsl.dataflow.dflow DEBUG: Task 4835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4835 has memoization hash f49fea3c0dda14cc4fbb5afd816f6e21 parsl.dataflow.memoization INFO: Task 4835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4835 try 0 launched on executor multi with executor id 4836 parsl.dataflow.dflow INFO: Standard output for task 4835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/cf627fa5-ff18-4652-a719-63a5d9f7486b_isrForCrosstalkSources_855436_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/cf627fa5-ff18-4652-a719-63a5d9f7486b_isrForCrosstalkSources_855436_6.stderr parsl.dataflow.dflow DEBUG: Task 4836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4836 has memoization hash 40549f0228708889cf9e58c92cab4a30 parsl.dataflow.memoization INFO: Task 4836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4836 try 0 launched on executor multi with executor id 4837 parsl.dataflow.dflow INFO: Standard output for task 4836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/446589b3-594c-4990-8d88-e200c3d32ea7_isrForCrosstalkSources_855420_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/446589b3-594c-4990-8d88-e200c3d32ea7_isrForCrosstalkSources_855420_61.stderr parsl.dataflow.dflow DEBUG: Task 4837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4837 has memoization hash a7e596efb51be165dfb8986d08ce6271 parsl.dataflow.memoization INFO: Task 4837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4837 try 0 launched on executor multi with executor id 4838 parsl.dataflow.dflow INFO: Standard output for task 4837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/675dd014-1231-4671-ab1e-1956ec2947aa_isrForCrosstalkSources_855416_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/675dd014-1231-4671-ab1e-1956ec2947aa_isrForCrosstalkSources_855416_19.stderr parsl.dataflow.dflow DEBUG: Task 4838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4838 has memoization hash 7097673ef3a4f7e233f28510f1f73777 parsl.dataflow.memoization INFO: Task 4838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4838 try 0 launched on executor multi with executor id 4839 parsl.dataflow.dflow INFO: Standard output for task 4838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/4dcd3b5b-ad80-47ed-bfa4-668c01913bdb_isrForCrosstalkSources_855410_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/4dcd3b5b-ad80-47ed-bfa4-668c01913bdb_isrForCrosstalkSources_855410_8.stderr parsl.dataflow.dflow DEBUG: Task 4839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4839 has memoization hash 941741b941863b7604be6d9d2cd7a875 parsl.dataflow.memoization INFO: Task 4839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4839 try 0 launched on executor multi with executor id 4840 parsl.dataflow.dflow INFO: Standard output for task 4839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/6f0db164-931e-4bae-870f-41cf75421dd9_isrForCrosstalkSources_855434_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/6f0db164-931e-4bae-870f-41cf75421dd9_isrForCrosstalkSources_855434_43.stderr parsl.dataflow.dflow DEBUG: Task 4840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4840 has memoization hash 3f3b9e43e339107560f48fddc0ce357c parsl.dataflow.memoization INFO: Task 4840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4840 try 0 launched on executor multi with executor id 4841 parsl.dataflow.dflow INFO: Standard output for task 4840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/93236832-ade0-4d80-850e-921856358f43_isrForCrosstalkSources_855405_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/93236832-ade0-4d80-850e-921856358f43_isrForCrosstalkSources_855405_38.stderr parsl.dataflow.dflow DEBUG: Task 4841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4841 has memoization hash 23ea09925c900a9140ef25a79ec5b731 parsl.dataflow.memoization INFO: Task 4841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4841 try 0 launched on executor multi with executor id 4842 parsl.dataflow.dflow INFO: Standard output for task 4841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/207dc7c7-cca6-413c-b01b-3efd3b677184_isrForCrosstalkSources_855397_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/207dc7c7-cca6-413c-b01b-3efd3b677184_isrForCrosstalkSources_855397_22.stderr parsl.dataflow.dflow DEBUG: Task 4842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4842 has memoization hash c58156b105cd28f8b3f3199270a6d0c5 parsl.dataflow.memoization INFO: Task 4842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4842 try 0 launched on executor multi with executor id 4843 parsl.dataflow.dflow INFO: Standard output for task 4842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8ca73e5-8eb3-4fb0-9755-c9a161dd7db6_isrForCrosstalkSources_855441_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8ca73e5-8eb3-4fb0-9755-c9a161dd7db6_isrForCrosstalkSources_855441_48.stderr parsl.dataflow.dflow DEBUG: Task 4843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4843 has memoization hash 39f9eb95ae6a68b03dc0f4ac7e48e91d parsl.dataflow.memoization INFO: Task 4843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4843 try 0 launched on executor multi with executor id 4844 parsl.dataflow.dflow INFO: Standard output for task 4843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e42025e-5015-40f8-aab1-df67ac0a739c_isrForCrosstalkSources_855384_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e42025e-5015-40f8-aab1-df67ac0a739c_isrForCrosstalkSources_855384_33.stderr parsl.dataflow.dflow DEBUG: Task 4844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4844 has memoization hash cbb1705fbd8d70e9af156a387d35ed17 parsl.dataflow.memoization INFO: Task 4844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4844 try 0 launched on executor multi with executor id 4845 parsl.dataflow.dflow INFO: Standard output for task 4844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01437f8b-e0aa-417f-9086-62bc231def8a_isrForCrosstalkSources_855433_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01437f8b-e0aa-417f-9086-62bc231def8a_isrForCrosstalkSources_855433_41.stderr parsl.dataflow.dflow DEBUG: Task 4845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4845 has memoization hash 5dbd4f7b36bbd025aecf1c042982e368 parsl.dataflow.memoization INFO: Task 4845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4845 try 0 launched on executor multi with executor id 4846 parsl.dataflow.dflow INFO: Standard output for task 4845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/72105a25-69dd-496d-8010-ac0e01663008_isrForCrosstalkSources_855427_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/72105a25-69dd-496d-8010-ac0e01663008_isrForCrosstalkSources_855427_15.stderr parsl.dataflow.dflow DEBUG: Task 4846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4846 has memoization hash 0f5ccb4fcc8223b9b5d55a440e1dbc11 parsl.dataflow.memoization INFO: Task 4846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4846 try 0 launched on executor multi with executor id 4847 parsl.dataflow.dflow INFO: Standard output for task 4846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/1c51dc82-fe7b-4985-b3b1-bfae6f7daf5d_isrForCrosstalkSources_855438_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/1c51dc82-fe7b-4985-b3b1-bfae6f7daf5d_isrForCrosstalkSources_855438_56.stderr parsl.dataflow.dflow DEBUG: Task 4847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4847 has memoization hash 8bf20d67db78c17450a4c05e8fbfa5d4 parsl.dataflow.memoization INFO: Task 4847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4847 try 0 launched on executor multi with executor id 4848 parsl.dataflow.dflow INFO: Standard output for task 4847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/fb84c2e8-1e93-4a6a-889b-c31533122803_isrForCrosstalkSources_855431_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/fb84c2e8-1e93-4a6a-889b-c31533122803_isrForCrosstalkSources_855431_12.stderr parsl.dataflow.dflow DEBUG: Task 4848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4848 has memoization hash 1b05c375e1eeae4587e65b3d0313bebb parsl.dataflow.memoization INFO: Task 4848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4848 try 0 launched on executor multi with executor id 4849 parsl.dataflow.dflow INFO: Standard output for task 4848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8b640d39-889d-4bf3-9ff0-c2c178a11bea_isrForCrosstalkSources_855454_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8b640d39-889d-4bf3-9ff0-c2c178a11bea_isrForCrosstalkSources_855454_29.stderr parsl.dataflow.dflow DEBUG: Task 4849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4849 has memoization hash 1ba17af13e84ebf6107a3ef1af4c11cd parsl.dataflow.memoization INFO: Task 4849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4849 try 0 launched on executor multi with executor id 4850 parsl.dataflow.dflow INFO: Standard output for task 4849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/550d8b55-464c-4a63-8c5a-e65cf29d287c_isrForCrosstalkSources_855405_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/550d8b55-464c-4a63-8c5a-e65cf29d287c_isrForCrosstalkSources_855405_34.stderr parsl.dataflow.dflow DEBUG: Task 4850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4850 has memoization hash 0006c2cf965c34c051ad32cf50016da9 parsl.dataflow.memoization INFO: Task 4850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4850 try 0 launched on executor multi with executor id 4851 parsl.dataflow.dflow INFO: Standard output for task 4850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/dc915129-1a86-47c7-9826-aa200d4abd3b_isrForCrosstalkSources_855420_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/dc915129-1a86-47c7-9826-aa200d4abd3b_isrForCrosstalkSources_855420_21.stderr parsl.dataflow.dflow DEBUG: Task 4851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4851 has memoization hash 7b1eb86627aa89c12e0b2e6e9359b87c parsl.dataflow.memoization INFO: Task 4851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4851 try 0 launched on executor multi with executor id 4852 parsl.dataflow.dflow INFO: Standard output for task 4851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c6aede6-cb46-4ab4-8186-e0fb38d7a603_isrForCrosstalkSources_855458_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c6aede6-cb46-4ab4-8186-e0fb38d7a603_isrForCrosstalkSources_855458_43.stderr parsl.dataflow.dflow DEBUG: Task 4852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4852 has memoization hash b1d915493a378a52fc055073c9954387 parsl.dataflow.memoization INFO: Task 4852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4852 try 0 launched on executor multi with executor id 4853 parsl.dataflow.dflow INFO: Standard output for task 4852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/04cfe93f-124d-4c79-ae6c-eb8e817b97f7_isrForCrosstalkSources_855405_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/04cfe93f-124d-4c79-ae6c-eb8e817b97f7_isrForCrosstalkSources_855405_2.stderr parsl.dataflow.dflow DEBUG: Task 4853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4853 has memoization hash 43d581412f6d093660b85b2e4dc18346 parsl.dataflow.memoization INFO: Task 4853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964206 proc_lsst.multi INFO: [multi] job_id 16964206 2024-03-09 14:51:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-09 14:51:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964206 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964206 parsl.executors.status_handling DEBUG: Launched block 180 on executor multi with job ID 16964206 parsl.executors.status_handling INFO: Allocated block ID 181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-181 2024-03-09 14:51:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-09 14:51:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4853 try 0 launched on executor multi with executor id 4854 parsl.dataflow.dflow INFO: Standard output for task 4853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6228ccb3-903b-4604-9d62-c9decdf2cb97_isrForCrosstalkSources_855404_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6228ccb3-903b-4604-9d62-c9decdf2cb97_isrForCrosstalkSources_855404_28.stderr parsl.dataflow.dflow DEBUG: Task 4854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4854 has memoization hash a7467cc20c012bd8780ac75190c746d8 parsl.dataflow.memoization INFO: Task 4854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4854 try 0 launched on executor multi with executor id 4855 parsl.dataflow.dflow INFO: Standard output for task 4854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ec0c2573-7ea1-4e51-919b-03d1074f090e_isrForCrosstalkSources_855397_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ec0c2573-7ea1-4e51-919b-03d1074f090e_isrForCrosstalkSources_855397_27.stderr parsl.dataflow.dflow DEBUG: Task 4855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4855 has memoization hash 2f85b3d7d32bc4efefadd6844cae28d3 parsl.dataflow.memoization INFO: Task 4855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4855 try 0 launched on executor multi with executor id 4856 parsl.dataflow.dflow INFO: Standard output for task 4855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de607337-986f-49cd-a7ce-5fbaae50ad32_isrForCrosstalkSources_855459_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de607337-986f-49cd-a7ce-5fbaae50ad32_isrForCrosstalkSources_855459_51.stderr parsl.dataflow.dflow DEBUG: Task 4856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4856 has memoization hash 38a209e51954cc4d634a7074c2829bb1 parsl.dataflow.memoization INFO: Task 4856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4856 try 0 launched on executor multi with executor id 4857 parsl.dataflow.dflow INFO: Standard output for task 4856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/427d879f-a56b-4172-9e21-a81656117038_isrForCrosstalkSources_855453_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/427d879f-a56b-4172-9e21-a81656117038_isrForCrosstalkSources_855453_7.stderr parsl.dataflow.dflow DEBUG: Task 4857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4857 has memoization hash d3f24f4d3964f91ddede3477ee9fa785 parsl.dataflow.memoization INFO: Task 4857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4857 try 0 launched on executor multi with executor id 4858 parsl.dataflow.dflow INFO: Standard output for task 4857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/174d128c-4467-41fd-918d-00fa8d50ebf4_isrForCrosstalkSources_855388_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/174d128c-4467-41fd-918d-00fa8d50ebf4_isrForCrosstalkSources_855388_48.stderr parsl.dataflow.dflow DEBUG: Task 4858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4858 has memoization hash 43f8e6f29dcd2e121ba47b8831cd0b55 parsl.dataflow.memoization INFO: Task 4858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4858 try 0 launched on executor multi with executor id 4859 parsl.dataflow.dflow INFO: Standard output for task 4858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e9041643-3869-46a4-8843-7d544381c689_isrForCrosstalkSources_855431_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e9041643-3869-46a4-8843-7d544381c689_isrForCrosstalkSources_855431_4.stderr parsl.dataflow.dflow DEBUG: Task 4859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4859 has memoization hash c64f6ebbb815b7c17699c636989305cd parsl.dataflow.memoization INFO: Task 4859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a58d94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4859 try 0 launched on executor multi with executor id 4860 parsl.dataflow.dflow INFO: Standard output for task 4859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8c5d357c-1e0c-40f2-a21e-d46d7486f434_isrForCrosstalkSources_855427_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8c5d357c-1e0c-40f2-a21e-d46d7486f434_isrForCrosstalkSources_855427_18.stderr parsl.dataflow.dflow DEBUG: Task 4860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4860 has memoization hash 297b7e0cf07654e40dadfd8443d40095 parsl.dataflow.memoization INFO: Task 4860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4860 try 0 launched on executor multi with executor id 4861 parsl.dataflow.dflow INFO: Standard output for task 4860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/35499a07-7474-4ef8-9a46-6c7c408e041c_isrForCrosstalkSources_855459_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/35499a07-7474-4ef8-9a46-6c7c408e041c_isrForCrosstalkSources_855459_1.stderr parsl.dataflow.dflow DEBUG: Task 4861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4861 has memoization hash 7c206e94af31b8d030ee0a426608ae2c parsl.dataflow.memoization INFO: Task 4861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4861 try 0 launched on executor multi with executor id 4862 parsl.dataflow.dflow INFO: Standard output for task 4861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3c248f39-57dd-42e5-bd29-3f8635ff64b8_isrForCrosstalkSources_855445_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3c248f39-57dd-42e5-bd29-3f8635ff64b8_isrForCrosstalkSources_855445_50.stderr parsl.dataflow.dflow DEBUG: Task 4862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4862 has memoization hash 9f4ff2bff43eda9e8743a4bf7b8bb037 parsl.dataflow.memoization INFO: Task 4862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4862 try 0 launched on executor multi with executor id 4863 parsl.dataflow.dflow INFO: Standard output for task 4862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d5f62376-37a9-4d50-b717-eafd0c541142_isrForCrosstalkSources_855449_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d5f62376-37a9-4d50-b717-eafd0c541142_isrForCrosstalkSources_855449_41.stderr parsl.dataflow.dflow DEBUG: Task 4863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4863 has memoization hash 19259ff49e5276c0a0114b42c0715036 parsl.dataflow.memoization INFO: Task 4863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4863 try 0 launched on executor multi with executor id 4864 parsl.dataflow.dflow INFO: Standard output for task 4863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0387494d-ddb3-4e49-bbdb-d728533e09b0_isrForCrosstalkSources_855408_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0387494d-ddb3-4e49-bbdb-d728533e09b0_isrForCrosstalkSources_855408_8.stderr parsl.dataflow.dflow DEBUG: Task 4864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4864 has memoization hash 03a299856a69a566d1d79bd4b22fbf0f parsl.dataflow.memoization INFO: Task 4864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4864 try 0 launched on executor multi with executor id 4865 parsl.dataflow.dflow INFO: Standard output for task 4864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7ed1bf79-e04d-4864-8365-127a32beb8bd_isrForCrosstalkSources_855414_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7ed1bf79-e04d-4864-8365-127a32beb8bd_isrForCrosstalkSources_855414_45.stderr parsl.dataflow.dflow DEBUG: Task 4865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4865 has memoization hash c216cced7afd91d854349b943e210050 parsl.dataflow.memoization INFO: Task 4865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4865 try 0 launched on executor multi with executor id 4866 parsl.dataflow.dflow INFO: Standard output for task 4865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/bd4ad25c-2e6b-4d8c-a713-50023a2e8f6d_isrForCrosstalkSources_855416_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/bd4ad25c-2e6b-4d8c-a713-50023a2e8f6d_isrForCrosstalkSources_855416_44.stderr parsl.dataflow.dflow DEBUG: Task 4866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4866 has memoization hash 992a2bf9e12d8f81bed313182bcf4e99 parsl.dataflow.memoization INFO: Task 4866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4866 try 0 launched on executor multi with executor id 4867 parsl.dataflow.dflow INFO: Standard output for task 4866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/1a44e85d-8fa5-44a5-a1ab-fb9277d9d6d0_isrForCrosstalkSources_855436_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/1a44e85d-8fa5-44a5-a1ab-fb9277d9d6d0_isrForCrosstalkSources_855436_21.stderr parsl.dataflow.dflow DEBUG: Task 4867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4867 has memoization hash 65c92434d519611d85eca6eb2c3cd761 parsl.dataflow.memoization INFO: Task 4867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4867 try 0 launched on executor multi with executor id 4868 parsl.dataflow.dflow INFO: Standard output for task 4867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/6a591308-efb9-47f8-8b0b-8d8f98143c26_isrForCrosstalkSources_855386_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/6a591308-efb9-47f8-8b0b-8d8f98143c26_isrForCrosstalkSources_855386_6.stderr parsl.dataflow.dflow DEBUG: Task 4868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4868 has memoization hash 07891d812bfabea49b88f993556afe98 parsl.dataflow.memoization INFO: Task 4868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4868 try 0 launched on executor multi with executor id 4869 parsl.dataflow.dflow INFO: Standard output for task 4868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b42ec553-19e3-4583-ba85-dca42432cfca_isrForCrosstalkSources_855440_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b42ec553-19e3-4583-ba85-dca42432cfca_isrForCrosstalkSources_855440_37.stderr parsl.dataflow.dflow DEBUG: Task 4869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4869 has memoization hash b9c6f238ac3aabcc15a040d3383025e3 parsl.dataflow.memoization INFO: Task 4869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4869 try 0 launched on executor multi with executor id 4870 parsl.dataflow.dflow INFO: Standard output for task 4869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/62621562-39ec-4122-8d9f-4525b9e372ec_isrForCrosstalkSources_855448_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/62621562-39ec-4122-8d9f-4525b9e372ec_isrForCrosstalkSources_855448_37.stderr parsl.dataflow.dflow DEBUG: Task 4870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4870 has memoization hash 15dd34003529e89493eb8d0b1f43f6b5 parsl.dataflow.memoization INFO: Task 4870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4870 try 0 launched on executor multi with executor id 4871 parsl.dataflow.dflow INFO: Standard output for task 4870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4eacd829-7b75-4c43-9fbc-523c015f99fd_isrForCrosstalkSources_855392_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4eacd829-7b75-4c43-9fbc-523c015f99fd_isrForCrosstalkSources_855392_38.stderr parsl.dataflow.dflow DEBUG: Task 4871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4871 has memoization hash a38494c057bce01125f283e9ae487cee parsl.dataflow.memoization INFO: Task 4871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4871 try 0 launched on executor multi with executor id 4872 parsl.dataflow.dflow INFO: Standard output for task 4871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7d7f939d-a3ec-490f-961c-940d5b16b3be_isrForCrosstalkSources_855388_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7d7f939d-a3ec-490f-961c-940d5b16b3be_isrForCrosstalkSources_855388_5.stderr parsl.dataflow.dflow DEBUG: Task 4872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4872 has memoization hash 09984d48381e5e6ee30a0237e07a6b66 parsl.dataflow.memoization INFO: Task 4872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4872 try 0 launched on executor multi with executor id 4873 parsl.dataflow.dflow INFO: Standard output for task 4872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/08a1d86a-01d5-44dc-8bfd-6533d5d8557e_isrForCrosstalkSources_855442_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/08a1d86a-01d5-44dc-8bfd-6533d5d8557e_isrForCrosstalkSources_855442_8.stderr parsl.dataflow.dflow DEBUG: Task 4873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4873 has memoization hash 4425e437562be82012e44af66f294068 parsl.dataflow.memoization INFO: Task 4873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4873 try 0 launched on executor multi with executor id 4874 parsl.dataflow.dflow INFO: Standard output for task 4873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/41a2a586-06bb-42d3-b195-8cf4ef9465bb_isrForCrosstalkSources_855412_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/41a2a586-06bb-42d3-b195-8cf4ef9465bb_isrForCrosstalkSources_855412_50.stderr parsl.dataflow.dflow DEBUG: Task 4874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4874 has memoization hash 1eec1cf14f13dba3eac56961481ecf2e parsl.dataflow.memoization INFO: Task 4874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4874 try 0 launched on executor multi with executor id 4875 parsl.dataflow.dflow INFO: Standard output for task 4874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32ace4ea-88b8-45a5-b826-d73dbf1c4f2e_isrForCrosstalkSources_855380_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32ace4ea-88b8-45a5-b826-d73dbf1c4f2e_isrForCrosstalkSources_855380_9.stderr parsl.dataflow.dflow DEBUG: Task 4875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4875 has memoization hash ffdac0c12acfd20ad1fd4467f1bf51ed parsl.dataflow.memoization INFO: Task 4875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4875 try 0 launched on executor multi with executor id 4876 parsl.dataflow.dflow INFO: Standard output for task 4875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/656f748e-7e10-4bc8-95b7-f5b10f2b1e15_isrForCrosstalkSources_855447_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/656f748e-7e10-4bc8-95b7-f5b10f2b1e15_isrForCrosstalkSources_855447_36.stderr parsl.dataflow.dflow DEBUG: Task 4876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4876 has memoization hash 60991185d3caca69d46d25fd29e34bc7 parsl.dataflow.memoization INFO: Task 4876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4876 try 0 launched on executor multi with executor id 4877 parsl.dataflow.dflow INFO: Standard output for task 4876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0c110788-8b45-4650-922b-1f152f62c9f6_isrForCrosstalkSources_855382_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0c110788-8b45-4650-922b-1f152f62c9f6_isrForCrosstalkSources_855382_34.stderr parsl.dataflow.dflow DEBUG: Task 4877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4877 has memoization hash dfd9062e8e681b4b642e8037a7f93463 parsl.dataflow.memoization INFO: Task 4877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4877 try 0 launched on executor multi with executor id 4878 parsl.dataflow.dflow INFO: Standard output for task 4877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e1b91ad9-d940-4c5a-92a1-d61468b99972_isrForCrosstalkSources_855454_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e1b91ad9-d940-4c5a-92a1-d61468b99972_isrForCrosstalkSources_855454_44.stderr parsl.dataflow.dflow DEBUG: Task 4878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4878 has memoization hash ef1659f06434a5eb931cc03dc5dee33a parsl.dataflow.memoization INFO: Task 4878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4878 try 0 launched on executor multi with executor id 4879 parsl.dataflow.dflow INFO: Standard output for task 4878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a48a9352-467b-47e5-a087-a916917bbd29_isrForCrosstalkSources_855441_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a48a9352-467b-47e5-a087-a916917bbd29_isrForCrosstalkSources_855441_2.stderr parsl.dataflow.dflow DEBUG: Task 4879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4879 has memoization hash a04ec3eb796d5858ce3b5e5722a25848 parsl.dataflow.memoization INFO: Task 4879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31aaf151fc1c59c857b1cec0929427e7 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6af8d7d8-1237-45fc-82a6-df104da8df4a_isrForCrosstalkSources_855456_26.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6af8d7d8-1237-45fc-82a6-df104da8df4a_isrForCrosstalkSources_855456_26.stderr parsl.dataflow.dflow INFO: Parsl task 4879 try 0 launched on executor multi with executor id 4880 parsl.dataflow.dflow INFO: Standard output for task 4879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/45a4dbd4-67ca-49f4-ac07-8b0acddcace5_isrForCrosstalkSources_855407_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/45a4dbd4-67ca-49f4-ac07-8b0acddcace5_isrForCrosstalkSources_855407_18.stderr parsl.dataflow.dflow DEBUG: Task 4880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4880 has memoization hash 8f0bfeec156c9a3921fbb83c7db67e8c parsl.dataflow.memoization INFO: Task 4880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4880 try 0 launched on executor multi with executor id 4881 parsl.dataflow.dflow INFO: Standard output for task 4880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/90fa3e93-c6a7-4b7c-a2ca-b6a30869a094_isrForCrosstalkSources_855394_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/90fa3e93-c6a7-4b7c-a2ca-b6a30869a094_isrForCrosstalkSources_855394_13.stderr parsl.dataflow.dflow DEBUG: Task 4881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4881 has memoization hash 1583ae77361a6889f992ce4411eec1e5 parsl.dataflow.memoization INFO: Task 4881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4881 try 0 launched on executor multi with executor id 4882 parsl.dataflow.dflow INFO: Standard output for task 4881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/a137844f-c259-4477-9294-351324f7d8ba_isrForCrosstalkSources_855436_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/a137844f-c259-4477-9294-351324f7d8ba_isrForCrosstalkSources_855436_12.stderr parsl.dataflow.dflow DEBUG: Task 4882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4882 has memoization hash 42df77d99efb55e917fe8eaa6d3b5b33 parsl.dataflow.memoization INFO: Task 4882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4882 try 0 launched on executor multi with executor id 4883 parsl.dataflow.dflow INFO: Standard output for task 4882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e95edb8-8136-49e5-9d40-1b7fea75ad5b_isrForCrosstalkSources_855449_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e95edb8-8136-49e5-9d40-1b7fea75ad5b_isrForCrosstalkSources_855449_38.stderr parsl.dataflow.dflow DEBUG: Task 4883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4883 has memoization hash 1e7efecde660bacbd1c86ad42cff0acf parsl.dataflow.memoization INFO: Task 4883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4883 try 0 launched on executor multi with executor id 4884 parsl.dataflow.dflow INFO: Standard output for task 4883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8bae7093-88f4-48f9-b56a-28fa786034b1_isrForCrosstalkSources_855443_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8bae7093-88f4-48f9-b56a-28fa786034b1_isrForCrosstalkSources_855443_31.stderr parsl.dataflow.dflow DEBUG: Task 4884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4884 has memoization hash dc3de497a6679df1864166e6d29363f0 parsl.dataflow.memoization INFO: Task 4884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4884 try 0 launched on executor multi with executor id 4885 parsl.dataflow.dflow INFO: Standard output for task 4884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3220b63e-d273-4c39-b115-d38c5d679ac9_isrForCrosstalkSources_855443_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3220b63e-d273-4c39-b115-d38c5d679ac9_isrForCrosstalkSources_855443_5.stderr parsl.dataflow.dflow DEBUG: Task 4885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4885 has memoization hash d4793e3b35bd6028ca4912814622bda5 parsl.dataflow.memoization INFO: Task 4885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4885 try 0 launched on executor multi with executor id 4886 parsl.dataflow.dflow INFO: Standard output for task 4885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/170f4d37-5768-4792-ba89-a32f06a34019_isrForCrosstalkSources_855424_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/170f4d37-5768-4792-ba89-a32f06a34019_isrForCrosstalkSources_855424_41.stderr parsl.dataflow.dflow DEBUG: Task 4886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4886 has memoization hash 59ba171e1328bba792ceb575b3a1e31d parsl.dataflow.memoization INFO: Task 4886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4886 try 0 launched on executor multi with executor id 4887 parsl.dataflow.dflow INFO: Standard output for task 4886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/af125d11-db53-4883-b147-ba67a946b86d_isrForCrosstalkSources_855383_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/af125d11-db53-4883-b147-ba67a946b86d_isrForCrosstalkSources_855383_25.stderr parsl.dataflow.dflow DEBUG: Task 4887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4887 has memoization hash 9e11b0fbca208d18883c8728bbe3bcbf parsl.dataflow.memoization INFO: Task 4887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2df8951954b410c9b5dc6b9098553bf7 with result from task 707 parsl.dataflow.dflow INFO: Parsl task 4887 try 0 launched on executor multi with executor id 4888 parsl.dataflow.dflow INFO: Standard output for task 4887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/f89faf4c-e467-4be4-b40f-f2d5bd891d56_isrForCrosstalkSources_855389_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/f89faf4c-e467-4be4-b40f-f2d5bd891d56_isrForCrosstalkSources_855389_13.stderr parsl.dataflow.dflow DEBUG: Task 4888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4888 has memoization hash 4f73b355584d9a97cfc6e41a2719f1f1 parsl.dataflow.memoization INFO: Task 4888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4888 try 0 launched on executor multi with executor id 4889 parsl.dataflow.dflow INFO: Standard output for task 4888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9c7fb924-82fe-4742-a58c-470c8396c28f_isrForCrosstalkSources_855434_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9c7fb924-82fe-4742-a58c-470c8396c28f_isrForCrosstalkSources_855434_42.stderr parsl.dataflow.dflow DEBUG: Task 4889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4889 has memoization hash 40bc729924b330ca066911f73644e14a parsl.dataflow.memoization INFO: Task 4889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4889 try 0 launched on executor multi with executor id 4890 parsl.dataflow.dflow INFO: Standard output for task 4889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3931c9cc-9e3d-4ad7-a5a9-2cb49c2aa602_isrForCrosstalkSources_855383_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3931c9cc-9e3d-4ad7-a5a9-2cb49c2aa602_isrForCrosstalkSources_855383_62.stderr parsl.dataflow.dflow DEBUG: Task 4890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4890 has memoization hash 4225ba58d2d5d5d16af39fca00c5b51c parsl.dataflow.memoization INFO: Task 4890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3a64e963-9a60-4a76-a180-ba28c195be1c_isrForCrosstalkSources_855402_14.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3a64e963-9a60-4a76-a180-ba28c195be1c_isrForCrosstalkSources_855402_14.stderr parsl.dataflow.dflow INFO: Parsl task 4890 try 0 launched on executor multi with executor id 4891 parsl.dataflow.dflow INFO: Standard output for task 4890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7028b4f7-b823-446a-a26c-053aa2b8d961_isrForCrosstalkSources_855437_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7028b4f7-b823-446a-a26c-053aa2b8d961_isrForCrosstalkSources_855437_49.stderr parsl.dataflow.dflow DEBUG: Task 4891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4891 has memoization hash e7ac28ae3f3abdc9c877a3000787fff1 parsl.dataflow.memoization INFO: Task 4891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4891 try 0 launched on executor multi with executor id 4892 parsl.dataflow.dflow INFO: Standard output for task 4891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18be49f4-6a86-476f-a1da-c256ae754150_isrForCrosstalkSources_855443_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18be49f4-6a86-476f-a1da-c256ae754150_isrForCrosstalkSources_855443_52.stderr parsl.dataflow.dflow DEBUG: Task 4892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4892 has memoization hash 593cf816c74681e2f600ef37a3e46543 parsl.dataflow.memoization INFO: Task 4892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4892 try 0 launched on executor multi with executor id 4893 parsl.dataflow.dflow INFO: Standard output for task 4892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/76911753-91d6-452b-b80a-f3e488295360_isrForCrosstalkSources_855450_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/76911753-91d6-452b-b80a-f3e488295360_isrForCrosstalkSources_855450_57.stderr parsl.dataflow.dflow DEBUG: Task 4893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4893 has memoization hash d1ccd23d73430e5e78f37eee8c59caac parsl.dataflow.memoization INFO: Task 4893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4893 try 0 launched on executor multi with executor id 4894 parsl.dataflow.dflow INFO: Standard output for task 4893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/48f7dc44-0468-4903-a31b-86cab03ab4af_isrForCrosstalkSources_855433_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/48f7dc44-0468-4903-a31b-86cab03ab4af_isrForCrosstalkSources_855433_17.stderr parsl.dataflow.dflow DEBUG: Task 4894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4894 has memoization hash aa3d83b8b18fd7981d9dda6ad47bffe2 parsl.dataflow.memoization INFO: Task 4894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4894 try 0 launched on executor multi with executor id 4895 parsl.dataflow.dflow INFO: Standard output for task 4894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/06502dcb-30dd-4032-ae28-49fe3a30fb4c_isrForCrosstalkSources_855423_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/06502dcb-30dd-4032-ae28-49fe3a30fb4c_isrForCrosstalkSources_855423_38.stderr parsl.dataflow.dflow DEBUG: Task 4895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4895 has memoization hash cdbb88b5f06c3538760a9dcda6e5369c parsl.dataflow.memoization INFO: Task 4895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4895 try 0 launched on executor multi with executor id 4896 parsl.dataflow.dflow INFO: Standard output for task 4895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/4c55e689-b11d-4112-8072-d9bd37d7f43d_isrForCrosstalkSources_855394_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/4c55e689-b11d-4112-8072-d9bd37d7f43d_isrForCrosstalkSources_855394_31.stderr parsl.dataflow.dflow DEBUG: Task 4896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4896 has memoization hash 3e2a6f4e5eb436ba8f5f2103061016de parsl.dataflow.memoization INFO: Task 4896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4896 try 0 launched on executor multi with executor id 4897 parsl.dataflow.dflow INFO: Standard output for task 4896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/540eccec-71cf-4c35-90be-403396901b25_isrForCrosstalkSources_855424_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/540eccec-71cf-4c35-90be-403396901b25_isrForCrosstalkSources_855424_20.stderr parsl.dataflow.dflow DEBUG: Task 4897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4897 has memoization hash 243969f0d9d8a32a721e7e48435ea493 parsl.dataflow.memoization INFO: Task 4897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4897 try 0 launched on executor multi with executor id 4898 parsl.dataflow.dflow INFO: Standard output for task 4897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/56c4f8c4-1e40-4169-baee-55ef48617184_isrForCrosstalkSources_855421_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/56c4f8c4-1e40-4169-baee-55ef48617184_isrForCrosstalkSources_855421_36.stderr parsl.dataflow.dflow DEBUG: Task 4898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4898 has memoization hash ff63b87837728dac60297e75cd7e8c2a parsl.dataflow.memoization INFO: Task 4898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4898 try 0 launched on executor multi with executor id 4899 parsl.dataflow.dflow INFO: Standard output for task 4898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c4c85315-660a-4137-9570-2b3f26caf300_isrForCrosstalkSources_855434_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c4c85315-660a-4137-9570-2b3f26caf300_isrForCrosstalkSources_855434_58.stderr parsl.dataflow.dflow DEBUG: Task 4899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4899 has memoization hash f05bc40ace9583764d3581ad33fde588 parsl.dataflow.memoization INFO: Task 4899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4899 try 0 launched on executor multi with executor id 4900 parsl.dataflow.dflow INFO: Standard output for task 4899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4f014aec-db58-4476-ab0b-371cd054d934_isrForCrosstalkSources_855385_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4f014aec-db58-4476-ab0b-371cd054d934_isrForCrosstalkSources_855385_61.stderr parsl.dataflow.dflow DEBUG: Task 4900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4900 has memoization hash 11c3a8e3395c0e767aa5b65232b41dca parsl.dataflow.memoization INFO: Task 4900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4900 try 0 launched on executor multi with executor id 4901 parsl.dataflow.dflow INFO: Standard output for task 4900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8d9496db-a647-4b77-921f-38c731e9a11f_isrForCrosstalkSources_855425_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8d9496db-a647-4b77-921f-38c731e9a11f_isrForCrosstalkSources_855425_41.stderr parsl.dataflow.dflow DEBUG: Task 4901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4901 has memoization hash 7d8d0dbc0f6c515cba2264729c2f4781 parsl.dataflow.memoization INFO: Task 4901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ed5a0a196fa38c3ea24fc248a785e1 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b68b7ed-8052-4213-9169-a35f737a0055_isrForCrosstalkSources_855400_1.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b68b7ed-8052-4213-9169-a35f737a0055_isrForCrosstalkSources_855400_1.stderr parsl.dataflow.dflow INFO: Parsl task 4901 try 0 launched on executor multi with executor id 4902 parsl.dataflow.dflow INFO: Standard output for task 4901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ec4ae99a-1710-450b-a0a1-b329c4ad5471_isrForCrosstalkSources_855405_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ec4ae99a-1710-450b-a0a1-b329c4ad5471_isrForCrosstalkSources_855405_42.stderr parsl.dataflow.dflow DEBUG: Task 4902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4902 has memoization hash 3f1e953f54a9b33de59e9a22ca12be51 parsl.dataflow.memoization INFO: Task 4902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4902 try 0 launched on executor multi with executor id 4903 parsl.dataflow.dflow INFO: Standard output for task 4902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/176ab8d5-d3d2-4d74-abb3-ce4e33a41161_isrForCrosstalkSources_855385_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/176ab8d5-d3d2-4d74-abb3-ce4e33a41161_isrForCrosstalkSources_855385_24.stderr parsl.dataflow.dflow DEBUG: Task 4903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4903 has memoization hash ca083509a5215038def2b9b33babd532 parsl.dataflow.memoization INFO: Task 4903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4903 try 0 launched on executor multi with executor id 4904 parsl.dataflow.dflow INFO: Standard output for task 4903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/b37fcca0-4a69-4d0c-b5c4-361a4797147b_isrForCrosstalkSources_855404_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/b37fcca0-4a69-4d0c-b5c4-361a4797147b_isrForCrosstalkSources_855404_3.stderr parsl.dataflow.dflow DEBUG: Task 4904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4904 has memoization hash 833b492fcec8e92554980aaf96fc1a43 parsl.dataflow.memoization INFO: Task 4904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4904 try 0 launched on executor multi with executor id 4905 parsl.dataflow.dflow INFO: Standard output for task 4904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9721d82c-bc19-450e-85cc-70d01aebd479_isrForCrosstalkSources_855414_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9721d82c-bc19-450e-85cc-70d01aebd479_isrForCrosstalkSources_855414_4.stderr parsl.dataflow.dflow DEBUG: Task 4905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4905 has memoization hash 0e74cd6e10303032cf385d1d1a400b64 parsl.dataflow.memoization INFO: Task 4905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964207 proc_lsst.multi INFO: [multi] job_id 16964207 2024-03-09 14:51:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-09 14:51:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964207 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964207 parsl.executors.status_handling DEBUG: Launched block 181 on executor multi with job ID 16964207 parsl.executors.status_handling INFO: Allocated block ID 182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-182 2024-03-09 14:51:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 2024-03-09 14:51:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-09 14:51:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4905 try 0 launched on executor multi with executor id 4906 parsl.dataflow.dflow INFO: Standard output for task 4905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/08157e45-c397-49bc-ba61-e2d7a71daf36_isrForCrosstalkSources_855409_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/08157e45-c397-49bc-ba61-e2d7a71daf36_isrForCrosstalkSources_855409_28.stderr parsl.dataflow.dflow DEBUG: Task 4906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4906 has memoization hash 2e17af8509dfd06e1836eff4179ac3ad parsl.dataflow.memoization INFO: Task 4906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4906 try 0 launched on executor multi with executor id 4907 parsl.dataflow.dflow INFO: Standard output for task 4906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cb24dbdc-78ad-4c88-a7ce-fb6574be18ac_isrForCrosstalkSources_855432_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cb24dbdc-78ad-4c88-a7ce-fb6574be18ac_isrForCrosstalkSources_855432_62.stderr parsl.dataflow.dflow DEBUG: Task 4907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4907 has memoization hash dfba593a60b4bc4994394d0e61ff9677 parsl.dataflow.memoization INFO: Task 4907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4907 try 0 launched on executor multi with executor id 4908 parsl.dataflow.dflow INFO: Standard output for task 4907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8f2265f4-4007-4522-9f50-a00a3cf9d3fa_isrForCrosstalkSources_855379_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8f2265f4-4007-4522-9f50-a00a3cf9d3fa_isrForCrosstalkSources_855379_55.stderr parsl.dataflow.dflow DEBUG: Task 4908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4908 has memoization hash 8845bae49298e2df5c9daa79fecbd466 parsl.dataflow.memoization INFO: Task 4908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a576e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4908 try 0 launched on executor multi with executor id 4909 parsl.dataflow.dflow INFO: Standard output for task 4908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/59fe27f0-7d4e-4967-b41d-4f42830831e3_isrForCrosstalkSources_855408_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/59fe27f0-7d4e-4967-b41d-4f42830831e3_isrForCrosstalkSources_855408_37.stderr parsl.dataflow.dflow DEBUG: Task 4909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4909 has memoization hash 8c89269f8bc6155effd8f0eae570a24d parsl.dataflow.memoization INFO: Task 4909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4909 try 0 launched on executor multi with executor id 4910 parsl.dataflow.dflow INFO: Standard output for task 4909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ba97768e-b8ee-4329-9459-63af57c463bc_isrForCrosstalkSources_855451_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ba97768e-b8ee-4329-9459-63af57c463bc_isrForCrosstalkSources_855451_7.stderr parsl.dataflow.dflow DEBUG: Task 4910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4910 has memoization hash 75eea4c6075a7bcc494f568d62cb49ab parsl.dataflow.memoization INFO: Task 4910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4910 try 0 launched on executor multi with executor id 4911 parsl.dataflow.dflow INFO: Standard output for task 4910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0ef66155-4149-4439-babf-57bfbf8fdb71_isrForCrosstalkSources_855450_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0ef66155-4149-4439-babf-57bfbf8fdb71_isrForCrosstalkSources_855450_25.stderr parsl.dataflow.dflow DEBUG: Task 4911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4911 has memoization hash eb08fa4ed680f8c296e2989135c374cd parsl.dataflow.memoization INFO: Task 4911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4911 try 0 launched on executor multi with executor id 4912 parsl.dataflow.dflow INFO: Standard output for task 4911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a883641e-b10e-4b58-bc1f-7eb3b84d2256_isrForCrosstalkSources_855434_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a883641e-b10e-4b58-bc1f-7eb3b84d2256_isrForCrosstalkSources_855434_29.stderr parsl.dataflow.dflow DEBUG: Task 4912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4912 has memoization hash 4c82ac4f89f01fc7f2d241cb99fda994 parsl.dataflow.memoization INFO: Task 4912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4912 try 0 launched on executor multi with executor id 4913 parsl.dataflow.dflow INFO: Standard output for task 4912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/795d3cf5-b8f6-4bf5-a8f8-b0362fe41d80_isrForCrosstalkSources_855417_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/795d3cf5-b8f6-4bf5-a8f8-b0362fe41d80_isrForCrosstalkSources_855417_44.stderr parsl.dataflow.dflow DEBUG: Task 4913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4913 has memoization hash a1ee657b57e50dec85ae2770d2d1122e parsl.dataflow.memoization INFO: Task 4913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4913 try 0 launched on executor multi with executor id 4914 parsl.dataflow.dflow INFO: Standard output for task 4913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ea0449e-5d4a-4ed6-ba3e-fcd9a52c8683_isrForCrosstalkSources_855418_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ea0449e-5d4a-4ed6-ba3e-fcd9a52c8683_isrForCrosstalkSources_855418_20.stderr parsl.dataflow.dflow DEBUG: Task 4914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4914 has memoization hash fcc7acb95410f412e3ba0f11afe3ffef parsl.dataflow.memoization INFO: Task 4914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4914 try 0 launched on executor multi with executor id 4915 parsl.dataflow.dflow INFO: Standard output for task 4914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/97dda977-a30d-4e06-a339-3f4888a58cf8_isrForCrosstalkSources_855437_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/97dda977-a30d-4e06-a339-3f4888a58cf8_isrForCrosstalkSources_855437_7.stderr parsl.dataflow.dflow DEBUG: Task 4915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4915 has memoization hash 33bfd1db72715b674f3bf89c3a3a429e parsl.dataflow.memoization INFO: Task 4915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 953848568fc55dc8087ed8072bb740a7 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8463841-f13e-41ae-accb-4a462ddefd53_isrForCrosstalkSources_855435_21.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8463841-f13e-41ae-accb-4a462ddefd53_isrForCrosstalkSources_855435_21.stderr parsl.dataflow.dflow INFO: Parsl task 4915 try 0 launched on executor multi with executor id 4916 parsl.dataflow.dflow INFO: Standard output for task 4915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/aa63724e-7131-4086-a41b-25f8a3daf8f9_isrForCrosstalkSources_855406_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/aa63724e-7131-4086-a41b-25f8a3daf8f9_isrForCrosstalkSources_855406_28.stderr parsl.dataflow.dflow DEBUG: Task 4916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4916 has memoization hash d35e1619307caa1ab3c8c1b7bba1f660 parsl.dataflow.memoization INFO: Task 4916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4916 try 0 launched on executor multi with executor id 4917 parsl.dataflow.dflow INFO: Standard output for task 4916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e899da78-9c5c-43a5-aab9-0ee0df0a5ff1_isrForCrosstalkSources_855451_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e899da78-9c5c-43a5-aab9-0ee0df0a5ff1_isrForCrosstalkSources_855451_11.stderr parsl.dataflow.dflow DEBUG: Task 4917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4917 has memoization hash 32a202ce82269cb724a3fe9d18f4a1b7 parsl.dataflow.memoization INFO: Task 4917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4917 try 0 launched on executor multi with executor id 4918 parsl.dataflow.dflow INFO: Standard output for task 4917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c5cf71e6-8f99-448a-ae3f-a6c8879358d4_isrForCrosstalkSources_855390_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c5cf71e6-8f99-448a-ae3f-a6c8879358d4_isrForCrosstalkSources_855390_50.stderr parsl.dataflow.dflow DEBUG: Task 4918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4918 has memoization hash 68600accc86407b3582d5b0d8c0b6185 parsl.dataflow.memoization INFO: Task 4918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4918 try 0 launched on executor multi with executor id 4919 parsl.dataflow.dflow INFO: Standard output for task 4918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0c3bf50a-a500-4289-9e0f-fc3b579ae90e_isrForCrosstalkSources_855458_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0c3bf50a-a500-4289-9e0f-fc3b579ae90e_isrForCrosstalkSources_855458_56.stderr parsl.dataflow.dflow DEBUG: Task 4919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4919 has memoization hash 77c7ae4c5a58d07fc701dbcccace2f94 parsl.dataflow.memoization INFO: Task 4919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4919 try 0 launched on executor multi with executor id 4920 parsl.dataflow.dflow INFO: Standard output for task 4919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/932b6ee8-0b58-490d-bc15-7338e41934cf_isrForCrosstalkSources_855417_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/932b6ee8-0b58-490d-bc15-7338e41934cf_isrForCrosstalkSources_855417_60.stderr parsl.dataflow.dflow DEBUG: Task 4920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4920 has memoization hash 3ed0d665c2e6ef999deda36122abe586 parsl.dataflow.memoization INFO: Task 4920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4920 try 0 launched on executor multi with executor id 4921 parsl.dataflow.dflow INFO: Standard output for task 4920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fd04b6a5-98f5-4379-8e88-3c8e3a5ee15e_isrForCrosstalkSources_855459_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fd04b6a5-98f5-4379-8e88-3c8e3a5ee15e_isrForCrosstalkSources_855459_50.stderr parsl.dataflow.dflow DEBUG: Task 4921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4921 has memoization hash d7a4eefa50764e061a48fcb233af0d6a parsl.dataflow.memoization INFO: Task 4921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4921 try 0 launched on executor multi with executor id 4922 parsl.dataflow.dflow INFO: Standard output for task 4921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36eb6f22-8242-4dba-809a-65f12afa8f8f_isrForCrosstalkSources_855390_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36eb6f22-8242-4dba-809a-65f12afa8f8f_isrForCrosstalkSources_855390_52.stderr parsl.dataflow.dflow DEBUG: Task 4922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4922 has memoization hash bdc3454d478c22082eec6d22d3917c9a parsl.dataflow.memoization INFO: Task 4922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4922 try 0 launched on executor multi with executor id 4923 parsl.dataflow.dflow INFO: Standard output for task 4922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8fed1919-3876-472f-935c-a4862782c3df_isrForCrosstalkSources_855402_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8fed1919-3876-472f-935c-a4862782c3df_isrForCrosstalkSources_855402_22.stderr parsl.dataflow.dflow DEBUG: Task 4923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4923 has memoization hash b95936935ca02fe5bc9bc740609a7804 parsl.dataflow.memoization INFO: Task 4923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4923 try 0 launched on executor multi with executor id 4924 parsl.dataflow.dflow INFO: Standard output for task 4923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/71e62f60-3261-4c5c-b3f9-afbb73dc9a81_isrForCrosstalkSources_855381_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/71e62f60-3261-4c5c-b3f9-afbb73dc9a81_isrForCrosstalkSources_855381_34.stderr parsl.dataflow.dflow DEBUG: Task 4924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4924 has memoization hash b09062fde37b5802fda2f22ab3f7b780 parsl.dataflow.memoization INFO: Task 4924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4924 try 0 launched on executor multi with executor id 4925 parsl.dataflow.dflow INFO: Standard output for task 4924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/0ee8b3aa-c50c-485d-939e-117c975ebaa2_isrForCrosstalkSources_855415_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/0ee8b3aa-c50c-485d-939e-117c975ebaa2_isrForCrosstalkSources_855415_57.stderr parsl.dataflow.dflow DEBUG: Task 4925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4925 has memoization hash 63d6c7c04c0faef2c9984c71d5431770 parsl.dataflow.memoization INFO: Task 4925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4925 try 0 launched on executor multi with executor id 4926 parsl.dataflow.dflow INFO: Standard output for task 4925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9a1ef7a8-8cd7-467b-b79f-3d3addebc993_isrForCrosstalkSources_855393_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9a1ef7a8-8cd7-467b-b79f-3d3addebc993_isrForCrosstalkSources_855393_37.stderr parsl.dataflow.dflow DEBUG: Task 4926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4926 has memoization hash f43c5dc9c10a6d9a32bcfccee416adf8 parsl.dataflow.memoization INFO: Task 4926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4926 try 0 launched on executor multi with executor id 4927 parsl.dataflow.dflow INFO: Standard output for task 4926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2b3111ba-7bd3-4d5e-be5e-3eb560f11298_isrForCrosstalkSources_855418_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2b3111ba-7bd3-4d5e-be5e-3eb560f11298_isrForCrosstalkSources_855418_29.stderr parsl.dataflow.dflow DEBUG: Task 4927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4927 has memoization hash 09b17e1db10d5782e70bf87e80199ab9 parsl.dataflow.memoization INFO: Task 4927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4927 try 0 launched on executor multi with executor id 4928 parsl.dataflow.dflow INFO: Standard output for task 4927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c0090916-f123-4eac-8c1a-c07c3ea70807_isrForCrosstalkSources_855401_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c0090916-f123-4eac-8c1a-c07c3ea70807_isrForCrosstalkSources_855401_39.stderr parsl.dataflow.dflow DEBUG: Task 4928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4928 has memoization hash dbf11202a07fb1c9d2f94fe5c1253c20 parsl.dataflow.memoization INFO: Task 4928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4928 try 0 launched on executor multi with executor id 4929 parsl.dataflow.dflow INFO: Standard output for task 4928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1e889bae-85e9-44ef-a4a1-aeac2e265628_isrForCrosstalkSources_855423_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1e889bae-85e9-44ef-a4a1-aeac2e265628_isrForCrosstalkSources_855423_7.stderr parsl.dataflow.dflow DEBUG: Task 4929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4929 has memoization hash 5f51e3db88a1715037e17274e8e54cd7 parsl.dataflow.memoization INFO: Task 4929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4929 try 0 launched on executor multi with executor id 4930 parsl.dataflow.dflow INFO: Standard output for task 4929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/6b36c139-aeea-4981-aa02-48d61c31b189_isrForCrosstalkSources_855381_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/6b36c139-aeea-4981-aa02-48d61c31b189_isrForCrosstalkSources_855381_62.stderr parsl.dataflow.dflow DEBUG: Task 4930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4930 has memoization hash bdce8b5024ad354905c685ce2563d8b5 parsl.dataflow.memoization INFO: Task 4930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4930 try 0 launched on executor multi with executor id 4931 parsl.dataflow.dflow INFO: Standard output for task 4930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5fc6a04c-60e0-4f4a-98fe-288705230346_isrForCrosstalkSources_855427_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5fc6a04c-60e0-4f4a-98fe-288705230346_isrForCrosstalkSources_855427_41.stderr parsl.dataflow.dflow DEBUG: Task 4931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4931 has memoization hash 2f77ca5eed3196ae593e0710a01e0ce3 parsl.dataflow.memoization INFO: Task 4931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4931 try 0 launched on executor multi with executor id 4932 parsl.dataflow.dflow INFO: Standard output for task 4931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1327d6fc-3bd0-4c49-8ba3-292432a91b64_isrForCrosstalkSources_855446_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1327d6fc-3bd0-4c49-8ba3-292432a91b64_isrForCrosstalkSources_855446_30.stderr parsl.dataflow.dflow DEBUG: Task 4932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4932 has memoization hash d8a181c9559636233ad5ec70245fa35c parsl.dataflow.memoization INFO: Task 4932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4932 try 0 launched on executor multi with executor id 4933 parsl.dataflow.dflow INFO: Standard output for task 4932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35cd9a08-4763-42c3-93a0-00ec65a3e796_isrForCrosstalkSources_855385_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35cd9a08-4763-42c3-93a0-00ec65a3e796_isrForCrosstalkSources_855385_28.stderr parsl.dataflow.dflow DEBUG: Task 4933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4933 has memoization hash 7ef64a953ae40416114e2bcfa8a5613b parsl.dataflow.memoization INFO: Task 4933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4933 try 0 launched on executor multi with executor id 4934 parsl.dataflow.dflow INFO: Standard output for task 4933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9b0d3e02-628b-4efc-bafb-b2ef045305b4_isrForCrosstalkSources_855452_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9b0d3e02-628b-4efc-bafb-b2ef045305b4_isrForCrosstalkSources_855452_48.stderr parsl.dataflow.dflow DEBUG: Task 4934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4934 has memoization hash ca6cd41f3c839e41584b60648d99485b parsl.dataflow.memoization INFO: Task 4934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4934 try 0 launched on executor multi with executor id 4935 parsl.dataflow.dflow INFO: Standard output for task 4934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b6f802ed-1426-40fb-b6b3-735eaf9675b6_isrForCrosstalkSources_855398_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b6f802ed-1426-40fb-b6b3-735eaf9675b6_isrForCrosstalkSources_855398_27.stderr parsl.dataflow.dflow DEBUG: Task 4935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4935 has memoization hash 43755f9b110b66435e9e8f80c73c1f38 parsl.dataflow.memoization INFO: Task 4935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4935 try 0 launched on executor multi with executor id 4936 parsl.dataflow.dflow INFO: Standard output for task 4935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6a23d830-d0f9-4bee-93cf-3a15dd6a9e3b_isrForCrosstalkSources_855379_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6a23d830-d0f9-4bee-93cf-3a15dd6a9e3b_isrForCrosstalkSources_855379_58.stderr parsl.dataflow.dflow DEBUG: Task 4936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4936 has memoization hash ea64453bc5977322af9b05bb60febac1 parsl.dataflow.memoization INFO: Task 4936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4936 try 0 launched on executor multi with executor id 4937 parsl.dataflow.dflow INFO: Standard output for task 4936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99c6f6d4-2617-42e0-a919-7e1db4efe6ae_isrForCrosstalkSources_855406_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99c6f6d4-2617-42e0-a919-7e1db4efe6ae_isrForCrosstalkSources_855406_50.stderr parsl.dataflow.dflow DEBUG: Task 4937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4937 has memoization hash 0f2e342eba714b91c9046fdc49f199b1 parsl.dataflow.memoization INFO: Task 4937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4937 try 0 launched on executor multi with executor id 4938 parsl.dataflow.dflow INFO: Standard output for task 4937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/98bf6c0c-c2fc-438e-a3f1-bd9ab9fae140_isrForCrosstalkSources_855396_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/98bf6c0c-c2fc-438e-a3f1-bd9ab9fae140_isrForCrosstalkSources_855396_4.stderr parsl.dataflow.dflow DEBUG: Task 4938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4938 has memoization hash 6cfc59a8dfbcd48911199c8ca6415fd3 parsl.dataflow.memoization INFO: Task 4938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4938 try 0 launched on executor multi with executor id 4939 parsl.dataflow.dflow INFO: Standard output for task 4938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/81c58a58-5817-4556-9f6d-cbd55256530e_isrForCrosstalkSources_855456_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/81c58a58-5817-4556-9f6d-cbd55256530e_isrForCrosstalkSources_855456_35.stderr parsl.dataflow.dflow DEBUG: Task 4939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4939 has memoization hash d803974eac97cd386b3a707a43e978d5 parsl.dataflow.memoization INFO: Task 4939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4939 try 0 launched on executor multi with executor id 4940 parsl.dataflow.dflow INFO: Standard output for task 4939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/71f0216f-a407-40a9-93e3-2df632f27883_isrForCrosstalkSources_855418_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/71f0216f-a407-40a9-93e3-2df632f27883_isrForCrosstalkSources_855418_2.stderr parsl.dataflow.dflow DEBUG: Task 4940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4940 has memoization hash 55c77082a4fd96ec57571780bd1af486 parsl.dataflow.memoization INFO: Task 4940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4940 try 0 launched on executor multi with executor id 4941 parsl.dataflow.dflow INFO: Standard output for task 4940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ae1542fd-8ccf-46e0-858e-4ad0930ac6be_isrForCrosstalkSources_855455_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ae1542fd-8ccf-46e0-858e-4ad0930ac6be_isrForCrosstalkSources_855455_18.stderr parsl.dataflow.dflow DEBUG: Task 4941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4941 has memoization hash fcce2f13405da39f2606ffc5561202f5 parsl.dataflow.memoization INFO: Task 4941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4941 try 0 launched on executor multi with executor id 4942 parsl.dataflow.dflow INFO: Standard output for task 4941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fcbf673a-f943-4309-a91e-f1b16086e56a_isrForCrosstalkSources_855450_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fcbf673a-f943-4309-a91e-f1b16086e56a_isrForCrosstalkSources_855450_17.stderr parsl.dataflow.dflow DEBUG: Task 4942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4942 has memoization hash 79a1fe6e0ceae2ac30968eb40ad535c0 parsl.dataflow.memoization INFO: Task 4942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57edb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4942 try 0 launched on executor multi with executor id 4943 parsl.dataflow.dflow INFO: Standard output for task 4942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ac6928fe-2708-43c6-a90c-bde55426dfa1_isrForCrosstalkSources_855399_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ac6928fe-2708-43c6-a90c-bde55426dfa1_isrForCrosstalkSources_855399_24.stderr parsl.dataflow.dflow DEBUG: Task 4943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4943 has memoization hash 78ec3c8eff7e50e5d39f4eb88d70785c parsl.dataflow.memoization INFO: Task 4943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4943 try 0 launched on executor multi with executor id 4944 parsl.dataflow.dflow INFO: Standard output for task 4943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/33a047ff-dbff-4dc9-8048-a47535aa0fad_isrForCrosstalkSources_855409_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/33a047ff-dbff-4dc9-8048-a47535aa0fad_isrForCrosstalkSources_855409_30.stderr parsl.dataflow.dflow DEBUG: Task 4944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4944 has memoization hash c8040b158119134df9691f9998da48e4 parsl.dataflow.memoization INFO: Task 4944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4944 try 0 launched on executor multi with executor id 4945 parsl.dataflow.dflow INFO: Standard output for task 4944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b17f5560-fe2e-4474-b170-7dc2fdc1da3d_isrForCrosstalkSources_855438_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b17f5560-fe2e-4474-b170-7dc2fdc1da3d_isrForCrosstalkSources_855438_23.stderr parsl.dataflow.dflow DEBUG: Task 4945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4945 has memoization hash d9d4cd0f1e79e7fc5204840502f45536 parsl.dataflow.memoization INFO: Task 4945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4945 try 0 launched on executor multi with executor id 4946 parsl.dataflow.dflow INFO: Standard output for task 4945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/60dba67e-46a4-431c-8173-c295fa18651b_isrForCrosstalkSources_855423_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/60dba67e-46a4-431c-8173-c295fa18651b_isrForCrosstalkSources_855423_3.stderr parsl.dataflow.dflow DEBUG: Task 4946 will be sent to executor multi 2024-03-09 14:51:58 proc_lsst.multi:170 [INFO] [multi] job_id 16964208 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4946 has memoization hash e4cb362881ce3ad81c910d2da54b353d parsl.dataflow.memoization INFO: Task 4946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] job_id 16964208 2024-03-09 14:51:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-09 14:51:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964208 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964208 parsl.executors.status_handling DEBUG: Launched block 182 on executor multi with job ID 16964208 parsl.executors.status_handling INFO: Allocated block ID 183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-183 2024-03-09 14:51:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4946 try 0 launched on executor multi with executor id 4947 parsl.dataflow.dflow INFO: Standard output for task 4946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/50710850-64b7-4a36-b7b6-a98eac832c56_isrForCrosstalkSources_855435_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/50710850-64b7-4a36-b7b6-a98eac832c56_isrForCrosstalkSources_855435_27.stderr parsl.dataflow.dflow DEBUG: Task 4947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4947 has memoization hash 972a139eb64f8ce1c2b1df425846fb4f parsl.dataflow.memoization INFO: Task 4947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4947 try 0 launched on executor multi with executor id 4948 parsl.dataflow.dflow INFO: Standard output for task 4947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fc4473d2-a88d-4d81-a0a1-3d0801d811d6_isrForCrosstalkSources_855454_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fc4473d2-a88d-4d81-a0a1-3d0801d811d6_isrForCrosstalkSources_855454_6.stderr parsl.dataflow.dflow DEBUG: Task 4948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4948 has memoization hash 7f133cb89e550a7d0a3574cedb2e3bf4 parsl.dataflow.memoization INFO: Task 4948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4948 try 0 launched on executor multi with executor id 4949 parsl.dataflow.dflow INFO: Standard output for task 4948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f18f1e4b-6476-4cd9-b114-0d8710731725_isrForCrosstalkSources_855398_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f18f1e4b-6476-4cd9-b114-0d8710731725_isrForCrosstalkSources_855398_38.stderr parsl.dataflow.dflow DEBUG: Task 4949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4949 has memoization hash 0970410b25688da3ce6709982680fae1 parsl.dataflow.memoization INFO: Task 4949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97fa6d6547a67f1a95cdd9b5d055d283 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e0d8b53-337e-4cea-84f6-df967b63a85e_isrForCrosstalkSources_855451_9.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e0d8b53-337e-4cea-84f6-df967b63a85e_isrForCrosstalkSources_855451_9.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58631efc7cb73f85c9e436f9221172f with result from task 709 parsl.dataflow.dflow INFO: Parsl task 4949 try 0 launched on executor multi with executor id 4950 parsl.dataflow.dflow INFO: Standard output for task 4949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/26575103-6d77-4cfc-9529-6ea147681642_isrForCrosstalkSources_855389_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/26575103-6d77-4cfc-9529-6ea147681642_isrForCrosstalkSources_855389_42.stderr parsl.dataflow.dflow DEBUG: Task 4950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4950 has memoization hash fd5cf5996606dc91c42e4727271f6237 parsl.dataflow.memoization INFO: Task 4950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b2ca3646-64f1-4925-9a8b-529de070c8a1_isrForCrosstalkSources_855422_4.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b2ca3646-64f1-4925-9a8b-529de070c8a1_isrForCrosstalkSources_855422_4.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb0be581be3cf2145bb4c9b95c59d03 with result from task 452 parsl.dataflow.dflow INFO: Parsl task 4950 try 0 launched on executor multi with executor id 4951 parsl.dataflow.dflow INFO: Standard output for task 4950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/aa06734c-098c-4f55-bd0b-0a732079035b_isrForCrosstalkSources_855413_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/aa06734c-098c-4f55-bd0b-0a732079035b_isrForCrosstalkSources_855413_51.stderr parsl.dataflow.dflow DEBUG: Task 4951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4951 has memoization hash 45178c494abe8e4c9d0bd23fd5c654fc parsl.dataflow.memoization INFO: Task 4951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4951 try 0 launched on executor multi with executor id 4952 parsl.dataflow.dflow INFO: Standard output for task 4951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9431a826-5879-46f7-b3a9-862303e14652_isrForCrosstalkSources_855383_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9431a826-5879-46f7-b3a9-862303e14652_isrForCrosstalkSources_855383_5.stderr parsl.dataflow.dflow DEBUG: Task 4952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4952 has memoization hash 689cd89a2bb49e766e75c6f7a5c2439e parsl.dataflow.memoization INFO: Task 4952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/af0ebe5d-b6cf-4c2b-a394-d98248e2a9ad_isrForCrosstalkSources_855405_15.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/af0ebe5d-b6cf-4c2b-a394-d98248e2a9ad_isrForCrosstalkSources_855405_15.stderr parsl.dataflow.dflow INFO: Parsl task 4952 try 0 launched on executor multi with executor id 4953 parsl.dataflow.dflow INFO: Standard output for task 4952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5edeae24-5e64-4a2e-b3a8-7869b74c2cb1_isrForCrosstalkSources_855436_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5edeae24-5e64-4a2e-b3a8-7869b74c2cb1_isrForCrosstalkSources_855436_25.stderr parsl.dataflow.dflow DEBUG: Task 4953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4953 has memoization hash 67f143adbabc2aa1afa171709194841b parsl.dataflow.memoization INFO: Task 4953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4953 try 0 launched on executor multi with executor id 4954 parsl.dataflow.dflow INFO: Standard output for task 4953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c0d4430-316c-4a1f-be9e-66857a6dad38_isrForCrosstalkSources_855458_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c0d4430-316c-4a1f-be9e-66857a6dad38_isrForCrosstalkSources_855458_19.stderr parsl.dataflow.dflow DEBUG: Task 4954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4954 has memoization hash 6ba8c5c43b0a19c8c03a28cc2ff43c04 parsl.dataflow.memoization INFO: Task 4954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4954 try 0 launched on executor multi with executor id 4955 parsl.dataflow.dflow INFO: Standard output for task 4954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/fad810c0-d6fe-485d-b690-0bd42c243b19_isrForCrosstalkSources_855403_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/fad810c0-d6fe-485d-b690-0bd42c243b19_isrForCrosstalkSources_855403_30.stderr parsl.dataflow.dflow DEBUG: Task 4955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4955 has memoization hash ee600c8a8397e4fdb13552003fa17cc9 parsl.dataflow.memoization INFO: Task 4955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4955 try 0 launched on executor multi with executor id 4956 parsl.dataflow.dflow INFO: Standard output for task 4955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/076f1b5a-971b-40c0-98eb-6cf8dc73783f_isrForCrosstalkSources_855416_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/076f1b5a-971b-40c0-98eb-6cf8dc73783f_isrForCrosstalkSources_855416_38.stderr parsl.dataflow.dflow DEBUG: Task 4956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4956 has memoization hash 2f98b31e0409ff42e92de80d555aba38 parsl.dataflow.memoization INFO: Task 4956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4956 try 0 launched on executor multi with executor id 4957 parsl.dataflow.dflow INFO: Standard output for task 4956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d3ffd278-a791-4ca8-97d4-dd3771b818d2_isrForCrosstalkSources_855381_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d3ffd278-a791-4ca8-97d4-dd3771b818d2_isrForCrosstalkSources_855381_52.stderr parsl.dataflow.dflow DEBUG: Task 4957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4957 has memoization hash 97d028f3c632a306a5aea55164266b26 parsl.dataflow.memoization INFO: Task 4957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4957 try 0 launched on executor multi with executor id 4958 parsl.dataflow.dflow INFO: Standard output for task 4957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/545affee-9a8e-46d4-9dfc-42bb115bc79a_isrForCrosstalkSources_855390_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/545affee-9a8e-46d4-9dfc-42bb115bc79a_isrForCrosstalkSources_855390_13.stderr parsl.dataflow.dflow DEBUG: Task 4958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4958 has memoization hash 69296ba7f859f5f58018f855421da7b6 parsl.dataflow.memoization INFO: Task 4958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4958 try 0 launched on executor multi with executor id 4959 parsl.dataflow.dflow INFO: Standard output for task 4958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f155e6ed-9254-418c-900c-675c684f71b7_isrForCrosstalkSources_855425_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f155e6ed-9254-418c-900c-675c684f71b7_isrForCrosstalkSources_855425_11.stderr parsl.dataflow.dflow DEBUG: Task 4959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4959 has memoization hash 0c1eaf98afa17a167505906429b9cea4 parsl.dataflow.memoization INFO: Task 4959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4959 try 0 launched on executor multi with executor id 4960 parsl.dataflow.dflow INFO: Standard output for task 4959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/240d2022-503d-4c0c-91d4-74ed70879e00_isrForCrosstalkSources_855451_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/240d2022-503d-4c0c-91d4-74ed70879e00_isrForCrosstalkSources_855451_16.stderr parsl.dataflow.dflow DEBUG: Task 4960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4960 has memoization hash a7cf53e9fc3c0d58ee5cc5cb998a4acd parsl.dataflow.memoization INFO: Task 4960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4960 try 0 launched on executor multi with executor id 4961 parsl.dataflow.dflow INFO: Standard output for task 4960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/6f827389-0af5-4b46-909c-b4a40ca3a424_isrForCrosstalkSources_855437_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/6f827389-0af5-4b46-909c-b4a40ca3a424_isrForCrosstalkSources_855437_46.stderr parsl.dataflow.dflow DEBUG: Task 4961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4961 has memoization hash 78eb0196e3b919c98815355cd1b95838 parsl.dataflow.memoization INFO: Task 4961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4961 try 0 launched on executor multi with executor id 4962 parsl.dataflow.dflow INFO: Standard output for task 4961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/42324106-00bf-4466-9bdc-dfb6ea20db32_isrForCrosstalkSources_855445_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/42324106-00bf-4466-9bdc-dfb6ea20db32_isrForCrosstalkSources_855445_21.stderr parsl.dataflow.dflow DEBUG: Task 4962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4962 has memoization hash 12b58f645f74bb5b65b16f4dc2e775a4 parsl.dataflow.memoization INFO: Task 4962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4962 try 0 launched on executor multi with executor id 4963 parsl.dataflow.dflow INFO: Standard output for task 4962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/1419659e-b5c4-49b3-9927-123923e8b0db_isrForCrosstalkSources_855440_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/1419659e-b5c4-49b3-9927-123923e8b0db_isrForCrosstalkSources_855440_12.stderr parsl.dataflow.dflow DEBUG: Task 4963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4963 has memoization hash 8dbc76fe6be7c0df61fa3969ccb9d524 parsl.dataflow.memoization INFO: Task 4963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4963 try 0 launched on executor multi with executor id 4964 parsl.dataflow.dflow INFO: Standard output for task 4963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/52e7373c-2159-4471-9474-1863d9d0dae2_isrForCrosstalkSources_855436_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/52e7373c-2159-4471-9474-1863d9d0dae2_isrForCrosstalkSources_855436_10.stderr parsl.dataflow.dflow DEBUG: Task 4964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4964 has memoization hash 267f9ad37ec0793ab061ab5878ab25ff parsl.dataflow.memoization INFO: Task 4964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4964 try 0 launched on executor multi with executor id 4965 parsl.dataflow.dflow INFO: Standard output for task 4964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1e09d5e2-7996-4e9e-9878-42fd0fce099b_isrForCrosstalkSources_855385_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1e09d5e2-7996-4e9e-9878-42fd0fce099b_isrForCrosstalkSources_855385_12.stderr parsl.dataflow.dflow DEBUG: Task 4965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4965 has memoization hash e105a1cfde662bed66ddb57a41cdb140 parsl.dataflow.memoization INFO: Task 4965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4965 try 0 launched on executor multi with executor id 4966 parsl.dataflow.dflow INFO: Standard output for task 4965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/38820c9e-6adb-4217-bfb1-0dad5df2edd2_isrForCrosstalkSources_855426_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/38820c9e-6adb-4217-bfb1-0dad5df2edd2_isrForCrosstalkSources_855426_10.stderr parsl.dataflow.dflow DEBUG: Task 4966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4966 has memoization hash edbdf924a0f06be71e4a5c389c9f5110 parsl.dataflow.memoization INFO: Task 4966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4966 try 0 launched on executor multi with executor id 4967 parsl.dataflow.dflow INFO: Standard output for task 4966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a1c51628-922f-4927-adad-3b0161f5353a_isrForCrosstalkSources_855435_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a1c51628-922f-4927-adad-3b0161f5353a_isrForCrosstalkSources_855435_61.stderr parsl.dataflow.dflow DEBUG: Task 4967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4967 has memoization hash 725baa715c08bd14c42cbe3963cf83ca parsl.dataflow.memoization INFO: Task 4967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac4aede68f2ad8d5c66a2f7bbcb2f279 with result from task 456 parsl.dataflow.dflow INFO: Parsl task 4967 try 0 launched on executor multi with executor id 4968 parsl.dataflow.dflow INFO: Standard output for task 4967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b3682ac3-470a-45d4-949f-21aa6d84cfd0_isrForCrosstalkSources_855448_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b3682ac3-470a-45d4-949f-21aa6d84cfd0_isrForCrosstalkSources_855448_10.stderr parsl.dataflow.dflow DEBUG: Task 4968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4968 has memoization hash 1616478c7a0aec9a287c029910bb1a2a parsl.dataflow.memoization INFO: Task 4968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4968 try 0 launched on executor multi with executor id 4969 parsl.dataflow.dflow INFO: Standard output for task 4968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b791de74-c527-445f-9413-668c30c7db85_isrForCrosstalkSources_855438_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b791de74-c527-445f-9413-668c30c7db85_isrForCrosstalkSources_855438_32.stderr parsl.dataflow.dflow DEBUG: Task 4969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4969 has memoization hash 2888ff1e287824f1dc7ee4781c9ef9cd parsl.dataflow.memoization INFO: Task 4969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/de115f10-d816-4469-a28f-34d9125ae7c8_isrForCrosstalkSources_855395_35.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/de115f10-d816-4469-a28f-34d9125ae7c8_isrForCrosstalkSources_855395_35.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0caf6b6623f699a2d6af43d02496fd8 with result from task 455 parsl.dataflow.dflow INFO: Parsl task 4969 try 0 launched on executor multi with executor id 4970 parsl.dataflow.dflow INFO: Standard output for task 4969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5d9f9848-060e-4325-b5db-65e656616570_isrForCrosstalkSources_855405_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5d9f9848-060e-4325-b5db-65e656616570_isrForCrosstalkSources_855405_54.stderr parsl.dataflow.dflow DEBUG: Task 4970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4970 has memoization hash 9f9778797f710333aa76eae41f29e87f parsl.dataflow.memoization INFO: Task 4970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57efa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:58 proc_lsst.multi:170 [INFO] [multi] job_id 16964209 proc_lsst.multi INFO: [multi] job_id 16964209 2024-03-09 14:51:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-09 14:51:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964209 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964209 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b4301f3a-edc0-4115-a71c-98ea59950265_isrForCrosstalkSources_855402_10.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b4301f3a-edc0-4115-a71c-98ea59950265_isrForCrosstalkSources_855402_10.stderr parsl.executors.status_handling DEBUG: Launched block 183 on executor multi with job ID 16964209 parsl.executors.status_handling INFO: Allocated block ID 184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-184 2024-03-09 14:51:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-09 14:51:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 4970 try 0 launched on executor multi with executor id 4971 parsl.dataflow.dflow INFO: Standard output for task 4970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/6b29227e-5e36-4b4d-89ef-eecfb4aa2431_isrForCrosstalkSources_855450_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/6b29227e-5e36-4b4d-89ef-eecfb4aa2431_isrForCrosstalkSources_855450_59.stderr parsl.dataflow.dflow DEBUG: Task 4971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4971 has memoization hash 30663c1915a00c2c3882bdd9552c6ecd parsl.dataflow.memoization INFO: Task 4971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4971 try 0 launched on executor multi with executor id 4972 parsl.dataflow.dflow INFO: Standard output for task 4971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7c2d7901-4495-488e-8957-e41394acba2a_isrForCrosstalkSources_855396_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7c2d7901-4495-488e-8957-e41394acba2a_isrForCrosstalkSources_855396_8.stderr parsl.dataflow.dflow DEBUG: Task 4972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4972 has memoization hash 18450aa8b3cd7e541f389e54a27e1102 parsl.dataflow.memoization INFO: Task 4972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4972 try 0 launched on executor multi with executor id 4973 parsl.dataflow.dflow INFO: Standard output for task 4972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8c6c2b89-aee6-4676-a777-423683b84ae7_isrForCrosstalkSources_855430_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8c6c2b89-aee6-4676-a777-423683b84ae7_isrForCrosstalkSources_855430_40.stderr parsl.dataflow.dflow DEBUG: Task 4973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4973 has memoization hash 3fed7f4ff9b9df7e8f23df8ed963726d parsl.dataflow.memoization INFO: Task 4973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57eefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4973 try 0 launched on executor multi with executor id 4974 parsl.dataflow.dflow INFO: Standard output for task 4973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eda1ee8d-432a-4a7e-b031-8887f3792a98_isrForCrosstalkSources_855446_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eda1ee8d-432a-4a7e-b031-8887f3792a98_isrForCrosstalkSources_855446_47.stderr parsl.dataflow.dflow DEBUG: Task 4974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4974 has memoization hash f4c3044173bb7cfb7e8d092b2bfca68b parsl.dataflow.memoization INFO: Task 4974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4974 try 0 launched on executor multi with executor id 4975 parsl.dataflow.dflow INFO: Standard output for task 4974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/546b7446-f54f-4715-8b5c-066332464609_isrForCrosstalkSources_855408_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/546b7446-f54f-4715-8b5c-066332464609_isrForCrosstalkSources_855408_27.stderr parsl.dataflow.dflow DEBUG: Task 4975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4975 has memoization hash 06dd1b8cade6c017c4b317332f1f8ae4 parsl.dataflow.memoization INFO: Task 4975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fdae345ca26e6568e2e97bc8bd7268d with result from task 579 parsl.dataflow.dflow INFO: Parsl task 4975 try 0 launched on executor multi with executor id 4976 parsl.dataflow.dflow INFO: Standard output for task 4975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fde5c120-f749-4bc0-855a-a309863ca09a_isrForCrosstalkSources_855438_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fde5c120-f749-4bc0-855a-a309863ca09a_isrForCrosstalkSources_855438_44.stderr parsl.dataflow.dflow DEBUG: Task 4976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4976 has memoization hash 068858275610c0db25c7484293638112 parsl.dataflow.memoization INFO: Task 4976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4976 try 0 launched on executor multi with executor id 4977 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Standard output for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stdout parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard error for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f899d0a0-70ae-4684-9b54-c164a323d9ed_isrForCrosstalkSources_855436_35.stdout parsl.dataflow.dflow DEBUG: Task 4977 will be sent to executor multi parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f899d0a0-70ae-4684-9b54-c164a323d9ed_isrForCrosstalkSources_855436_35.stderr parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.dflow INFO: Task 4977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.memoization DEBUG: Storing app cache entry e3438e3ae166726fa41b76f517f8119c with result from task 705 parsl.dataflow.dflow DEBUG: Task 4977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4977 has memoization hash 1206d1cdb7dc7c9881b8129305dd1b61 parsl.dataflow.memoization INFO: Task 4977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b3613d6a-b21d-4f70-97bd-ec5e8a19961e_isrForCrosstalkSources_855408_46.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b3613d6a-b21d-4f70-97bd-ec5e8a19961e_isrForCrosstalkSources_855408_46.stderr parsl.dataflow.dflow INFO: Parsl task 4977 try 0 launched on executor multi with executor id 4978 parsl.dataflow.dflow INFO: Standard output for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stderr parsl.dataflow.dflow DEBUG: Task 4978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4978 has memoization hash 55ee5a74f16cb86e0656e48296dddfd9 parsl.dataflow.memoization INFO: Task 4978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4978 try 0 launched on executor multi with executor id 4979 parsl.dataflow.dflow INFO: Standard output for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stderr parsl.dataflow.dflow DEBUG: Task 4979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4979 has memoization hash 99206de481ec81770d21f62194e39a86 parsl.dataflow.memoization INFO: Task 4979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4979 try 0 launched on executor multi with executor id 4980 parsl.dataflow.dflow INFO: Standard output for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stderr parsl.dataflow.dflow DEBUG: Task 4980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4980 has memoization hash 34a8fcf1ab4ca09eb886036155926839 parsl.dataflow.memoization INFO: Task 4980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4980 try 0 launched on executor multi with executor id 4981 parsl.dataflow.dflow INFO: Standard output for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stderr parsl.dataflow.dflow DEBUG: Task 4981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4981 has memoization hash a95860496d8dce78b509f989bbc14672 parsl.dataflow.memoization INFO: Task 4981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4981 try 0 launched on executor multi with executor id 4982 parsl.dataflow.dflow INFO: Standard output for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stderr parsl.dataflow.dflow DEBUG: Task 4982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4982 has memoization hash 7222e2a9c8d07a56cba94e02d6279b75 parsl.dataflow.memoization INFO: Task 4982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4982 try 0 launched on executor multi with executor id 4983 parsl.dataflow.dflow INFO: Standard output for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stderr parsl.dataflow.dflow DEBUG: Task 4983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4983 has memoization hash 37a5025841d0f98d14f916b221cc6e00 parsl.dataflow.memoization INFO: Task 4983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ca4696f0faeff499fb777c81c4df23 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a46259a7-6acb-497c-882b-62605aa1c439_isrForCrosstalkSources_855395_51.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a46259a7-6acb-497c-882b-62605aa1c439_isrForCrosstalkSources_855395_51.stderr parsl.dataflow.dflow INFO: Parsl task 4983 try 0 launched on executor multi with executor id 4984 parsl.dataflow.dflow INFO: Standard output for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stderr parsl.dataflow.dflow DEBUG: Task 4984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4984 has memoization hash 724de7214076ca09833b5721e5ac99e2 parsl.dataflow.memoization INFO: Task 4984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4984 try 0 launched on executor multi with executor id 4985 parsl.dataflow.dflow INFO: Standard output for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stderr parsl.dataflow.dflow DEBUG: Task 4985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4985 has memoization hash 212508430e8f427def6b995e5cddf899 parsl.dataflow.memoization INFO: Task 4985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4985 try 0 launched on executor multi with executor id 4986 parsl.dataflow.dflow INFO: Standard output for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stderr parsl.dataflow.dflow DEBUG: Task 4986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4986 has memoization hash ec219040399c7ddb65ebfd74da300d83 parsl.dataflow.memoization INFO: Task 4986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4986 try 0 launched on executor multi with executor id 4987 parsl.dataflow.dflow INFO: Standard output for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stderr parsl.dataflow.dflow DEBUG: Task 4987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4987 has memoization hash fdd034bfb3e9f4be7eb7587c5f69e5ad parsl.dataflow.memoization INFO: Task 4987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4987 try 0 launched on executor multi with executor id 4988 parsl.dataflow.dflow INFO: Standard output for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stderr parsl.dataflow.dflow DEBUG: Task 4988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4988 has memoization hash b6f3b62139cdc193fbb39ce774284375 parsl.dataflow.memoization INFO: Task 4988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4988 try 0 launched on executor multi with executor id 4989 parsl.dataflow.dflow INFO: Standard output for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stderr parsl.dataflow.dflow DEBUG: Task 4989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4989 has memoization hash 4ffba1b4102873cd5de6f809ab7098b0 parsl.dataflow.memoization INFO: Task 4989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4989 try 0 launched on executor multi with executor id 4990 parsl.dataflow.dflow INFO: Standard output for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stderr parsl.dataflow.dflow DEBUG: Task 4990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4990 has memoization hash 1d1d185ef6c991a59c7043eb4afb2631 parsl.dataflow.memoization INFO: Task 4990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4990 try 0 launched on executor multi with executor id 4991 parsl.dataflow.dflow INFO: Standard output for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stderr parsl.dataflow.dflow DEBUG: Task 4991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4991 has memoization hash 7c2052f7bc83cfa610552ed96d0085f6 parsl.dataflow.memoization INFO: Task 4991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae8f65b5e367f68b403a2f5b537102dd with result from task 703 parsl.dataflow.dflow INFO: Parsl task 4991 try 0 launched on executor multi with executor id 4992 parsl.dataflow.dflow INFO: Standard output for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stderr parsl.dataflow.dflow DEBUG: Task 4992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4992 has memoization hash 5dbece7a960a0ba2561c883360cc8987 parsl.dataflow.memoization INFO: Task 4992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4992 try 0 launched on executor multi with executor id 4993 parsl.dataflow.dflow INFO: Standard output for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stderr parsl.dataflow.dflow DEBUG: Task 4993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4993 has memoization hash 3789066774b1c2013e72659d669f384c parsl.dataflow.memoization INFO: Task 4993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/91897d18-ce8c-41d6-b97a-d2d706c202e5_isrForCrosstalkSources_855448_56.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/91897d18-ce8c-41d6-b97a-d2d706c202e5_isrForCrosstalkSources_855448_56.stderr parsl.dataflow.dflow INFO: Parsl task 4993 try 0 launched on executor multi with executor id 4994 parsl.dataflow.dflow INFO: Standard output for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stderr parsl.dataflow.dflow DEBUG: Task 4994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4994 has memoization hash e0e2cd6a35b99700dbc768fd792e27fc parsl.dataflow.memoization INFO: Task 4994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e68a3cf6321f072643eab7c01eb0472a with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0befe143-a041-45fa-b7ad-19e0ad7fe1bc_isrForCrosstalkSources_855451_15.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0befe143-a041-45fa-b7ad-19e0ad7fe1bc_isrForCrosstalkSources_855451_15.stderr parsl.dataflow.dflow INFO: Parsl task 4994 try 0 launched on executor multi with executor id 4995 parsl.dataflow.dflow INFO: Standard output for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stderr parsl.dataflow.dflow DEBUG: Task 4995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4995 has memoization hash ed67e44dc1337a7378c11504c33e6f60 parsl.dataflow.memoization INFO: Task 4995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4995 try 0 launched on executor multi with executor id 4996 parsl.dataflow.dflow INFO: Standard output for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stderr parsl.dataflow.dflow DEBUG: Task 4996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4996 has memoization hash d0ca23344bafe5fdb23732b17ec718de parsl.dataflow.memoization INFO: Task 4996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-09 14:51:59 proc_lsst.multi:170 [INFO] [multi] job_id 16964210 proc_lsst.multi INFO: [multi] job_id 16964210 2024-03-09 14:51:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-09 14:51:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964210 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964210 parsl.executors.status_handling DEBUG: Launched block 184 on executor multi with job ID 16964210 parsl.executors.status_handling INFO: Allocated block ID 185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-185 2024-03-09 14:51:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 4996 try 0 launched on executor multi with executor id 4997 parsl.dataflow.dflow INFO: Standard output for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stderr parsl.dataflow.dflow DEBUG: Task 4997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4997 has memoization hash 5106fa85b08887127e2b23337a274286 parsl.dataflow.memoization INFO: Task 4997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4997 try 0 launched on executor multi with executor id 4998 parsl.dataflow.dflow INFO: Standard output for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stderr parsl.dataflow.dflow DEBUG: Task 4998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4998 has memoization hash c42a7c2e2934760b901ba9520f6b6a04 parsl.dataflow.memoization INFO: Task 4998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4998 try 0 launched on executor multi with executor id 4999 parsl.dataflow.dflow INFO: Standard output for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stderr parsl.dataflow.dflow DEBUG: Task 4999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4999 has memoization hash b7efe11eeee3acb69723f5dc781f4556 parsl.dataflow.memoization INFO: Task 4999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91736ac89fcea696d36adb3170f45d3a with result from task 449 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/63ee067b-971c-4e7e-856a-cd55b92c1716_isrForCrosstalkSources_855427_5.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/63ee067b-971c-4e7e-856a-cd55b92c1716_isrForCrosstalkSources_855427_5.stderr parsl.dataflow.dflow INFO: Parsl task 4999 try 0 launched on executor multi with executor id 5000 parsl.dataflow.dflow INFO: Standard output for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stderr parsl.dataflow.dflow DEBUG: Task 5000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5000 has memoization hash 267c735a3cfaf9cd7b912c5c3c6eda5b parsl.dataflow.memoization INFO: Task 5000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5000 try 0 launched on executor multi with executor id 5001 parsl.dataflow.dflow INFO: Standard output for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stdout parsl.dataflow.dflow INFO: Standard error for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stderr parsl.dataflow.dflow DEBUG: Task 5001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5001 has memoization hash c9be6f4ded3e4d1f9edd0653798c2302 parsl.dataflow.memoization INFO: Task 5001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5001 try 0 launched on executor multi with executor id 5002 parsl.dataflow.dflow INFO: Standard output for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stdout parsl.dataflow.dflow INFO: Standard error for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stderr parsl.dataflow.dflow DEBUG: Task 5002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5002 has memoization hash c02d454f07ede8b12d6877d62ce14217 parsl.dataflow.memoization INFO: Task 5002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5002 try 0 launched on executor multi with executor id 5003 parsl.dataflow.dflow INFO: Standard output for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stdout parsl.dataflow.dflow INFO: Standard error for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stderr parsl.dataflow.dflow DEBUG: Task 5003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5003 has memoization hash 6fb44d566e8defe88d9c42e2e4054c2c parsl.dataflow.memoization INFO: Task 5003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5003 try 0 launched on executor multi with executor id 5004 parsl.dataflow.dflow INFO: Standard output for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stderr parsl.dataflow.dflow DEBUG: Task 5004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5004 has memoization hash 0781bc40ef3a12e7e94731748989b821 parsl.dataflow.memoization INFO: Task 5004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5004 try 0 launched on executor multi with executor id 5005 parsl.dataflow.dflow INFO: Standard output for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stderr parsl.dataflow.dflow DEBUG: Task 5005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5005 has memoization hash 38cbf8cd714e220118127c5f01fd21f5 parsl.dataflow.memoization INFO: Task 5005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5005 try 0 launched on executor multi with executor id 5006 parsl.dataflow.dflow INFO: Standard output for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stderr parsl.dataflow.dflow DEBUG: Task 5006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5006 has memoization hash 5f2cf339a0546f49d7d1a7411056aa07 parsl.dataflow.memoization INFO: Task 5006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5006 try 0 launched on executor multi with executor id 5007 parsl.dataflow.dflow INFO: Standard output for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stdout parsl.dataflow.dflow INFO: Standard error for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stderr parsl.dataflow.dflow DEBUG: Task 5007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5007 has memoization hash a22b1d9743bbbe69dad0519416559bcf parsl.dataflow.memoization INFO: Task 5007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5007 try 0 launched on executor multi with executor id 5008 parsl.dataflow.dflow INFO: Standard output for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stderr parsl.dataflow.dflow DEBUG: Task 5008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5008 has memoization hash 39f26669241af09695a25c42055b155a parsl.dataflow.memoization INFO: Task 5008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5008 try 0 launched on executor multi with executor id 5009 parsl.dataflow.dflow INFO: Standard output for task 5008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f42dd90c-5b14-4f95-9a63-87d4911482e5_isrForCrosstalkSources_855423_51.stdout parsl.dataflow.dflow INFO: Standard error for task 5008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f42dd90c-5b14-4f95-9a63-87d4911482e5_isrForCrosstalkSources_855423_51.stderr parsl.dataflow.dflow DEBUG: Task 5009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5009 has memoization hash 9ffbde344c0e30bed8e53a01bf859339 parsl.dataflow.memoization INFO: Task 5009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5009 try 0 launched on executor multi with executor id 5010 parsl.dataflow.dflow INFO: Standard output for task 5009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3f37cb57-6601-4f4d-b03d-0cde40f30557_isrForCrosstalkSources_855411_49.stdout parsl.dataflow.dflow INFO: Standard error for task 5009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3f37cb57-6601-4f4d-b03d-0cde40f30557_isrForCrosstalkSources_855411_49.stderr parsl.dataflow.dflow DEBUG: Task 5010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5010 has memoization hash 680d5f19f29947e68a6372db64eb421b parsl.dataflow.memoization INFO: Task 5010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5010 try 0 launched on executor multi with executor id 5011 parsl.dataflow.dflow INFO: Standard output for task 5010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/969d9e5b-d948-4d31-806c-4178d3991628_isrForCrosstalkSources_855438_22.stdout parsl.dataflow.dflow INFO: Standard error for task 5010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/969d9e5b-d948-4d31-806c-4178d3991628_isrForCrosstalkSources_855438_22.stderr parsl.dataflow.dflow DEBUG: Task 5011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5011 has memoization hash 4023f2cd117b8b85ee3a45c069d2ddfe parsl.dataflow.memoization INFO: Task 5011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5011 try 0 launched on executor multi with executor id 5012 parsl.dataflow.dflow INFO: Standard output for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stderr parsl.dataflow.dflow DEBUG: Task 5012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5012 has memoization hash d0ad46b40fa77e0b506efeadeb768904 parsl.dataflow.memoization INFO: Task 5012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5012 try 0 launched on executor multi with executor id 5013 parsl.dataflow.dflow INFO: Standard output for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stdout parsl.dataflow.dflow INFO: Standard error for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stderr parsl.dataflow.dflow DEBUG: Task 5013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5013 has memoization hash ba985d823064dcd2db48d9d0a8776ee2 parsl.dataflow.memoization INFO: Task 5013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5013 try 0 launched on executor multi with executor id 5014 parsl.dataflow.dflow INFO: Standard output for task 5013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e05ee-2ce8-43ed-a1d4-82e4819e604d_isrForCrosstalkSources_855448_46.stdout parsl.dataflow.dflow INFO: Standard error for task 5013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e05ee-2ce8-43ed-a1d4-82e4819e604d_isrForCrosstalkSources_855448_46.stderr parsl.dataflow.dflow DEBUG: Task 5014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5014 has memoization hash e601272d001a24d344ca56f8dbc31185 parsl.dataflow.memoization INFO: Task 5014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5014 try 0 launched on executor multi with executor id 5015 parsl.dataflow.dflow INFO: Standard output for task 5014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fe7a871c-c7f1-484f-8054-0e21d371bd28_isrForCrosstalkSources_855413_18.stdout parsl.dataflow.dflow INFO: Standard error for task 5014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fe7a871c-c7f1-484f-8054-0e21d371bd28_isrForCrosstalkSources_855413_18.stderr parsl.dataflow.dflow DEBUG: Task 5015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5015 has memoization hash 4d019931c12f1bb700ce52d80293e7ed parsl.dataflow.memoization INFO: Task 5015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5015 try 0 launched on executor multi with executor id 5016 parsl.dataflow.dflow INFO: Standard output for task 5015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/187a97b8-c230-4c9e-a35c-190954859aee_isrForCrosstalkSources_855452_10.stdout parsl.dataflow.dflow INFO: Standard error for task 5015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/187a97b8-c230-4c9e-a35c-190954859aee_isrForCrosstalkSources_855452_10.stderr parsl.dataflow.dflow DEBUG: Task 5016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5016 has memoization hash 7e53dec194564a6de49e0f32250b7b79 parsl.dataflow.memoization INFO: Task 5016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5016 try 0 launched on executor multi with executor id 5017 parsl.dataflow.dflow INFO: Standard output for task 5016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8c15864c-e207-4e47-85cc-625e046e8d90_isrForCrosstalkSources_855426_11.stdout parsl.dataflow.dflow INFO: Standard error for task 5016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8c15864c-e207-4e47-85cc-625e046e8d90_isrForCrosstalkSources_855426_11.stderr parsl.dataflow.dflow DEBUG: Task 5017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5017 has memoization hash de2a63c4f0047168d5cdf57f96f3c2ef parsl.dataflow.memoization INFO: Task 5017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5017 try 0 launched on executor multi with executor id 5018 parsl.dataflow.dflow INFO: Standard output for task 5017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/aba73dfa-6f31-4c02-a532-001e42f69fb0_isrForCrosstalkSources_855422_30.stdout parsl.dataflow.dflow INFO: Standard error for task 5017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/aba73dfa-6f31-4c02-a532-001e42f69fb0_isrForCrosstalkSources_855422_30.stderr parsl.dataflow.dflow DEBUG: Task 5018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5018 has memoization hash fc6691b4abf432ea6e9e37df7fb1b982 parsl.dataflow.memoization INFO: Task 5018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5018 try 0 launched on executor multi with executor id 5019 parsl.dataflow.dflow INFO: Standard output for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stdout parsl.dataflow.dflow INFO: Standard error for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stderr parsl.dataflow.dflow DEBUG: Task 5019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5019 has memoization hash 92b1c156d619f4d9b861bdffefd0d72f parsl.dataflow.memoization INFO: Task 5019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5019 try 0 launched on executor multi with executor id 5020 parsl.dataflow.dflow INFO: Standard output for task 5019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/779d8c58-9f0b-4d48-8969-e94fda7d1595_isrForCrosstalkSources_855439_25.stdout parsl.dataflow.dflow INFO: Standard error for task 5019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/779d8c58-9f0b-4d48-8969-e94fda7d1595_isrForCrosstalkSources_855439_25.stderr parsl.dataflow.dflow DEBUG: Task 5020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5020 has memoization hash e3de93384695852f13a347aa2b3d027b parsl.dataflow.memoization INFO: Task 5020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec809708698468ab97fd6b4d69bc5ce5 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0ccb1095-8e15-4b3d-933a-2a799b5ab45f_isrForCrosstalkSources_855406_42.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0ccb1095-8e15-4b3d-933a-2a799b5ab45f_isrForCrosstalkSources_855406_42.stderr parsl.dataflow.dflow INFO: Parsl task 5020 try 0 launched on executor multi with executor id 5021 parsl.dataflow.dflow INFO: Standard output for task 5020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6ff1ce12-a25b-4a26-bc54-1bdbf70e2eaa_isrForCrosstalkSources_855458_7.stdout parsl.dataflow.dflow INFO: Standard error for task 5020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6ff1ce12-a25b-4a26-bc54-1bdbf70e2eaa_isrForCrosstalkSources_855458_7.stderr parsl.dataflow.dflow DEBUG: Task 5021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5021 has memoization hash 1c1ee446b6e27cab1508d3948e5328fc parsl.dataflow.memoization INFO: Task 5021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5021 try 0 launched on executor multi with executor id 5022 parsl.dataflow.dflow INFO: Standard output for task 5021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/4e3286bd-abdb-4910-8013-9aa09185b766_isrForCrosstalkSources_855424_26.stdout parsl.dataflow.dflow INFO: Standard error for task 5021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/4e3286bd-abdb-4910-8013-9aa09185b766_isrForCrosstalkSources_855424_26.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b512f4152e943ab82d7549267066669 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6ecc9803-8814-48c3-a5be-e142edfb4809_isrForCrosstalkSources_855440_60.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6ecc9803-8814-48c3-a5be-e142edfb4809_isrForCrosstalkSources_855440_60.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 202149f4a333021b68c2161820a2c965 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e61d012f-918a-49fb-8636-308eaf0c3dc1_isrForCrosstalkSources_855451_60.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e61d012f-918a-49fb-8636-308eaf0c3dc1_isrForCrosstalkSources_855451_60.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae4a350728ea8e90e77b31f1344ff05 with result from task 615 2024-03-09 14:51:59 proc_lsst.multi:170 [INFO] [multi] job_id 16964211 proc_lsst.multi INFO: [multi] job_id 16964211 2024-03-09 14:51:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-09 14:51:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964211 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964211 parsl.executors.status_handling DEBUG: Launched block 185 on executor multi with job ID 16964211 parsl.executors.status_handling INFO: Allocated block ID 186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-186 2024-03-09 14:51:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:51:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:51:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-09 14:51:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:51:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2f48c4c9-b9c5-4de8-9434-abdae44a1ff4_isrForCrosstalkSources_855436_4.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2f48c4c9-b9c5-4de8-9434-abdae44a1ff4_isrForCrosstalkSources_855436_4.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ae39f20fa584598a74d84e6210508d5 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36b41f05-361a-4893-a221-d94e30d8cce9_isrForCrosstalkSources_855390_56.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36b41f05-361a-4893-a221-d94e30d8cce9_isrForCrosstalkSources_855390_56.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed692d4b597d96666ba894e8a638a475 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46698106-192f-4b40-b775-325c6c1eb79d_isrForCrosstalkSources_855448_50.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46698106-192f-4b40-b775-325c6c1eb79d_isrForCrosstalkSources_855448_50.stderr 2024-03-09 14:52:00 proc_lsst.multi:170 [INFO] [multi] job_id 16964212 parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec1901ed9f35cd6eb561a3cb8342854 with result from task 702 proc_lsst.multi INFO: [multi] job_id 16964212 2024-03-09 14:52:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-09 14:52:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964212 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964212 parsl.executors.status_handling DEBUG: Launched block 186 on executor multi with job ID 16964212 parsl.executors.status_handling INFO: Allocated block ID 187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-187 2024-03-09 14:52:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 2024-03-09 14:52:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-09 14:52:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/18ae984d-cd7b-4be6-935d-7031906d327e_isrForCrosstalkSources_855383_18.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/18ae984d-cd7b-4be6-935d-7031906d327e_isrForCrosstalkSources_855383_18.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6851995ac95ad0e0799aecb73090b85 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/db983aeb-829a-46a8-a45f-8fc2ac0458c7_isrForCrosstalkSources_855435_52.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/db983aeb-829a-46a8-a45f-8fc2ac0458c7_isrForCrosstalkSources_855435_52.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d62555587939b3e68cd2e04c085997ce with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bf4bae2a-45c1-4f36-8648-a21878221511_isrForCrosstalkSources_855427_20.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bf4bae2a-45c1-4f36-8648-a21878221511_isrForCrosstalkSources_855427_20.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2719f610407cf71a3cb0d12305f09d3b with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac040bcd-0d8c-48f8-9c7f-e59856acc977_isrForCrosstalkSources_855401_36.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac040bcd-0d8c-48f8-9c7f-e59856acc977_isrForCrosstalkSources_855401_36.stderr 2024-03-09 14:52:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964213 proc_lsst.multi INFO: [multi] job_id 16964213 2024-03-09 14:52:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-09 14:52:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964213 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964213 parsl.executors.status_handling DEBUG: Launched block 187 on executor multi with job ID 16964213 parsl.executors.status_handling INFO: Allocated block ID 188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-188 2024-03-09 14:52:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2507ba8f8b85bbebc1a6f08d2869237c with result from task 696 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/649397c9-0264-4c7d-b228-20dd8af92d74_isrForCrosstalkSources_855398_31.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/649397c9-0264-4c7d-b228-20dd8af92d74_isrForCrosstalkSources_855398_31.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecec74d700fda83fa46159b954704a93 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf3bc02-836c-4ef4-9303-c7f6360f99d5_isrForCrosstalkSources_855388_53.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf3bc02-836c-4ef4-9303-c7f6360f99d5_isrForCrosstalkSources_855388_53.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73ccd0482e1f2e3fd719a9458f739fb with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f247ee06-a7db-4525-ba14-745959ca61d1_isrForCrosstalkSources_855397_5.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f247ee06-a7db-4525-ba14-745959ca61d1_isrForCrosstalkSources_855397_5.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cab36b455e3e24095167ad61627893cb with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3e5f98d4-8b12-4531-8248-4dda92aae82c_isrForCrosstalkSources_855388_16.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3e5f98d4-8b12-4531-8248-4dda92aae82c_isrForCrosstalkSources_855388_16.stderr 2024-03-09 14:52:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964214 proc_lsst.multi INFO: [multi] job_id 16964214 2024-03-09 14:52:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-09 14:52:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964214 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964214 parsl.executors.status_handling DEBUG: Launched block 188 on executor multi with job ID 16964214 parsl.executors.status_handling INFO: Allocated block ID 189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-189 2024-03-09 14:52:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-09 14:52:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d58749269eb9dd07e5e3600842f2960 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4e59b6c7-6bb9-4dba-8711-df4f736e20d6_isrForCrosstalkSources_855426_18.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4e59b6c7-6bb9-4dba-8711-df4f736e20d6_isrForCrosstalkSources_855426_18.stderr 2024-03-09 14:52:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964215 proc_lsst.multi INFO: [multi] job_id 16964215 2024-03-09 14:52:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-09 14:52:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964215 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964215 parsl.executors.status_handling DEBUG: Launched block 189 on executor multi with job ID 16964215 parsl.executors.status_handling INFO: Allocated block ID 190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-190 2024-03-09 14:52:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5540f1cb644e836dff8b4174892794da with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d24acbe8-b57f-4d5a-9710-99fdbf93a3ca_isrForCrosstalkSources_855409_3.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d24acbe8-b57f-4d5a-9710-99fdbf93a3ca_isrForCrosstalkSources_855409_3.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29327225753b152e9810ef105c0c3c5b with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0876f5d8-29e8-43c0-84c9-fcab28bbf342_isrForCrosstalkSources_855439_41.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0876f5d8-29e8-43c0-84c9-fcab28bbf342_isrForCrosstalkSources_855439_41.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67ab6adfa11fce616cefeb0cce9a90b8 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/29ed965a-81b5-485b-a601-9d91f80bcbc1_isrForCrosstalkSources_855441_3.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/29ed965a-81b5-485b-a601-9d91f80bcbc1_isrForCrosstalkSources_855441_3.stderr 2024-03-09 14:52:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964216 proc_lsst.multi INFO: [multi] job_id 16964216 2024-03-09 14:52:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-09 14:52:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964216 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964216 parsl.executors.status_handling DEBUG: Launched block 190 on executor multi with job ID 16964216 parsl.executors.status_handling INFO: Allocated block ID 191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-191 2024-03-09 14:52:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-09 14:52:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f470ce79f51443e60cdfef64808a161a with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d101d21a-d9fd-4d80-94dc-ba181c2423b4_isrForCrosstalkSources_855401_32.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d101d21a-d9fd-4d80-94dc-ba181c2423b4_isrForCrosstalkSources_855401_32.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eaaf5138e6f1ab6575740eabc3fcc80 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ef3d2cb6-403a-4231-9bb6-02dd14f87dc5_isrForCrosstalkSources_855399_51.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ef3d2cb6-403a-4231-9bb6-02dd14f87dc5_isrForCrosstalkSources_855399_51.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121494858abb101b6a76118873b87b13 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f7bb8cfb-d0ac-4445-82f3-c033e006ec43_isrForCrosstalkSources_855388_54.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f7bb8cfb-d0ac-4445-82f3-c033e006ec43_isrForCrosstalkSources_855388_54.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec5b179de778abebcf9a93b2e8837cab with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/252d58f5-678c-4107-8703-9d8c0ea407a2_isrForCrosstalkSources_855393_51.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/252d58f5-678c-4107-8703-9d8c0ea407a2_isrForCrosstalkSources_855393_51.stderr 2024-03-09 14:52:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964217 proc_lsst.multi INFO: [multi] job_id 16964217 2024-03-09 14:52:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-09 14:52:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964217 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964217 parsl.executors.status_handling DEBUG: Launched block 191 on executor multi with job ID 16964217 parsl.executors.status_handling INFO: Allocated block ID 192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-192 2024-03-09 14:52:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06f2928e7811c72a873767147f210d31 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6332d152-7fd9-4f01-8004-7ab6ac8a17b1_isrForCrosstalkSources_855407_52.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6332d152-7fd9-4f01-8004-7ab6ac8a17b1_isrForCrosstalkSources_855407_52.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8f0127a3b73351dc11a0c9f0d2c8c4a with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/75b6adbc-3de6-4682-be9a-f3058aa85d23_isrForCrosstalkSources_855448_7.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/75b6adbc-3de6-4682-be9a-f3058aa85d23_isrForCrosstalkSources_855448_7.stderr 2024-03-09 14:52:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964218 proc_lsst.multi INFO: [multi] job_id 16964218 2024-03-09 14:52:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-09 14:52:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964218 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964218 parsl.executors.status_handling DEBUG: Launched block 192 on executor multi with job ID 16964218 parsl.executors.status_handling INFO: Allocated block ID 193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-193 2024-03-09 14:52:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-09 14:52:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964220 proc_lsst.multi INFO: [multi] job_id 16964220 2024-03-09 14:52:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-09 14:52:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964220 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964220 parsl.executors.status_handling DEBUG: Launched block 193 on executor multi with job ID 16964220 parsl.executors.status_handling INFO: Allocated block ID 194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-194 2024-03-09 14:52:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51af4695def8dec9b87ba12c6c90425f with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a7ec152e-1f46-491c-acf0-d272fb5bc754_isrForCrosstalkSources_855420_33.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a7ec152e-1f46-491c-acf0-d272fb5bc754_isrForCrosstalkSources_855420_33.stderr 2024-03-09 14:52:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964221 proc_lsst.multi INFO: [multi] job_id 16964221 2024-03-09 14:52:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-09 14:52:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964221 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964221 parsl.executors.status_handling DEBUG: Launched block 194 on executor multi with job ID 16964221 parsl.executors.status_handling INFO: Allocated block ID 195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-195 2024-03-09 14:52:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-09 14:52:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cad52d0ee83afd11b7b39fc868227f24 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/666cb698-5146-479c-bf0d-72fa52f04e7b_isrForCrosstalkSources_855391_23.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/666cb698-5146-479c-bf0d-72fa52f04e7b_isrForCrosstalkSources_855391_23.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f051779f6fa49301a0f6b15f14d82c5b with result from task 463 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4dc7553c-c7e5-4c91-a122-d38a7a14a6d6_isrForCrosstalkSources_855452_13.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4dc7553c-c7e5-4c91-a122-d38a7a14a6d6_isrForCrosstalkSources_855452_13.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0d55a5158c68766274d7f014f04888b with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5857eefa-91b9-4a80-b879-d734b006375e_isrForCrosstalkSources_855433_37.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5857eefa-91b9-4a80-b879-d734b006375e_isrForCrosstalkSources_855433_37.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71684b9ebc0592c4884ec1b0475fec20 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/215eeadf-0426-4d31-b5b7-2a37e42f21d3_isrForCrosstalkSources_855458_37.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/215eeadf-0426-4d31-b5b7-2a37e42f21d3_isrForCrosstalkSources_855458_37.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68aedc568b3afe7c5ef8d1dde4dfa691 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/51c707e1-0074-4755-94f9-03e4983f5099_isrForCrosstalkSources_855419_50.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/51c707e1-0074-4755-94f9-03e4983f5099_isrForCrosstalkSources_855419_50.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcc6f8ab676028039a9043f793cb69f5 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/06bb76db-4e06-4c5c-b0b1-cd792f1128a1_isrForCrosstalkSources_855438_61.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/06bb76db-4e06-4c5c-b0b1-cd792f1128a1_isrForCrosstalkSources_855438_61.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab5c2a2075a90b97f94147766e28e93e with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d9d72fd1-0439-4bcd-9d98-d27d0bc2acbf_isrForCrosstalkSources_855384_1.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d9d72fd1-0439-4bcd-9d98-d27d0bc2acbf_isrForCrosstalkSources_855384_1.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fed76a8dfe266515ce356626fd972d2a with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64ea3858-1eb4-43e6-823c-3ab49a43ec32_isrForCrosstalkSources_855424_5.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64ea3858-1eb4-43e6-823c-3ab49a43ec32_isrForCrosstalkSources_855424_5.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45db70f9f26782e708ed4c0f0d546cfc with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e226c445-9702-41d3-8d69-9e911b1982d5_isrForCrosstalkSources_855411_37.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e226c445-9702-41d3-8d69-9e911b1982d5_isrForCrosstalkSources_855411_37.stderr 2024-03-09 14:52:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964222 proc_lsst.multi INFO: [multi] job_id 16964222 2024-03-09 14:52:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-09 14:52:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964222 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964222 parsl.executors.status_handling DEBUG: Launched block 195 on executor multi with job ID 16964222 parsl.executors.status_handling INFO: Allocated block ID 196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-196 2024-03-09 14:52:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dfd00c4d4872e50767625250a87d240 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/28d6aa5b-ad88-4f7e-b4fa-c2c34819972a_isrForCrosstalkSources_855424_4.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/28d6aa5b-ad88-4f7e-b4fa-c2c34819972a_isrForCrosstalkSources_855424_4.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3987e25239bcad74de544db555181f69 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ae1e9950-8237-4a2b-8e31-ddb1a00db78d_isrForCrosstalkSources_855395_37.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ae1e9950-8237-4a2b-8e31-ddb1a00db78d_isrForCrosstalkSources_855395_37.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 783bc7e7892fcb9dc9c227dbf8ca82a2 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3943638-5ea3-4aa4-8e1c-008b7474b3ab_isrForCrosstalkSources_855379_51.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3943638-5ea3-4aa4-8e1c-008b7474b3ab_isrForCrosstalkSources_855379_51.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20b2befd526cab64f6ae4b397407e6e8 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e7abd982-e230-4d71-a2e9-ced546f99d4c_isrForCrosstalkSources_855394_52.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e7abd982-e230-4d71-a2e9-ced546f99d4c_isrForCrosstalkSources_855394_52.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9606a946a0dcec590f31a341928d7f8 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/718b5130-436b-4873-bd34-bcde617e454c_isrForCrosstalkSources_855423_8.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/718b5130-436b-4873-bd34-bcde617e454c_isrForCrosstalkSources_855423_8.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8668ab71e686958ec57e76c9e33f8a8 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/af238054-654c-452d-824f-2599a615df9a_isrForCrosstalkSources_855388_3.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/af238054-654c-452d-824f-2599a615df9a_isrForCrosstalkSources_855388_3.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df514658f0853303aa0ab25ecc3b72a8 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/d46e3f0c-6e98-4198-bb24-063550de7cc9_isrForCrosstalkSources_855380_40.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/d46e3f0c-6e98-4198-bb24-063550de7cc9_isrForCrosstalkSources_855380_40.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d67f06805f32a399f79978e9b98e0453 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7ef09968-9bcd-4e4c-86eb-626deb8c175d_isrForCrosstalkSources_855392_19.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7ef09968-9bcd-4e4c-86eb-626deb8c175d_isrForCrosstalkSources_855392_19.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbda82a47c334a41722c65524270fde9 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/765a8962-0a9a-4d93-b0ed-d90a2c391d20_isrForCrosstalkSources_855400_23.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/765a8962-0a9a-4d93-b0ed-d90a2c391d20_isrForCrosstalkSources_855400_23.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13adbd5e67788e21ad8ae75abf73d512 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/760f2261-c0a4-4a7e-a3b3-62666fb8b5cd_isrForCrosstalkSources_855458_1.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/760f2261-c0a4-4a7e-a3b3-62666fb8b5cd_isrForCrosstalkSources_855458_1.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff0000674fcf8c686d5c395556f50d89 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5e6776f9-1619-4fdc-b91d-246ba3a20ac0_isrForCrosstalkSources_855381_7.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5e6776f9-1619-4fdc-b91d-246ba3a20ac0_isrForCrosstalkSources_855381_7.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6951cf74c15619c538301d6e1e8882bc with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/22a56394-798f-493a-9450-f3d1700abfe9_isrForCrosstalkSources_855401_12.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/22a56394-798f-493a-9450-f3d1700abfe9_isrForCrosstalkSources_855401_12.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 368358961223256aaba4eae45f25c964 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a39000-6584-4e75-b414-bed1e5929546_isrForCrosstalkSources_855391_32.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a39000-6584-4e75-b414-bed1e5929546_isrForCrosstalkSources_855391_32.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e9f4c4ba8541076cfb9ecb7a5d8faea with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f6171bf4-b2ac-4b1f-93d9-a2ecf9c62fd3_isrForCrosstalkSources_855430_52.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/f6171bf4-b2ac-4b1f-93d9-a2ecf9c62fd3_isrForCrosstalkSources_855430_52.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4c83238742b37372125b71bef18b3fe with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b8573cee-4239-497c-a68d-5327ad98bd14_isrForCrosstalkSources_855447_61.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b8573cee-4239-497c-a68d-5327ad98bd14_isrForCrosstalkSources_855447_61.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 994873518b00b2c7644709b2e92a0a4f with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dfd43642-dff0-48c6-bc2e-cb491a024352_isrForCrosstalkSources_855446_42.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dfd43642-dff0-48c6-bc2e-cb491a024352_isrForCrosstalkSources_855446_42.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d6b03509d79e17b2b5ab5a6a6c7a83d with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8bfe6c0d-e8e7-4224-84d8-07e2134378ac_isrForCrosstalkSources_855435_42.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8bfe6c0d-e8e7-4224-84d8-07e2134378ac_isrForCrosstalkSources_855435_42.stderr 2024-03-09 14:52:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964223 proc_lsst.multi INFO: [multi] job_id 16964223 2024-03-09 14:52:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-09 14:52:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964223 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964223 parsl.executors.status_handling DEBUG: Launched block 196 on executor multi with job ID 16964223 parsl.executors.status_handling INFO: Allocated block ID 197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-197 2024-03-09 14:52:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-09 14:52:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 181d984f847a5c233ebb5a1e499e49d0 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/2158ddf2-db5c-4a22-a911-6e4c1049bec9_isrForCrosstalkSources_855386_40.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/2158ddf2-db5c-4a22-a911-6e4c1049bec9_isrForCrosstalkSources_855386_40.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6c7d57b465ce2616c21958bad2e4ef with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4f63fbc7-c2bb-4441-a592-7ba014fdec84_isrForCrosstalkSources_855391_57.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4f63fbc7-c2bb-4441-a592-7ba014fdec84_isrForCrosstalkSources_855391_57.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5f6726729b3c0ce7e895606cd5513f0 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7f0b80e9-bece-4826-8859-167df417c9b0_isrForCrosstalkSources_855392_28.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7f0b80e9-bece-4826-8859-167df417c9b0_isrForCrosstalkSources_855392_28.stderr 2024-03-09 14:52:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964224 proc_lsst.multi INFO: [multi] job_id 16964224 2024-03-09 14:52:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-09 14:52:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964224 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964224 parsl.executors.status_handling DEBUG: Launched block 197 on executor multi with job ID 16964224 parsl.executors.status_handling INFO: Allocated block ID 198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-198 2024-03-09 14:52:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9385a862575aeb324087ae5e07da63c6 with result from task 619 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b9b1c96c-07dd-4ee6-8e53-1ac487b19805_isrForCrosstalkSources_855437_11.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b9b1c96c-07dd-4ee6-8e53-1ac487b19805_isrForCrosstalkSources_855437_11.stderr 2024-03-09 14:52:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964225 proc_lsst.multi INFO: [multi] job_id 16964225 2024-03-09 14:52:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-09 14:52:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964225 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964225 parsl.executors.status_handling DEBUG: Launched block 198 on executor multi with job ID 16964225 parsl.executors.status_handling INFO: Allocated block ID 199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-199 2024-03-09 14:52:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964226 proc_lsst.multi INFO: [multi] job_id 16964226 2024-03-09 14:52:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-09 14:52:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964226 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964226 parsl.executors.status_handling DEBUG: Launched block 199 on executor multi with job ID 16964226 parsl.executors.status_handling INFO: Allocated block ID 200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-200 2024-03-09 14:52:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f337da864a327ee61b113636e5deacb with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/94ecfe57-bea2-445f-8a4c-290e85106e38_isrForCrosstalkSources_855399_28.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/94ecfe57-bea2-445f-8a4c-290e85106e38_isrForCrosstalkSources_855399_28.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb8cd89ca35008a5cdac71a53a8d1fb8 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/dd58d932-fee4-42ed-b185-6d5ca10b775f_isrForCrosstalkSources_855416_6.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/dd58d932-fee4-42ed-b185-6d5ca10b775f_isrForCrosstalkSources_855416_6.stderr 2024-03-09 14:52:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964227 proc_lsst.multi INFO: [multi] job_id 16964227 2024-03-09 14:52:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-09 14:52:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964227 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964227 parsl.executors.status_handling DEBUG: Launched block 200 on executor multi with job ID 16964227 parsl.executors.status_handling INFO: Allocated block ID 201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-201 2024-03-09 14:52:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-09 14:52:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) 2024-03-09 14:52:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964228 parsl.dataflow.memoization DEBUG: Storing app cache entry d0e5d9e6e4030857c45b1ef27eb30ed3 with result from task 635 proc_lsst.multi INFO: [multi] job_id 16964228 2024-03-09 14:52:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-09 14:52:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964228 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964228 parsl.executors.status_handling DEBUG: Launched block 201 on executor multi with job ID 16964228 parsl.executors.status_handling INFO: Allocated block ID 202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-202 2024-03-09 14:52:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/88d189fe-4fef-4cae-b66b-f96fcd232a4e_isrForCrosstalkSources_855427_12.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/88d189fe-4fef-4cae-b66b-f96fcd232a4e_isrForCrosstalkSources_855427_12.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc2f171520afd15906eb0c55c9d214a4 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/82f53011-9ee1-404d-9da1-0beac0ec0f65_isrForCrosstalkSources_855380_41.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/82f53011-9ee1-404d-9da1-0beac0ec0f65_isrForCrosstalkSources_855380_41.stderr 2024-03-09 14:52:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964229 proc_lsst.multi INFO: [multi] job_id 16964229 2024-03-09 14:52:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-09 14:52:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964229 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964229 parsl.executors.status_handling DEBUG: Launched block 202 on executor multi with job ID 16964229 parsl.executors.status_handling INFO: Allocated block ID 203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-203 2024-03-09 14:52:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27a0e33ef1d0d1239335be944fa7c67c with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18c50c4a-f444-498a-970d-dae13cd0c338_isrForCrosstalkSources_855423_41.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18c50c4a-f444-498a-970d-dae13cd0c338_isrForCrosstalkSources_855423_41.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964230 proc_lsst.multi INFO: [multi] job_id 16964230 2024-03-09 14:52:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-09 14:52:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964230 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964230 parsl.executors.status_handling DEBUG: Launched block 203 on executor multi with job ID 16964230 parsl.executors.status_handling INFO: Allocated block ID 204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-204 2024-03-09 14:52:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-09 14:52:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964231 proc_lsst.multi INFO: [multi] job_id 16964231 2024-03-09 14:52:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-09 14:52:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964231 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964231 parsl.executors.status_handling DEBUG: Launched block 204 on executor multi with job ID 16964231 parsl.executors.status_handling INFO: Allocated block ID 205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-205 2024-03-09 14:52:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964232 proc_lsst.multi INFO: [multi] job_id 16964232 2024-03-09 14:52:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-09 14:52:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964232 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964232 parsl.executors.status_handling DEBUG: Launched block 205 on executor multi with job ID 16964232 parsl.executors.status_handling INFO: Allocated block ID 206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-206 2024-03-09 14:52:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e493d70816f9dfc4141fa7638781a6b6 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d7ed5e7b-264c-44c8-b1ce-2f247c4d64b5_isrForCrosstalkSources_855432_53.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d7ed5e7b-264c-44c8-b1ce-2f247c4d64b5_isrForCrosstalkSources_855432_53.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26d12f51622ed42e9f220f52aacb7c21 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81ed0258-4bbb-4d29-8058-9f1bff78a566_isrForCrosstalkSources_855398_59.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81ed0258-4bbb-4d29-8058-9f1bff78a566_isrForCrosstalkSources_855398_59.stderr 2024-03-09 14:52:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964233 proc_lsst.multi INFO: [multi] job_id 16964233 2024-03-09 14:52:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-09 14:52:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964233 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964233 parsl.executors.status_handling DEBUG: Launched block 206 on executor multi with job ID 16964233 parsl.executors.status_handling INFO: Allocated block ID 207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-207 2024-03-09 14:52:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-09 14:52:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7299218449a9c44b44fb854cb2cecc04 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/060da2a9-6b4c-4900-a594-9ae5ad1cc333_isrForCrosstalkSources_855396_31.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/060da2a9-6b4c-4900-a594-9ae5ad1cc333_isrForCrosstalkSources_855396_31.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24614113ae7f8bb4ea7bfa6954d96469 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8bdd2d29-d09c-46e7-95e8-c1f99d5fd8a7_isrForCrosstalkSources_855449_44.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8bdd2d29-d09c-46e7-95e8-c1f99d5fd8a7_isrForCrosstalkSources_855449_44.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4facd724ae3e25302622b7bca6643e08 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3a6b6cba-6d45-4dca-837b-7b76ae85a197_isrForCrosstalkSources_855426_42.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3a6b6cba-6d45-4dca-837b-7b76ae85a197_isrForCrosstalkSources_855426_42.stderr 2024-03-09 14:52:09 proc_lsst.multi:170 [INFO] [multi] job_id 16964234 proc_lsst.multi INFO: [multi] job_id 16964234 2024-03-09 14:52:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-09 14:52:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964234 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964234 parsl.executors.status_handling DEBUG: Launched block 207 on executor multi with job ID 16964234 parsl.executors.status_handling INFO: Allocated block ID 208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-208 2024-03-09 14:52:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cfff598cde030f5624265dec5b0522b with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/6ece48b3-cab5-4f6b-81f3-bf16257a96c8_isrForCrosstalkSources_855433_1.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/6ece48b3-cab5-4f6b-81f3-bf16257a96c8_isrForCrosstalkSources_855433_1.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23e2699acef5b79768dd25115144ba99 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/81634ecb-18d0-424d-b3de-50083315fce8_isrForCrosstalkSources_855419_45.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/81634ecb-18d0-424d-b3de-50083315fce8_isrForCrosstalkSources_855419_45.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b7141479f9ee4fe9ba6a74e01029d3 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2ec74cda-6aff-4b94-9748-d6989d45b2fd_isrForCrosstalkSources_855414_51.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2ec74cda-6aff-4b94-9748-d6989d45b2fd_isrForCrosstalkSources_855414_51.stderr 2024-03-09 14:52:09 proc_lsst.multi:170 [INFO] [multi] job_id 16964235 proc_lsst.multi INFO: [multi] job_id 16964235 2024-03-09 14:52:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-09 14:52:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964235 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964235 parsl.executors.status_handling DEBUG: Launched block 208 on executor multi with job ID 16964235 parsl.executors.status_handling INFO: Allocated block ID 209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-209 2024-03-09 14:52:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66828c2d10a86b576ec58def41ce547a with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/ee9f8a90-d094-408b-9774-02fb1c558b4f_isrForCrosstalkSources_855385_3.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/ee9f8a90-d094-408b-9774-02fb1c558b4f_isrForCrosstalkSources_855385_3.stderr 2024-03-09 14:52:09 proc_lsst.multi:170 [INFO] [multi] job_id 16964236 proc_lsst.multi INFO: [multi] job_id 16964236 2024-03-09 14:52:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-09 14:52:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964236 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964236 parsl.executors.status_handling DEBUG: Launched block 209 on executor multi with job ID 16964236 parsl.executors.status_handling INFO: Allocated block ID 210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-210 2024-03-09 14:52:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-09 14:52:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f765bbabe557cf95bc36d3b23ee9e8d4 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d1bdf053-75be-44bd-a2c3-91cf57e7e971_isrForCrosstalkSources_855392_40.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d1bdf053-75be-44bd-a2c3-91cf57e7e971_isrForCrosstalkSources_855392_40.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c14dc5651ed0a0de0eeed3f9b0eadde4 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/498b8665-c7ae-4abc-9df9-890e81f7355d_isrForCrosstalkSources_855453_60.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/498b8665-c7ae-4abc-9df9-890e81f7355d_isrForCrosstalkSources_855453_60.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a7d6109904cc8a9f51e5ce2d5871ddb with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b49dbf64-cac5-4c8b-9c4a-d5eb54e62b03_isrForCrosstalkSources_855444_9.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b49dbf64-cac5-4c8b-9c4a-d5eb54e62b03_isrForCrosstalkSources_855444_9.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd7c3790694aa0cd1ae25d125013b50a with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24cd8d8f-cb82-4c5a-899e-99848b6fadc9_isrForCrosstalkSources_855390_17.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24cd8d8f-cb82-4c5a-899e-99848b6fadc9_isrForCrosstalkSources_855390_17.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 676d98a8bc200276033abad9ea7e916c with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/fd4df691-9511-4504-97be-73781a5b67b2_isrForCrosstalkSources_855383_26.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/fd4df691-9511-4504-97be-73781a5b67b2_isrForCrosstalkSources_855383_26.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6e9d6e7f0b009aad7b406d6f3354c2b with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c120748b-f606-40c9-8d5c-a8bfa1130cc9_isrForCrosstalkSources_855383_38.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c120748b-f606-40c9-8d5c-a8bfa1130cc9_isrForCrosstalkSources_855383_38.stderr parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7070dfc34003126244fd041cc216d6b6 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/8255a95d-1d6f-447b-b3a3-48925aa46b6d_isrForCrosstalkSources_855385_1.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/8255a95d-1d6f-447b-b3a3-48925aa46b6d_isrForCrosstalkSources_855385_1.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d72b19664876c098ac6844ecb01cc3ef with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/10ec6cfb-84b6-41c1-a187-1cbda34080f8_isrForCrosstalkSources_855381_22.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/10ec6cfb-84b6-41c1-a187-1cbda34080f8_isrForCrosstalkSources_855381_22.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33f96149595490533ee42e6ed8f02fad with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cb57b7cc-1a1f-497a-bfd3-6e9b5722c39c_isrForCrosstalkSources_855454_49.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cb57b7cc-1a1f-497a-bfd3-6e9b5722c39c_isrForCrosstalkSources_855454_49.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ab05c42fb1dde829c2d0114c53f64b9 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/61268c9e-8632-4773-8b0c-92ed263e52a5_isrForCrosstalkSources_855417_25.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/61268c9e-8632-4773-8b0c-92ed263e52a5_isrForCrosstalkSources_855417_25.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a90e201ad052b10f93506b4de12fcb67 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b020bf5b-dbd4-4374-a1f3-6a5d155b6f4c_isrForCrosstalkSources_855446_54.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b020bf5b-dbd4-4374-a1f3-6a5d155b6f4c_isrForCrosstalkSources_855446_54.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aa846290a2de4d1502df4db231d35c5 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c453eb69-1b9c-40f6-b221-a271e9960bfc_isrForCrosstalkSources_855404_47.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c453eb69-1b9c-40f6-b221-a271e9960bfc_isrForCrosstalkSources_855404_47.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddd0e83134c83b9659d2f451bca44300 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/8eee20c6-cae9-418a-a59f-6df8034582f9_isrForCrosstalkSources_855433_48.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/8eee20c6-cae9-418a-a59f-6df8034582f9_isrForCrosstalkSources_855433_48.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a88798880c386f04ea4cca15edb21444 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab4760bd-8ea2-4995-a4ee-4f9d756642b4_isrForCrosstalkSources_855437_41.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab4760bd-8ea2-4995-a4ee-4f9d756642b4_isrForCrosstalkSources_855437_41.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a5581a06ad8859fe673984f582ad429 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4315aed2-7a46-480e-b6a0-9e888ee04fe0_isrForCrosstalkSources_855405_3.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4315aed2-7a46-480e-b6a0-9e888ee04fe0_isrForCrosstalkSources_855405_3.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee232055de828da05ddbe0bfd39381e with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0b041358-705a-4d85-bcac-eb3100c4246a_isrForCrosstalkSources_855390_62.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0b041358-705a-4d85-bcac-eb3100c4246a_isrForCrosstalkSources_855390_62.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7244700e99a47d6e5113c39ac7715f58 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0aa14a4b-29d5-4284-a29f-04e360207b23_isrForCrosstalkSources_855405_50.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0aa14a4b-29d5-4284-a29f-04e360207b23_isrForCrosstalkSources_855405_50.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01459185a63e43c48e465199265ee2af with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/67fc0035-199c-4a6a-bb60-f3099491bcf0_isrForCrosstalkSources_855411_31.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/67fc0035-199c-4a6a-bb60-f3099491bcf0_isrForCrosstalkSources_855411_31.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13ca1b3f67df0dccda0f6be93e22a72f with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84563fe8-f65f-4901-a6bb-171bbd45c31f_isrForCrosstalkSources_855416_61.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84563fe8-f65f-4901-a6bb-171bbd45c31f_isrForCrosstalkSources_855416_61.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83df9b98a7037374c543592181532f48 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/96af1b50-7526-4a92-a5d4-23759e6d3aa4_isrForCrosstalkSources_855384_50.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/96af1b50-7526-4a92-a5d4-23759e6d3aa4_isrForCrosstalkSources_855384_50.stderr 2024-03-09 14:52:11 proc_lsst.multi:170 [INFO] [multi] job_id 16964237 proc_lsst.multi INFO: [multi] job_id 16964237 2024-03-09 14:52:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-09 14:52:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964237 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964237 parsl.executors.status_handling DEBUG: Launched block 210 on executor multi with job ID 16964237 parsl.executors.status_handling INFO: Allocated block ID 211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-211 2024-03-09 14:52:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 2024-03-09 14:52:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-09 14:52:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5beefa09b6ee3290e8c049492ad0a77d with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7299a1e4-89f6-4b87-8f94-ce2bff6bb28e_isrForCrosstalkSources_855450_33.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7299a1e4-89f6-4b87-8f94-ce2bff6bb28e_isrForCrosstalkSources_855450_33.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc898725e5c22b648094af1c0a707acf with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e6a290ef-cc63-4140-aa4d-354ee73204ba_isrForCrosstalkSources_855398_6.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e6a290ef-cc63-4140-aa4d-354ee73204ba_isrForCrosstalkSources_855398_6.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 717fd83b7c08da4c4768e9fd3085baa4 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f71e72ab-5d91-4689-be58-d5f1e5ef5b8e_isrForCrosstalkSources_855446_33.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f71e72ab-5d91-4689-be58-d5f1e5ef5b8e_isrForCrosstalkSources_855446_33.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 026431b7fc131279654f5e6a558c9ee5 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/0e763dae-5063-42d6-b37b-2e6261e77448_isrForCrosstalkSources_855414_53.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/0e763dae-5063-42d6-b37b-2e6261e77448_isrForCrosstalkSources_855414_53.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5909c988984f93cafdcbe8b8e00c823d with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8a64f2c-30aa-4a24-994c-c233779a41e2_isrForCrosstalkSources_855412_54.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8a64f2c-30aa-4a24-994c-c233779a41e2_isrForCrosstalkSources_855412_54.stderr 2024-03-09 14:52:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964238 proc_lsst.multi INFO: [multi] job_id 16964238 2024-03-09 14:52:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-09 14:52:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964238 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964238 parsl.executors.status_handling DEBUG: Launched block 211 on executor multi with job ID 16964238 parsl.executors.status_handling INFO: Allocated block ID 212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-212 2024-03-09 14:52:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e5b0985c0febf02315d2bc3d083e214 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8347059c-a5da-4897-8829-31a5006c48e8_isrForCrosstalkSources_855406_24.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/8347059c-a5da-4897-8829-31a5006c48e8_isrForCrosstalkSources_855406_24.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38ac94c757dced93c56a7316c64a6194 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ef7c6578-1a31-409a-b423-45c648c68aff_isrForCrosstalkSources_855388_58.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ef7c6578-1a31-409a-b423-45c648c68aff_isrForCrosstalkSources_855388_58.stderr 2024-03-09 14:52:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964239 proc_lsst.multi INFO: [multi] job_id 16964239 2024-03-09 14:52:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-09 14:52:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964239 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964239 parsl.executors.status_handling DEBUG: Launched block 212 on executor multi with job ID 16964239 parsl.executors.status_handling INFO: Allocated block ID 213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-213 2024-03-09 14:52:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-09 14:52:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d432874bfe45cc1d7e04850b74439aed with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/33f53db0-3efb-4435-b5e5-48d921616e02_isrForCrosstalkSources_855398_19.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/33f53db0-3efb-4435-b5e5-48d921616e02_isrForCrosstalkSources_855398_19.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 521cd15a1b2a0f111536ccd2dcd4b1ce with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/846826b1-44cd-4eba-b61c-3575ba511ac7_isrForCrosstalkSources_855405_40.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/846826b1-44cd-4eba-b61c-3575ba511ac7_isrForCrosstalkSources_855405_40.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb34ae0f099a0286bf96c4e116099731 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6630d9fa-12d6-459b-8ca3-e498150fe2c1_isrForCrosstalkSources_855407_32.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6630d9fa-12d6-459b-8ca3-e498150fe2c1_isrForCrosstalkSources_855407_32.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77d618a2ec5033bffbcf9cb164b43c73 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6800ab46-434d-4992-81eb-6ffa7717c284_isrForCrosstalkSources_855453_59.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/6800ab46-434d-4992-81eb-6ffa7717c284_isrForCrosstalkSources_855453_59.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956d1fd16751c535c5af9c0cdfa4579c with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cb891bdb-2b67-4ac6-aa20-a80d392ab9e4_isrForCrosstalkSources_855418_34.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cb891bdb-2b67-4ac6-aa20-a80d392ab9e4_isrForCrosstalkSources_855418_34.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2147803bec99e5dfa2f8cb181c416e7f with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9fab4831-8572-4578-8930-739e9f6f630d_isrForCrosstalkSources_855450_42.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9fab4831-8572-4578-8930-739e9f6f630d_isrForCrosstalkSources_855450_42.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138bd4a4f6740c67667bf6c8d66d03fa with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29a06e5d-9c95-4643-be3e-fde3ed4d5e0c_isrForCrosstalkSources_855447_50.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29a06e5d-9c95-4643-be3e-fde3ed4d5e0c_isrForCrosstalkSources_855447_50.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0351069189a465960005e36b55c62198 with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/491d3030-b499-4f42-a2ad-42a89640945b_isrForCrosstalkSources_855383_56.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/491d3030-b499-4f42-a2ad-42a89640945b_isrForCrosstalkSources_855383_56.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d56869b1dee5a8c6352a41f3fb0d3823 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5f94427a-6a45-40b4-b695-af6b29c88a22_isrForCrosstalkSources_855403_61.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5f94427a-6a45-40b4-b695-af6b29c88a22_isrForCrosstalkSources_855403_61.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce34db73e4504b72d0b90c5bb4af85f1 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ba43a520-6ccb-4fc8-9294-95a4533011d2_isrForCrosstalkSources_855384_19.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ba43a520-6ccb-4fc8-9294-95a4533011d2_isrForCrosstalkSources_855384_19.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b8b77faa88a4f73f87c0afe0623a0d5 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ed837a86-77e7-459a-b2f3-f16e378fe107_isrForCrosstalkSources_855445_32.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ed837a86-77e7-459a-b2f3-f16e378fe107_isrForCrosstalkSources_855445_32.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52185b8c7274102377511472be905c97 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/69dcc0f3-abec-4812-a1f0-f290deaa4db2_isrForCrosstalkSources_855429_46.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/69dcc0f3-abec-4812-a1f0-f290deaa4db2_isrForCrosstalkSources_855429_46.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1b67996eb6c9a0ad4bd4b0878f1df45 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/baae7d1a-7f29-43ba-97bc-c176accd720e_isrForCrosstalkSources_855406_39.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/baae7d1a-7f29-43ba-97bc-c176accd720e_isrForCrosstalkSources_855406_39.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2b81a540a46e6a556ad3f76b1656202 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e8e14a1a-bae1-4a68-90d5-8efde4cd4158_isrForCrosstalkSources_855405_25.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e8e14a1a-bae1-4a68-90d5-8efde4cd4158_isrForCrosstalkSources_855405_25.stderr 2024-03-09 14:52:14 proc_lsst.multi:170 [INFO] [multi] job_id 16964242 proc_lsst.multi INFO: [multi] job_id 16964242 2024-03-09 14:52:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-09 14:52:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964242 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964242 parsl.executors.status_handling DEBUG: Launched block 213 on executor multi with job ID 16964242 parsl.executors.status_handling INFO: Allocated block ID 214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-214 2024-03-09 14:52:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ececd3121fd2df7896fc01240131761 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/88af174a-8897-4161-bc16-b54dcf8b03c5_isrForCrosstalkSources_855410_52.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/88af174a-8897-4161-bc16-b54dcf8b03c5_isrForCrosstalkSources_855410_52.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd912447b2a2780bb752578623df423a with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5eda16c6-d840-4ea4-8f87-37ce09c11f90_isrForCrosstalkSources_855419_25.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5eda16c6-d840-4ea4-8f87-37ce09c11f90_isrForCrosstalkSources_855419_25.stderr 2024-03-09 14:52:14 proc_lsst.multi:170 [INFO] [multi] job_id 16964243 proc_lsst.multi INFO: [multi] job_id 16964243 2024-03-09 14:52:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-09 14:52:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964243 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964243 parsl.executors.status_handling DEBUG: Launched block 214 on executor multi with job ID 16964243 parsl.executors.status_handling INFO: Allocated block ID 215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-215 2024-03-09 14:52:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-09 14:52:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4041f97690b44dfe9e7bedfd0a3c4e6 with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3b1c8ae-b16a-48cb-a21c-885e6dd94a5d_isrForCrosstalkSources_855395_33.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3b1c8ae-b16a-48cb-a21c-885e6dd94a5d_isrForCrosstalkSources_855395_33.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d44f1ed7fad549dcf976185969332c33 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fce2a697-3e3a-4af3-86d1-eeb84338b8c8_isrForCrosstalkSources_855425_44.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fce2a697-3e3a-4af3-86d1-eeb84338b8c8_isrForCrosstalkSources_855425_44.stderr 2024-03-09 14:52:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964244 proc_lsst.multi INFO: [multi] job_id 16964244 2024-03-09 14:52:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-09 14:52:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964244 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964244 parsl.executors.status_handling DEBUG: Launched block 215 on executor multi with job ID 16964244 parsl.executors.status_handling INFO: Allocated block ID 216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-216 2024-03-09 14:52:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7058f8837898ffe049edaed01263ee51 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7d873681-0e44-47ce-ab17-726e00ecd218_isrForCrosstalkSources_855396_44.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7d873681-0e44-47ce-ab17-726e00ecd218_isrForCrosstalkSources_855396_44.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8ff932bdee9b50bd7fa85bb934bd23 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/3a158af4-597b-4506-8c8c-b932d1cbe34a_isrForCrosstalkSources_855396_56.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/3a158af4-597b-4506-8c8c-b932d1cbe34a_isrForCrosstalkSources_855396_56.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87111de109bf0e98ea9f7b67b7eab32e with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f7682276-6594-40bd-a009-fbbec542057f_isrForCrosstalkSources_855444_52.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f7682276-6594-40bd-a009-fbbec542057f_isrForCrosstalkSources_855444_52.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 716ee7d007cc59ddfdc8edd924b86809 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8460d0a5-9d8a-4ba8-95b7-17fd2aea2e2a_isrForCrosstalkSources_855382_27.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8460d0a5-9d8a-4ba8-95b7-17fd2aea2e2a_isrForCrosstalkSources_855382_27.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fadd7bb6b50febf6591732b9723c785b with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8ef3b5a6-5bec-441d-bcd4-1b9e36906a88_isrForCrosstalkSources_855446_37.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8ef3b5a6-5bec-441d-bcd4-1b9e36906a88_isrForCrosstalkSources_855446_37.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c35a01a5b737f5164685da6bca8e917c with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9cd2348d-f15c-4ad3-88d9-242fb0e4b4d6_isrForCrosstalkSources_855422_32.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9cd2348d-f15c-4ad3-88d9-242fb0e4b4d6_isrForCrosstalkSources_855422_32.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ddea91c7dcabafa79349f6536c1f606 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/90dec67f-699c-4136-87ba-0bd2e1b881b6_isrForCrosstalkSources_855448_54.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/90dec67f-699c-4136-87ba-0bd2e1b881b6_isrForCrosstalkSources_855448_54.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1708ead7e097ffca49829c656be685f6 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0a8a0e58-dd03-4a0f-bd76-895129a33b0f_isrForCrosstalkSources_855458_60.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0a8a0e58-dd03-4a0f-bd76-895129a33b0f_isrForCrosstalkSources_855458_60.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 741705bf9f016230a12cfe74df565a05 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8f54ebc1-5fb2-4947-962a-2a970080c14f_isrForCrosstalkSources_855382_13.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8f54ebc1-5fb2-4947-962a-2a970080c14f_isrForCrosstalkSources_855382_13.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ed16a7f786d354ff0cab8b59741723 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/eb355cfc-b734-44ea-bbae-977a7f195d82_isrForCrosstalkSources_855449_8.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/eb355cfc-b734-44ea-bbae-977a7f195d82_isrForCrosstalkSources_855449_8.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 568a441063b38582692718aa05cd8c6a with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ae517807-0780-4d35-bff8-5453c2a71b47_isrForCrosstalkSources_855435_35.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ae517807-0780-4d35-bff8-5453c2a71b47_isrForCrosstalkSources_855435_35.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edf2f57f2df04eed670d837b6c4390da with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/17740517-fe19-437e-a820-5d4f29f4692e_isrForCrosstalkSources_855398_51.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/17740517-fe19-437e-a820-5d4f29f4692e_isrForCrosstalkSources_855398_51.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de9d4a2e7a4a224d33ba999629ab66e6 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/03a43c11-1b90-40c3-be56-86d4e00c7f0b_isrForCrosstalkSources_855428_14.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/03a43c11-1b90-40c3-be56-86d4e00c7f0b_isrForCrosstalkSources_855428_14.stderr parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6f6187be6a1855315e997e3af9328a3 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6b3b6acd-242d-48e9-a3a9-0a912761cd68_isrForCrosstalkSources_855402_8.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6b3b6acd-242d-48e9-a3a9-0a912761cd68_isrForCrosstalkSources_855402_8.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66fcc9bc866d6dbfcab8b1ae7956e291 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8f5161dd-31c6-48c0-b682-291c0241bef7_isrForCrosstalkSources_855414_32.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8f5161dd-31c6-48c0-b682-291c0241bef7_isrForCrosstalkSources_855414_32.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5331c8f6ba74d7fd8939ae9fc3dae32e with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eecf7047-af51-423b-bb6e-be8ca98cf30e_isrForCrosstalkSources_855381_46.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eecf7047-af51-423b-bb6e-be8ca98cf30e_isrForCrosstalkSources_855381_46.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 937e1037db8671de1275f4fc3d42423b with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/53246840-1222-48a8-83a3-2588fe4f3fa5_isrForCrosstalkSources_855418_8.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/53246840-1222-48a8-83a3-2588fe4f3fa5_isrForCrosstalkSources_855418_8.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 715cfac95dbf2457bc68cff4c1870f32 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca5e15ae-7c0e-4990-b408-61410b7a383f_isrForCrosstalkSources_855400_38.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca5e15ae-7c0e-4990-b408-61410b7a383f_isrForCrosstalkSources_855400_38.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5e1892311fa7d41e95c5faaa45f1ec0 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/aac80d6f-1831-4513-8b22-af24c6c6dc32_isrForCrosstalkSources_855454_2.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/aac80d6f-1831-4513-8b22-af24c6c6dc32_isrForCrosstalkSources_855454_2.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68cdffb739cab136e540aca5e05f1945 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/27c45629-4d61-41fd-94b6-38ad218b752a_isrForCrosstalkSources_855418_42.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/27c45629-4d61-41fd-94b6-38ad218b752a_isrForCrosstalkSources_855418_42.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 329f5ae81dddf928982f3529de10d545 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bc631c22-5dd2-4fdd-b151-a9ca1df2060d_isrForCrosstalkSources_855442_3.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bc631c22-5dd2-4fdd-b151-a9ca1df2060d_isrForCrosstalkSources_855442_3.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9765f48ac6972ac90605d88fbaa72526 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf75224-ef9c-4a8f-98e0-a1cde723b061_isrForCrosstalkSources_855388_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/baf75224-ef9c-4a8f-98e0-a1cde723b061_isrForCrosstalkSources_855388_13.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae68f88c0ac9fc778a9e2a41be798e3c with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1ca5c878-5636-4658-9952-cedaa718524b_isrForCrosstalkSources_855383_59.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1ca5c878-5636-4658-9952-cedaa718524b_isrForCrosstalkSources_855383_59.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f69ec3f6450da9667fc1b850030fbfb4 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0cfb8404-c35a-4c46-90a4-549ce15422c2_isrForCrosstalkSources_855420_57.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0cfb8404-c35a-4c46-90a4-549ce15422c2_isrForCrosstalkSources_855420_57.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bc37832687e6a57794a9512f000405c with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/540cc22f-e2f4-4257-80e4-2e5d0d1df20f_isrForCrosstalkSources_855399_27.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/540cc22f-e2f4-4257-80e4-2e5d0d1df20f_isrForCrosstalkSources_855399_27.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bab5be049f7f2a8cb4a60caf48c9cb8 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/735bea73-d488-498e-be2a-ef0a9a54c40b_isrForCrosstalkSources_855440_53.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/735bea73-d488-498e-be2a-ef0a9a54c40b_isrForCrosstalkSources_855440_53.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52cc3dc1683ce876c1ca3840f02c0a6d with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8e4e9671-3f3a-4b77-8709-28a129c62908_isrForCrosstalkSources_855443_39.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8e4e9671-3f3a-4b77-8709-28a129c62908_isrForCrosstalkSources_855443_39.stderr 2024-03-09 14:52:15 proc_lsst.multi:170 [INFO] [multi] job_id 16964245 proc_lsst.multi INFO: [multi] job_id 16964245 2024-03-09 14:52:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-09 14:52:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964245 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964245 parsl.executors.status_handling DEBUG: Launched block 216 on executor multi with job ID 16964245 parsl.executors.status_handling INFO: Allocated block ID 217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-217 2024-03-09 14:52:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-09 14:52:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7638607d37f5e7e38003663108d6a74 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f99e1f06-dcbf-40c2-9483-6df1b5a98da3_isrForCrosstalkSources_855455_32.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f99e1f06-dcbf-40c2-9483-6df1b5a98da3_isrForCrosstalkSources_855455_32.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab72f2a978f637aed72439ba09664c9f with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4cb0f006-78a1-474e-8882-eb3638afdb3f_isrForCrosstalkSources_855419_56.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4cb0f006-78a1-474e-8882-eb3638afdb3f_isrForCrosstalkSources_855419_56.stderr 2024-03-09 14:52:16 proc_lsst.multi:170 [INFO] [multi] job_id 16964246 proc_lsst.multi INFO: [multi] job_id 16964246 2024-03-09 14:52:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-09 14:52:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964246 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964246 parsl.executors.status_handling DEBUG: Launched block 217 on executor multi with job ID 16964246 parsl.executors.status_handling INFO: Allocated block ID 218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-218 2024-03-09 14:52:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:16 proc_lsst.multi:170 [INFO] [multi] job_id 16964247 proc_lsst.multi INFO: [multi] job_id 16964247 2024-03-09 14:52:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-09 14:52:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964247 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964247 parsl.executors.status_handling DEBUG: Launched block 218 on executor multi with job ID 16964247 parsl.executors.status_handling INFO: Allocated block ID 219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-219 2024-03-09 14:52:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-09 14:52:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d772d65c82ca2590a4c4266eb320f0c8 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/80dc4fb6-865d-4049-83e5-e5fb9d8393b6_isrForCrosstalkSources_855379_27.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/80dc4fb6-865d-4049-83e5-e5fb9d8393b6_isrForCrosstalkSources_855379_27.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e255883d04e7266d1a613373a940c6b with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a849958-2c45-45bc-a6ab-d78ae65b674c_isrForCrosstalkSources_855452_24.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2a849958-2c45-45bc-a6ab-d78ae65b674c_isrForCrosstalkSources_855452_24.stderr 2024-03-09 14:52:17 proc_lsst.multi:170 [INFO] [multi] job_id 16964248 proc_lsst.multi INFO: [multi] job_id 16964248 2024-03-09 14:52:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-09 14:52:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964248 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964248 parsl.executors.status_handling DEBUG: Launched block 219 on executor multi with job ID 16964248 parsl.executors.status_handling INFO: Allocated block ID 220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-220 2024-03-09 14:52:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b61cad0310bad243fb803db772ee5a6a with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58182ce0-5f4e-4275-84a1-c45beeab8266_isrForCrosstalkSources_855407_25.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58182ce0-5f4e-4275-84a1-c45beeab8266_isrForCrosstalkSources_855407_25.stderr parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9ad25a32ef8ad6b8935d054274367e with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/75323d62-087f-4d7c-b288-416f9bfba915_isrForCrosstalkSources_855421_17.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/75323d62-087f-4d7c-b288-416f9bfba915_isrForCrosstalkSources_855421_17.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 991da20cff810f745703587a3489f848 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/a3333d50-3e97-4639-ba24-dfa9f628df4c_isrForCrosstalkSources_855414_14.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/a3333d50-3e97-4639-ba24-dfa9f628df4c_isrForCrosstalkSources_855414_14.stderr 2024-03-09 14:52:17 proc_lsst.multi:170 [INFO] [multi] job_id 16964249 proc_lsst.multi INFO: [multi] job_id 16964249 2024-03-09 14:52:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-09 14:52:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964249 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964249 parsl.executors.status_handling DEBUG: Launched block 220 on executor multi with job ID 16964249 parsl.executors.status_handling INFO: Allocated block ID 221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-221 2024-03-09 14:52:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57242c2021bb4b07abfe9cd516377f4f with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/223baf76-bc6d-4699-a591-6d39fc225bee_isrForCrosstalkSources_855403_19.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/223baf76-bc6d-4699-a591-6d39fc225bee_isrForCrosstalkSources_855403_19.stderr parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7ca4d55b52ee116a43c17985cc51bae with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/f27a13d6-19b9-42d9-8a96-17112f717e44_isrForCrosstalkSources_855394_5.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/f27a13d6-19b9-42d9-8a96-17112f717e44_isrForCrosstalkSources_855394_5.stderr 2024-03-09 14:52:17 proc_lsst.multi:170 [INFO] [multi] job_id 16964250 proc_lsst.multi INFO: [multi] job_id 16964250 2024-03-09 14:52:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-09 14:52:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964250 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964250 parsl.executors.status_handling DEBUG: Launched block 221 on executor multi with job ID 16964250 parsl.executors.status_handling INFO: Allocated block ID 222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-222 2024-03-09 14:52:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-09 14:52:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:52:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964251 proc_lsst.multi INFO: [multi] job_id 16964251 2024-03-09 14:52:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-09 14:52:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964251 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964251 parsl.executors.status_handling DEBUG: Launched block 222 on executor multi with job ID 16964251 parsl.executors.status_handling INFO: Allocated block ID 223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-223 2024-03-09 14:52:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b543843a842cc48a6c309ec44bbaeac with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/63fbdde6-5b23-4f2a-ae41-e35cdca51cac_isrForCrosstalkSources_855434_34.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/63fbdde6-5b23-4f2a-ae41-e35cdca51cac_isrForCrosstalkSources_855434_34.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 152a1755383de392c0fadd980cae8e74 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/37c03dbb-48d3-4cf4-b402-7ce6e6062c38_isrForCrosstalkSources_855454_16.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/37c03dbb-48d3-4cf4-b402-7ce6e6062c38_isrForCrosstalkSources_855454_16.stderr 2024-03-09 14:52:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964252 proc_lsst.multi INFO: [multi] job_id 16964252 2024-03-09 14:52:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-09 14:52:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964252 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964252 parsl.executors.status_handling DEBUG: Launched block 223 on executor multi with job ID 16964252 parsl.executors.status_handling INFO: Allocated block ID 224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-224 2024-03-09 14:52:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d489c72138927b25a9385a845c7fe155 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/aff462e7-f22c-476f-aad3-59503a30d98a_isrForCrosstalkSources_855421_28.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/aff462e7-f22c-476f-aad3-59503a30d98a_isrForCrosstalkSources_855421_28.stderr 2024-03-09 14:52:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964253 proc_lsst.multi INFO: [multi] job_id 16964253 2024-03-09 14:52:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-09 14:52:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964253 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964253 parsl.executors.status_handling DEBUG: Launched block 224 on executor multi with job ID 16964253 parsl.executors.status_handling INFO: Allocated block ID 225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-225 2024-03-09 14:52:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37f04303242f8baea72dc8319506ae7a with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7ff971a8-4032-4cac-8fac-eda54c6c2de5_isrForCrosstalkSources_855450_11.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7ff971a8-4032-4cac-8fac-eda54c6c2de5_isrForCrosstalkSources_855450_11.stderr 2024-03-09 14:52:18 proc_lsst.multi:170 [INFO] [multi] job_id 16964254 proc_lsst.multi INFO: [multi] job_id 16964254 2024-03-09 14:52:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-09 14:52:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964254 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964254 parsl.executors.status_handling DEBUG: Launched block 225 on executor multi with job ID 16964254 parsl.executors.status_handling INFO: Allocated block ID 226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-226 2024-03-09 14:52:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-09 14:52:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0871cfa635ac4ab9386e4e1c3c8f794c with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/664ac984-c781-400f-8d6a-246865ff1149_isrForCrosstalkSources_855394_33.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/664ac984-c781-400f-8d6a-246865ff1149_isrForCrosstalkSources_855394_33.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74931db9111aabee8646f4bd92124a66 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/39ed7f4a-856d-4301-9f94-e177c80c6d33_isrForCrosstalkSources_855433_62.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/39ed7f4a-856d-4301-9f94-e177c80c6d33_isrForCrosstalkSources_855433_62.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b99a57e2ecf69f081a73298ffb3346ee with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/74cb720b-34d1-44cc-9232-20cfb424deaf_isrForCrosstalkSources_855394_48.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/74cb720b-34d1-44cc-9232-20cfb424deaf_isrForCrosstalkSources_855394_48.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b04505f5695fa8f21be88eaeebda2e96 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/712bbbfc-aea2-4896-b4c2-08e775ccc05f_isrForCrosstalkSources_855450_62.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/712bbbfc-aea2-4896-b4c2-08e775ccc05f_isrForCrosstalkSources_855450_62.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b222a0f29fc9d0b062f93f4d12e267f3 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c00f6760-63b6-4498-b042-709af639604a_isrForCrosstalkSources_855441_9.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c00f6760-63b6-4498-b042-709af639604a_isrForCrosstalkSources_855441_9.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0091bc7489cf3bb18f20def541be080 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e127f447-54a8-464f-9b51-31176cb5d292_isrForCrosstalkSources_855413_39.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e127f447-54a8-464f-9b51-31176cb5d292_isrForCrosstalkSources_855413_39.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622d35ab7a95cd1cb9db9d27f312ce56 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4e3792d2-27da-47e9-9a79-9a89a1c4c5d1_isrForCrosstalkSources_855388_59.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4e3792d2-27da-47e9-9a79-9a89a1c4c5d1_isrForCrosstalkSources_855388_59.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73e2945a3c51917eda66262306e6ee3d with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/52c3fce0-e7e8-4536-bbff-727a4a3ae340_isrForCrosstalkSources_855434_6.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/52c3fce0-e7e8-4536-bbff-727a4a3ae340_isrForCrosstalkSources_855434_6.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00bbe1213e1268a466c4bb2b35d43129 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/87275bf1-d210-4b6f-8a0d-d2b2af963a39_isrForCrosstalkSources_855444_56.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/87275bf1-d210-4b6f-8a0d-d2b2af963a39_isrForCrosstalkSources_855444_56.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ab563e9d097ede93020e4f29c7945cc with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/976916db-1f5d-42f5-bc78-a595a95eed6c_isrForCrosstalkSources_855393_40.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/976916db-1f5d-42f5-bc78-a595a95eed6c_isrForCrosstalkSources_855393_40.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88a05a2c51635b8fe8f49da46b060905 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f33fc030-10f8-44b2-9d5e-a543f45f5f7e_isrForCrosstalkSources_855455_37.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f33fc030-10f8-44b2-9d5e-a543f45f5f7e_isrForCrosstalkSources_855455_37.stderr 2024-03-09 14:52:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964255 proc_lsst.multi INFO: [multi] job_id 16964255 2024-03-09 14:52:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-09 14:52:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964255 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964255 parsl.executors.status_handling DEBUG: Launched block 226 on executor multi with job ID 16964255 parsl.executors.status_handling INFO: Allocated block ID 227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-227 2024-03-09 14:52:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68c3d25b4356f86b195213a000ad99f1 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/204de22e-257e-41bb-9e30-e8de5a3d87e9_isrForCrosstalkSources_855403_24.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/204de22e-257e-41bb-9e30-e8de5a3d87e9_isrForCrosstalkSources_855403_24.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86cd24b455832b115aa1a24028dc2209 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/df07a6b6-bca1-48d1-86b2-1e552173c72a_isrForCrosstalkSources_855434_28.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/df07a6b6-bca1-48d1-86b2-1e552173c72a_isrForCrosstalkSources_855434_28.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc99ceefb9d3b715ef7ca005dc57788 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e4236383-5bf3-47fd-865d-f4937f354dcd_isrForCrosstalkSources_855451_51.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e4236383-5bf3-47fd-865d-f4937f354dcd_isrForCrosstalkSources_855451_51.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b73ac018643e8d75ef10b30fe9a70782 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/65cce904-c77c-4673-8810-1f9e160b5e65_isrForCrosstalkSources_855449_4.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/65cce904-c77c-4673-8810-1f9e160b5e65_isrForCrosstalkSources_855449_4.stderr 2024-03-09 14:52:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964256 proc_lsst.multi INFO: [multi] job_id 16964256 2024-03-09 14:52:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-09 14:52:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964256 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964256 parsl.executors.status_handling DEBUG: Launched block 227 on executor multi with job ID 16964256 parsl.executors.status_handling INFO: Allocated block ID 228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-228 2024-03-09 14:52:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5be0c1eafcfb3db729593ee63b869da7 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/567b334c-0e93-4778-9419-61fa562fd9d4_isrForCrosstalkSources_855454_11.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/567b334c-0e93-4778-9419-61fa562fd9d4_isrForCrosstalkSources_855454_11.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b03ed853a4752c4320d1139d8a2255d with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c12aef35-a75a-48b8-bd6a-a4f772b5483b_isrForCrosstalkSources_855409_38.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c12aef35-a75a-48b8-bd6a-a4f772b5483b_isrForCrosstalkSources_855409_38.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78eae4d5bbd5694768333dbc9d5cdf94 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/20a96fcb-c53a-4120-bd4f-e337da5304ae_isrForCrosstalkSources_855398_46.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/20a96fcb-c53a-4120-bd4f-e337da5304ae_isrForCrosstalkSources_855398_46.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f3039975af14ca3e3016e5b5f0f1087 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/6a86f019-0589-40e1-bc43-082831cb0e9c_isrForCrosstalkSources_855455_9.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/6a86f019-0589-40e1-bc43-082831cb0e9c_isrForCrosstalkSources_855455_9.stderr 2024-03-09 14:52:19 proc_lsst.multi:170 [INFO] [multi] job_id 16964257 proc_lsst.multi INFO: [multi] job_id 16964257 2024-03-09 14:52:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-09 14:52:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964257 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964257 parsl.executors.status_handling DEBUG: Launched block 228 on executor multi with job ID 16964257 parsl.executors.status_handling INFO: Allocated block ID 229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-229 2024-03-09 14:52:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-09 14:52:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34dc8cdfaa323c93540f0e35befa7191 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cc45d36e-147a-495a-9ce4-4fb659034644_isrForCrosstalkSources_855411_15.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cc45d36e-147a-495a-9ce4-4fb659034644_isrForCrosstalkSources_855411_15.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00f2b928fd9577cef90692f4a518818b with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6c373c01-7427-4810-a60a-d88a20b1d17e_isrForCrosstalkSources_855394_21.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6c373c01-7427-4810-a60a-d88a20b1d17e_isrForCrosstalkSources_855394_21.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06c02659f50e397ddcb71a8b07e6e429 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0beb003e-9437-45ee-885f-9856f621df6a_isrForCrosstalkSources_855420_24.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0beb003e-9437-45ee-885f-9856f621df6a_isrForCrosstalkSources_855420_24.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dc626c027106488a70fb33e93aeb4d1 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/76f91000-94af-4c85-98a1-38831260a687_isrForCrosstalkSources_855398_17.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/76f91000-94af-4c85-98a1-38831260a687_isrForCrosstalkSources_855398_17.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 223b2ade1ffd35428e832af51c0967b8 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e08dcc5a-b56c-4a1b-9be4-8f944091b9cb_isrForCrosstalkSources_855406_59.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e08dcc5a-b56c-4a1b-9be4-8f944091b9cb_isrForCrosstalkSources_855406_59.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d8d149826a0c39a31edd9b0e030ba27 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/343ca097-18e4-48e6-a817-60aa420c43a8_isrForCrosstalkSources_855428_22.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/343ca097-18e4-48e6-a817-60aa420c43a8_isrForCrosstalkSources_855428_22.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 266b6cda3a99e37e3cfeb1793031bff6 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/00330167-0d1e-4625-ab71-208a92c859b1_isrForCrosstalkSources_855394_53.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/00330167-0d1e-4625-ab71-208a92c859b1_isrForCrosstalkSources_855394_53.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3966093abc4b5b1dd4315144f9980023 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a2481450-1285-4e8c-b221-477bbb741648_isrForCrosstalkSources_855391_7.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a2481450-1285-4e8c-b221-477bbb741648_isrForCrosstalkSources_855391_7.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e493a653b583ca224e07da51a71876c3 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01074f34-8615-48f4-8766-54814148430d_isrForCrosstalkSources_855417_59.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01074f34-8615-48f4-8766-54814148430d_isrForCrosstalkSources_855417_59.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba32d4e236ac3a6fafce71b1c621ac6 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/781b3f27-e102-4c93-a567-bed34f26a5d7_isrForCrosstalkSources_855452_29.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/781b3f27-e102-4c93-a567-bed34f26a5d7_isrForCrosstalkSources_855452_29.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74addeb80a67d540f74f03a2060f1bec with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3006ad57-ef21-45a1-84bf-ba7cd821dd9c_isrForCrosstalkSources_855395_45.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/3006ad57-ef21-45a1-84bf-ba7cd821dd9c_isrForCrosstalkSources_855395_45.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37f1f838748fdee4bdd9a1e083aac9fc with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bb10a081-9383-442a-9fd2-a1a41c0de923_isrForCrosstalkSources_855417_54.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bb10a081-9383-442a-9fd2-a1a41c0de923_isrForCrosstalkSources_855417_54.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854b8e6fef11cf256f088eef4666aabf with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7604c41c-5da8-46b6-a7ce-e0f948aad4d4_isrForCrosstalkSources_855457_43.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7604c41c-5da8-46b6-a7ce-e0f948aad4d4_isrForCrosstalkSources_855457_43.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def1b2ceb2792a4617235027afca62bc with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20df5d3c-f489-487b-ad08-0161a8c4c6e9_isrForCrosstalkSources_855419_60.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20df5d3c-f489-487b-ad08-0161a8c4c6e9_isrForCrosstalkSources_855419_60.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65d758b31169eef7eae4b90fb2a80fc8 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8e24c85e-96c0-40ee-8fb7-48740df78339_isrForCrosstalkSources_855395_4.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8e24c85e-96c0-40ee-8fb7-48740df78339_isrForCrosstalkSources_855395_4.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f389f1bbfd3c979deb5980f1d7ab348 with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a68de22c-4767-4831-b6ae-a4ed3b33492b_isrForCrosstalkSources_855404_49.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a68de22c-4767-4831-b6ae-a4ed3b33492b_isrForCrosstalkSources_855404_49.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ad221a5ea3fe4a1fb4bc181f2c4922 with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/427d8e8c-bf40-474a-8030-e8941803dabd_isrForCrosstalkSources_855452_62.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/427d8e8c-bf40-474a-8030-e8941803dabd_isrForCrosstalkSources_855452_62.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88ca49365e697fff7c3e1e630eeb30de with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/950ce9ad-8c12-4bca-993f-b21ecfb72c3b_isrForCrosstalkSources_855409_37.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/950ce9ad-8c12-4bca-993f-b21ecfb72c3b_isrForCrosstalkSources_855409_37.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 546b49dd1d515e0c88d7178d9f8752f6 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d07b7e89-3fcd-44ee-b8f7-f81eb2e9e2e8_isrForCrosstalkSources_855421_58.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d07b7e89-3fcd-44ee-b8f7-f81eb2e9e2e8_isrForCrosstalkSources_855421_58.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70ec1270e1438b1000dc3077f421a70 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7a8187c5-a690-4a95-a267-df673fb1bc98_isrForCrosstalkSources_855421_16.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7a8187c5-a690-4a95-a267-df673fb1bc98_isrForCrosstalkSources_855421_16.stderr 2024-03-09 14:52:20 proc_lsst.multi:170 [INFO] [multi] job_id 16964258 proc_lsst.multi INFO: [multi] job_id 16964258 2024-03-09 14:52:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-09 14:52:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964258 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964258 parsl.executors.status_handling DEBUG: Launched block 229 on executor multi with job ID 16964258 parsl.executors.status_handling INFO: Allocated block ID 230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-230 2024-03-09 14:52:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a41b2182fe4d40b7311870479e763f8f with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/360c2b89-917e-42c6-8ab2-846ea21cf4a5_isrForCrosstalkSources_855456_34.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/360c2b89-917e-42c6-8ab2-846ea21cf4a5_isrForCrosstalkSources_855456_34.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea5ae940467bcd9d632518d96098ae54 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9cd9c5cd-776b-40c1-a120-1c8ebf960aaf_isrForCrosstalkSources_855432_8.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9cd9c5cd-776b-40c1-a120-1c8ebf960aaf_isrForCrosstalkSources_855432_8.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cc17ef991710e80a0eda45dc7405283 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2b420621-fd7e-4b7d-ac82-45773bf9b9e1_isrForCrosstalkSources_855379_44.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2b420621-fd7e-4b7d-ac82-45773bf9b9e1_isrForCrosstalkSources_855379_44.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e88b4c6f5bb9178aea5cf6bc294ea9ea with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d5d5af3a-b6de-4fdd-af6f-3c21caf316a3_isrForCrosstalkSources_855428_50.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d5d5af3a-b6de-4fdd-af6f-3c21caf316a3_isrForCrosstalkSources_855428_50.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa6621da994a4b9ca02f66bc8922c3e7 with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad78fedc-a8a4-47fe-aee4-c845e3c0d00c_isrForCrosstalkSources_855429_10.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad78fedc-a8a4-47fe-aee4-c845e3c0d00c_isrForCrosstalkSources_855429_10.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 335126924f01fc8ea428812b71b6ab23 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/4ffd7dd9-61f0-4863-851f-7ec9bf5d4ed9_isrForCrosstalkSources_855432_51.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/4ffd7dd9-61f0-4863-851f-7ec9bf5d4ed9_isrForCrosstalkSources_855432_51.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9500c044e741a99d4771ed1ebff15b7c with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/90e60d38-566e-430f-b2ab-d2f74ea27d00_isrForCrosstalkSources_855446_38.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/90e60d38-566e-430f-b2ab-d2f74ea27d00_isrForCrosstalkSources_855446_38.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aad0418e1837f4ff1cf3d4b8c1aa174 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8b944c93-4702-4153-a036-f3bd14db8838_isrForCrosstalkSources_855430_26.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8b944c93-4702-4153-a036-f3bd14db8838_isrForCrosstalkSources_855430_26.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1d670f55b9dde4200279d8e9d2c4cbf with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/93bbe6da-3f4a-4daf-9dd2-1685a9fea4d1_isrForCrosstalkSources_855430_27.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/93bbe6da-3f4a-4daf-9dd2-1685a9fea4d1_isrForCrosstalkSources_855430_27.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4bbc7c2fe1fda1132699ff62160402 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/6dc97f8f-c8c8-43a0-8782-826d1d1373b6_isrForCrosstalkSources_855409_39.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/6dc97f8f-c8c8-43a0-8782-826d1d1373b6_isrForCrosstalkSources_855409_39.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7bf03b082448e3864d46e866b48b15f with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cba4755c-d239-48b3-9d23-6907616fcc74_isrForCrosstalkSources_855446_26.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cba4755c-d239-48b3-9d23-6907616fcc74_isrForCrosstalkSources_855446_26.stderr 2024-03-09 14:52:20 proc_lsst.multi:170 [INFO] [multi] job_id 16964259 proc_lsst.multi INFO: [multi] job_id 16964259 2024-03-09 14:52:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-09 14:52:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964259 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964259 parsl.executors.status_handling DEBUG: Launched block 230 on executor multi with job ID 16964259 parsl.executors.status_handling INFO: Allocated block ID 231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-231 2024-03-09 14:52:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7164d31499e5957892b1e1c2d08a8e03 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c4639f4b-85f7-4ff8-bbae-dd18a2f38bf9_isrForCrosstalkSources_855405_9.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c4639f4b-85f7-4ff8-bbae-dd18a2f38bf9_isrForCrosstalkSources_855405_9.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d89595c5b8702bdd6da07ec78c7e3db with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a24c86d2-38bf-45dc-a73e-c3299bebe355_isrForCrosstalkSources_855394_26.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a24c86d2-38bf-45dc-a73e-c3299bebe355_isrForCrosstalkSources_855394_26.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd03808e885ba3500b50648841ba45ef with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/685d9ab0-5608-4e25-a0fe-3c1d5cfca3ed_isrForCrosstalkSources_855391_6.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/685d9ab0-5608-4e25-a0fe-3c1d5cfca3ed_isrForCrosstalkSources_855391_6.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13f14a627b64c978aa7fe8c4320a018b with result from task 481 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/28c04c8a-6c35-4bab-b959-f7685fae2e82_isrForCrosstalkSources_855406_49.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/28c04c8a-6c35-4bab-b959-f7685fae2e82_isrForCrosstalkSources_855406_49.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59746ea1850084325ed99a3b2d2ad8fd with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8530d9c9-953c-4f00-8bca-74afb74e13a6_isrForCrosstalkSources_855440_56.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8530d9c9-953c-4f00-8bca-74afb74e13a6_isrForCrosstalkSources_855440_56.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d418952514eb3cde4ee5675be65a146 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/214e181d-dfaa-4a17-9e20-bc96065ebb1e_isrForCrosstalkSources_855416_53.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/214e181d-dfaa-4a17-9e20-bc96065ebb1e_isrForCrosstalkSources_855416_53.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e928dc1f1bb0a6fef49dc83b543700e0 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/1a2d0d85-ba4b-4452-9767-2188496d694d_isrForCrosstalkSources_855396_51.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/1a2d0d85-ba4b-4452-9767-2188496d694d_isrForCrosstalkSources_855396_51.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f80f28f7c6b11a9e1303544e63bbdb07 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/aff5adc8-48ae-479f-a817-c2e6652ece98_isrForCrosstalkSources_855393_24.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/aff5adc8-48ae-479f-a817-c2e6652ece98_isrForCrosstalkSources_855393_24.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 253a21a8e019ae889783fbab47bec605 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c8b7f14b-5c25-4ebc-b9e1-4b445fe295e0_isrForCrosstalkSources_855425_22.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c8b7f14b-5c25-4ebc-b9e1-4b445fe295e0_isrForCrosstalkSources_855425_22.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 412172a91c66bd796710392e651467f4 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a9208d1-10c0-4797-b466-5cf5d68fcee2_isrForCrosstalkSources_855434_1.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a9208d1-10c0-4797-b466-5cf5d68fcee2_isrForCrosstalkSources_855434_1.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32b83ca19d7ac6c514e3e3ab66357323 with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/95f5abde-c49d-4810-97d1-3a11d464d57f_isrForCrosstalkSources_855382_46.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/95f5abde-c49d-4810-97d1-3a11d464d57f_isrForCrosstalkSources_855382_46.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ebfa022cd761c59d224e321ed2b25d7 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/80d26bfc-91a6-4052-9c8e-a23e2d130dfb_isrForCrosstalkSources_855383_2.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/80d26bfc-91a6-4052-9c8e-a23e2d130dfb_isrForCrosstalkSources_855383_2.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac0a28b18a9c1cf734962f28a5762549 with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1d679a65-4529-480c-ae1a-bb05a4a3d4ac_isrForCrosstalkSources_855417_42.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1d679a65-4529-480c-ae1a-bb05a4a3d4ac_isrForCrosstalkSources_855417_42.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5d11e8f2e1625c6ae151bb28cc0f119 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/319379bd-3d64-405b-aeb5-c57f91b48e59_isrForCrosstalkSources_855403_51.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/319379bd-3d64-405b-aeb5-c57f91b48e59_isrForCrosstalkSources_855403_51.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c47e2fbfb9224b4b07e7f3ebc3c392d with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/202ea002-fa94-4728-af39-d8d8ff3b58e6_isrForCrosstalkSources_855415_5.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/202ea002-fa94-4728-af39-d8d8ff3b58e6_isrForCrosstalkSources_855415_5.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72d231f71f65d5f8cee714e620fdd622 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9c356007-d17c-49bb-a2d3-e16a3ecd483e_isrForCrosstalkSources_855396_10.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9c356007-d17c-49bb-a2d3-e16a3ecd483e_isrForCrosstalkSources_855396_10.stderr 2024-03-09 14:52:20 proc_lsst.multi:170 [INFO] [multi] job_id 16964260 proc_lsst.multi INFO: [multi] job_id 16964260 2024-03-09 14:52:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-09 14:52:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964260 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964260 parsl.executors.status_handling DEBUG: Launched block 231 on executor multi with job ID 16964260 parsl.executors.status_handling INFO: Allocated block ID 232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-232 2024-03-09 14:52:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-09 14:52:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9a4fddfd0bd21acadf8a3754e76ca7b with result from task 817 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3b2c964b-cd63-414e-a8b5-8c72e6e08a63_isrForCrosstalkSources_855432_23.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3b2c964b-cd63-414e-a8b5-8c72e6e08a63_isrForCrosstalkSources_855432_23.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cac51e42ee7659764c34c01c6cd043d with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/136de2ca-4868-4af3-bf02-a48ca8633e3c_isrForCrosstalkSources_855436_36.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/136de2ca-4868-4af3-bf02-a48ca8633e3c_isrForCrosstalkSources_855436_36.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b24429508f0cbc47187692b7cd29311 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8cfa0af3-1626-4039-9939-d12b7c7a1416_isrForCrosstalkSources_855426_17.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8cfa0af3-1626-4039-9939-d12b7c7a1416_isrForCrosstalkSources_855426_17.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6398c4f8f9964a0d044885e361818c07 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/27f88db1-561c-4448-8afc-ee4ba79073da_isrForCrosstalkSources_855397_7.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/27f88db1-561c-4448-8afc-ee4ba79073da_isrForCrosstalkSources_855397_7.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd18d4d1d41cde6ae8501dd335807526 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5478fbc9-cfa8-43e4-9007-5eb40e3271c7_isrForCrosstalkSources_855445_24.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5478fbc9-cfa8-43e4-9007-5eb40e3271c7_isrForCrosstalkSources_855445_24.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2e5aceb81806e04fde72332140e6f3d with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d12537f-fac3-4cf3-b385-56712b6a294e_isrForCrosstalkSources_855444_51.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d12537f-fac3-4cf3-b385-56712b6a294e_isrForCrosstalkSources_855444_51.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 550c2342ac0273ce736deca147192289 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8a01e716-1a06-4128-9403-d28a66e28cd4_isrForCrosstalkSources_855431_57.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8a01e716-1a06-4128-9403-d28a66e28cd4_isrForCrosstalkSources_855431_57.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613358720b0b4cc878a9075801de3d32 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b9ad56ad-b601-4580-ad48-cf5e5993e4f3_isrForCrosstalkSources_855393_18.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b9ad56ad-b601-4580-ad48-cf5e5993e4f3_isrForCrosstalkSources_855393_18.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec9f7b35f97a2e9fe08fc4e309046da5 with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cb429303-f6d3-44e3-82a7-f7fe08d5ef85_isrForCrosstalkSources_855458_44.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cb429303-f6d3-44e3-82a7-f7fe08d5ef85_isrForCrosstalkSources_855458_44.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 066bc57b7ce8914ad76a9ac9f1f9dbd5 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7ec8fc48-cfb5-45a0-9d66-9a78777d6004_isrForCrosstalkSources_855382_37.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7ec8fc48-cfb5-45a0-9d66-9a78777d6004_isrForCrosstalkSources_855382_37.stderr 2024-03-09 14:52:21 proc_lsst.multi:170 [INFO] [multi] job_id 16964261 proc_lsst.multi INFO: [multi] job_id 16964261 2024-03-09 14:52:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-09 14:52:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964261 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964261 parsl.executors.status_handling DEBUG: Launched block 232 on executor multi with job ID 16964261 parsl.executors.status_handling INFO: Allocated block ID 233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-233 2024-03-09 14:52:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 2024-03-09 14:52:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-09 14:52:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96bcad031dac6c03ff3203ff3fa75150 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9db72332-4b46-4b3c-95f4-1807645132fe_isrForCrosstalkSources_855452_16.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9db72332-4b46-4b3c-95f4-1807645132fe_isrForCrosstalkSources_855452_16.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71bf1f6cecc18742a8566f49bc814c26 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1fc73d4b-e997-4cf8-8353-75275602aa4e_isrForCrosstalkSources_855388_35.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1fc73d4b-e997-4cf8-8353-75275602aa4e_isrForCrosstalkSources_855388_35.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a2e0bc0b7235ed30c2cb32842dc7c60 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c2930f49-0bfc-4e67-a3c5-6fcc14eb9fa1_isrForCrosstalkSources_855402_27.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c2930f49-0bfc-4e67-a3c5-6fcc14eb9fa1_isrForCrosstalkSources_855402_27.stderr parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46c33b159d04989558b83cf5022510f4 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a879df5c-27c5-4512-ac54-b52bffadec70_isrForCrosstalkSources_855449_30.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a879df5c-27c5-4512-ac54-b52bffadec70_isrForCrosstalkSources_855449_30.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1cf3b99872ca6e2f8c0331950cc3561 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60e80b68-faf3-43ff-9929-58e7a33dea5e_isrForCrosstalkSources_855433_60.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60e80b68-faf3-43ff-9929-58e7a33dea5e_isrForCrosstalkSources_855433_60.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6f68b3ef3d9103df948d4191b552eaf with result from task 408 2024-03-09 14:52:22 proc_lsst.multi:170 [INFO] [multi] job_id 16964262 proc_lsst.multi INFO: [multi] job_id 16964262 2024-03-09 14:52:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-09 14:52:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964262 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964262 parsl.executors.status_handling DEBUG: Launched block 233 on executor multi with job ID 16964262 parsl.executors.status_handling INFO: Allocated block ID 234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-234 2024-03-09 14:52:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6cfd199c-b9e1-4e7a-9173-d20b2e001614_isrForCrosstalkSources_855411_34.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6cfd199c-b9e1-4e7a-9173-d20b2e001614_isrForCrosstalkSources_855411_34.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295fd91dbec263e9a577b45316e3b576 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/8d3483b7-e813-4abd-b7b2-2399221eb3fd_isrForCrosstalkSources_855439_46.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/8d3483b7-e813-4abd-b7b2-2399221eb3fd_isrForCrosstalkSources_855439_46.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956a4a6e79bb4322bad50889ef855bee with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b9bfb513-e48e-4899-ac26-0cf55d94c58c_isrForCrosstalkSources_855380_1.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b9bfb513-e48e-4899-ac26-0cf55d94c58c_isrForCrosstalkSources_855380_1.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0235b524b9a9f66004faea04c286a076 with result from task 425 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c2e77f37-8ed9-4bd6-aada-67958e6c2ba3_isrForCrosstalkSources_855450_2.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c2e77f37-8ed9-4bd6-aada-67958e6c2ba3_isrForCrosstalkSources_855450_2.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 216c8ed93c8ff9f8bb3329e9f4058f05 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a9c208ea-3661-43ef-9518-26359e58ea7b_isrForCrosstalkSources_855383_6.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a9c208ea-3661-43ef-9518-26359e58ea7b_isrForCrosstalkSources_855383_6.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c61a920a5df611ac7296b2646e0cd5e with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6ca531a5-1d9f-4abd-aefc-20c0aaf833f6_isrForCrosstalkSources_855457_16.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6ca531a5-1d9f-4abd-aefc-20c0aaf833f6_isrForCrosstalkSources_855457_16.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea1f1c10b1b74dbfe9dd506613a9d87 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1606d311-11b8-4f0b-bae1-c20bb531121b_isrForCrosstalkSources_855407_55.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1606d311-11b8-4f0b-bae1-c20bb531121b_isrForCrosstalkSources_855407_55.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5938c2e5eaaf2ab2c99ffd4217e6056e with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/af73dbac-083e-4170-bdf5-5647681f7ec3_isrForCrosstalkSources_855413_16.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/af73dbac-083e-4170-bdf5-5647681f7ec3_isrForCrosstalkSources_855413_16.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05b5f80c5ab7c8e6b12bb2f740d403fb with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dbf9db19-28a5-4114-9f2f-b1422a880327_isrForCrosstalkSources_855405_22.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dbf9db19-28a5-4114-9f2f-b1422a880327_isrForCrosstalkSources_855405_22.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa5c2a677872e3cd9102f6e6a24da6a with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/50d2edfb-9467-4c99-9bae-31907593a724_isrForCrosstalkSources_855455_51.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/50d2edfb-9467-4c99-9bae-31907593a724_isrForCrosstalkSources_855455_51.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd5c2bb355e6e2f350ce1465d23cd6d2 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/585f568c-a3d4-400f-a559-fefa753b89c1_isrForCrosstalkSources_855433_61.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/585f568c-a3d4-400f-a559-fefa753b89c1_isrForCrosstalkSources_855433_61.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b3782971d35bb5b83c4ebdd87fdaf8 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14a697f5-913b-495f-b5ca-1eb9f6413efe_isrForCrosstalkSources_855433_58.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14a697f5-913b-495f-b5ca-1eb9f6413efe_isrForCrosstalkSources_855433_58.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6885549b9a65d0c028fdbff38106a855 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d49098ac-6ddd-4425-8006-c9c029be6ab8_isrForCrosstalkSources_855426_12.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d49098ac-6ddd-4425-8006-c9c029be6ab8_isrForCrosstalkSources_855426_12.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 801838fe3d72759a6b149cd55cc6db2d with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/90e8385f-be2d-4b98-8779-add06850af47_isrForCrosstalkSources_855383_57.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/90e8385f-be2d-4b98-8779-add06850af47_isrForCrosstalkSources_855383_57.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 903d3d45e838e41345ee35822365948d with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/23ec2d2a-bf41-488c-b94d-0e1e5d7d47ac_isrForCrosstalkSources_855386_23.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/23ec2d2a-bf41-488c-b94d-0e1e5d7d47ac_isrForCrosstalkSources_855386_23.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f44942cabe7df105d2a9058a504e0c4 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0528d0ea-7381-4f0f-87b1-60e91c93139f_isrForCrosstalkSources_855386_11.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0528d0ea-7381-4f0f-87b1-60e91c93139f_isrForCrosstalkSources_855386_11.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a6d0dd8eef7d94f3e29cc64f1e0906e with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36fdc32f-2b20-4315-a9d3-f79158c1f342_isrForCrosstalkSources_855388_41.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36fdc32f-2b20-4315-a9d3-f79158c1f342_isrForCrosstalkSources_855388_41.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d9ded3fdb18b33f145c4a0e4da371e with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3a85ab9-8185-4f36-9f68-bdadc56e1225_isrForCrosstalkSources_855395_12.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c3a85ab9-8185-4f36-9f68-bdadc56e1225_isrForCrosstalkSources_855395_12.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792b22cf5bd305c7d60c0c20fbcac193 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3488ac9c-8425-4925-93db-80f8eea4c0da_isrForCrosstalkSources_855448_3.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3488ac9c-8425-4925-93db-80f8eea4c0da_isrForCrosstalkSources_855448_3.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 694bbec11854bd402520e056cde601d6 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/70b6fe76-eec1-4692-b438-94cfbe1f577a_isrForCrosstalkSources_855449_15.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/70b6fe76-eec1-4692-b438-94cfbe1f577a_isrForCrosstalkSources_855449_15.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2855794e3bcc0817f377bee6c480c61 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4bcfc7aa-7bcc-419f-8fa2-5b8ded007f05_isrForCrosstalkSources_855425_49.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4bcfc7aa-7bcc-419f-8fa2-5b8ded007f05_isrForCrosstalkSources_855425_49.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc307c2facf84d0e7d65962d25fe585 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/bbe51069-a6bd-4e23-9f72-b0bb6f08a173_isrForCrosstalkSources_855402_33.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/bbe51069-a6bd-4e23-9f72-b0bb6f08a173_isrForCrosstalkSources_855402_33.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c581c64af9cb2caf7432e30c43f3ec67 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/66c66b28-7048-4238-85c9-c0baa0cdf845_isrForCrosstalkSources_855456_4.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/66c66b28-7048-4238-85c9-c0baa0cdf845_isrForCrosstalkSources_855456_4.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3732adb2301a0b7adfc32ff6b89e5ade with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e0ef2bb4-a91c-4924-9e3d-95af47d8cc83_isrForCrosstalkSources_855421_38.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e0ef2bb4-a91c-4924-9e3d-95af47d8cc83_isrForCrosstalkSources_855421_38.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4793af67401dbdf2efcafbff1d22af50 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9240df80-4da9-45b3-b96a-e32a71c4de51_isrForCrosstalkSources_855410_24.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9240df80-4da9-45b3-b96a-e32a71c4de51_isrForCrosstalkSources_855410_24.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63bd53e7bec34e44fcb3012f0e50efc2 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67fe03fa-68dd-46bf-9c17-4ee4d6505615_isrForCrosstalkSources_855382_51.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67fe03fa-68dd-46bf-9c17-4ee4d6505615_isrForCrosstalkSources_855382_51.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a82710866fd95f0d6e36982d62d4bdf3 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/95f65656-957b-4103-8055-64be67db81d7_isrForCrosstalkSources_855384_16.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/95f65656-957b-4103-8055-64be67db81d7_isrForCrosstalkSources_855384_16.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ded540258b6d684fef06f09caf08c2 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/db7934fb-bade-443b-8f4c-3401b8186833_isrForCrosstalkSources_855410_36.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/db7934fb-bade-443b-8f4c-3401b8186833_isrForCrosstalkSources_855410_36.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be87c50eafe29d0ce0227b1ed0e84528 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5e299dc3-92c1-4c50-9aa1-04d6bbcba966_isrForCrosstalkSources_855395_58.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5e299dc3-92c1-4c50-9aa1-04d6bbcba966_isrForCrosstalkSources_855395_58.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f932ee08b87c8f33f1160e8ec1562b with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8a711335-405e-46b4-8bcd-c1e36dd1d919_isrForCrosstalkSources_855381_6.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8a711335-405e-46b4-8bcd-c1e36dd1d919_isrForCrosstalkSources_855381_6.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe94bdd9c25887514ae73180f8c3696 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c00f8ee9-1b28-40fe-904d-26d159387e1c_isrForCrosstalkSources_855388_18.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c00f8ee9-1b28-40fe-904d-26d159387e1c_isrForCrosstalkSources_855388_18.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95be5ff5adbb75503f00704bf06ca129 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32e8ca45-f9dd-4785-913d-893e6b1e492e_isrForCrosstalkSources_855447_16.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32e8ca45-f9dd-4785-913d-893e6b1e492e_isrForCrosstalkSources_855447_16.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dea5290d03ab6dbd2bb5d9064eb05be with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9f7b8-7f72-49e7-a17c-679505b035ed_isrForCrosstalkSources_855429_41.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9f7b8-7f72-49e7-a17c-679505b035ed_isrForCrosstalkSources_855429_41.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 007bed86f82a7baedd011f8afc5f78e0 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/397ef0e1-4a1a-46c2-9f91-ff520b284f39_isrForCrosstalkSources_855419_28.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/397ef0e1-4a1a-46c2-9f91-ff520b284f39_isrForCrosstalkSources_855419_28.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ebb9ebdb8cddfbbe9f7ae370b83241e with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/9554fd02-8269-484d-bb25-4d92e11143ef_isrForCrosstalkSources_855429_17.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/9554fd02-8269-484d-bb25-4d92e11143ef_isrForCrosstalkSources_855429_17.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e68bebeebad790e962cf580e941d985 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/431e4453-cbf6-460e-ae20-c8bdf45ed5f5_isrForCrosstalkSources_855459_17.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/431e4453-cbf6-460e-ae20-c8bdf45ed5f5_isrForCrosstalkSources_855459_17.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15d08d5eae9b76953df171b95f6d9e2e with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/6b0493e2-d321-4233-8912-29c63a81f39d_isrForCrosstalkSources_855441_22.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/6b0493e2-d321-4233-8912-29c63a81f39d_isrForCrosstalkSources_855441_22.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7419a599081981e4536153962a538e47 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8708523-3410-4545-a5d5-e725424f5b17_isrForCrosstalkSources_855412_53.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c8708523-3410-4545-a5d5-e725424f5b17_isrForCrosstalkSources_855412_53.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63082eb999790c02492c082192dad41d with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/361258bc-5db2-425a-8fa4-fd639b47f38a_isrForCrosstalkSources_855430_32.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/361258bc-5db2-425a-8fa4-fd639b47f38a_isrForCrosstalkSources_855430_32.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7ede6bd9f84631d3196be66d7c67c95 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/11299445-a97b-49bd-aeed-49571f17b89e_isrForCrosstalkSources_855450_16.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/11299445-a97b-49bd-aeed-49571f17b89e_isrForCrosstalkSources_855450_16.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15fa15e9f1ea1e10173063be0011f1d2 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c66a3270-5685-4cdd-a439-0b0e2eaf1f53_isrForCrosstalkSources_855436_27.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/c66a3270-5685-4cdd-a439-0b0e2eaf1f53_isrForCrosstalkSources_855436_27.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 192362c3a9dd6f5e006dee4a88786120 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be2b72bb-93e9-4e2e-9da6-99bf342594e5_isrForCrosstalkSources_855425_16.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be2b72bb-93e9-4e2e-9da6-99bf342594e5_isrForCrosstalkSources_855425_16.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8506a5bf9ae1bb323a6972f21674c60f with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/05c6f1e7-3bd7-43e1-a9ef-dea1defbe65d_isrForCrosstalkSources_855421_53.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/05c6f1e7-3bd7-43e1-a9ef-dea1defbe65d_isrForCrosstalkSources_855421_53.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 077e16f5e3c8168168c45faf06bb7af1 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4069fa53-eabd-4c94-ba0c-1cbb5c2fa0a2_isrForCrosstalkSources_855405_46.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/4069fa53-eabd-4c94-ba0c-1cbb5c2fa0a2_isrForCrosstalkSources_855405_46.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6cbd2ac20bf49e63fbdfe05ad95eacb with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c06ad283-a872-41cc-a3bb-645aac23f7b6_isrForCrosstalkSources_855380_22.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c06ad283-a872-41cc-a3bb-645aac23f7b6_isrForCrosstalkSources_855380_22.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1c817d95a52e5a7388d595280facd8c with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80431459-e8c9-4acc-aee6-828824343d85_isrForCrosstalkSources_855426_54.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80431459-e8c9-4acc-aee6-828824343d85_isrForCrosstalkSources_855426_54.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2ffc6b4b78466e1bc28eeaf66d4a871 with result from task 695 2024-03-09 14:52:22 proc_lsst.multi:170 [INFO] [multi] job_id 16964263 proc_lsst.multi INFO: [multi] job_id 16964263 2024-03-09 14:52:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-09 14:52:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964263 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964263 parsl.executors.status_handling DEBUG: Launched block 234 on executor multi with job ID 16964263 parsl.executors.status_handling INFO: Allocated block ID 235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-235 2024-03-09 14:52:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-09 14:52:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e7d54203-58fb-4964-a48e-109cdc21460f_isrForCrosstalkSources_855398_5.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e7d54203-58fb-4964-a48e-109cdc21460f_isrForCrosstalkSources_855398_5.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17e2b858bcb91dd6faa0981067c46595 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f2c43b46-8d8b-4868-b57b-a1ec29164d77_isrForCrosstalkSources_855417_5.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f2c43b46-8d8b-4868-b57b-a1ec29164d77_isrForCrosstalkSources_855417_5.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9c45030ad034cd77849434b6780611d with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/762e2699-d365-4709-8768-38d15842d89c_isrForCrosstalkSources_855392_23.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/762e2699-d365-4709-8768-38d15842d89c_isrForCrosstalkSources_855392_23.stderr 2024-03-09 14:52:23 proc_lsst.multi:170 [INFO] [multi] job_id 16964264 proc_lsst.multi INFO: [multi] job_id 16964264 2024-03-09 14:52:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-09 14:52:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964264 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964264 parsl.executors.status_handling DEBUG: Launched block 235 on executor multi with job ID 16964264 parsl.executors.status_handling INFO: Allocated block ID 236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-236 2024-03-09 14:52:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 632dae238f47efeca70956d7a3fc74c6 with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/daf1bf58-422c-42ec-a4cf-2fe4ef2c801e_isrForCrosstalkSources_855421_23.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/daf1bf58-422c-42ec-a4cf-2fe4ef2c801e_isrForCrosstalkSources_855421_23.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6270e987da110feada2fbb31dfdb05e9 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/734f7c68-4483-4c19-9879-c3a37cb1321a_isrForCrosstalkSources_855381_30.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/734f7c68-4483-4c19-9879-c3a37cb1321a_isrForCrosstalkSources_855381_30.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c49c85db1d073a6ad28ef13002949cd2 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bf2b8cbc-a619-4aa1-a242-8e160c8f36d2_isrForCrosstalkSources_855405_58.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bf2b8cbc-a619-4aa1-a242-8e160c8f36d2_isrForCrosstalkSources_855405_58.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb2e4f713bf804d7e0bcbc81a9db36d9 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/abe38278-5b9d-428a-b046-10e9ff7cfe1c_isrForCrosstalkSources_855431_13.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/abe38278-5b9d-428a-b046-10e9ff7cfe1c_isrForCrosstalkSources_855431_13.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aad5ef06313d297f04545d553ae5c882 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/caad7119-da14-43df-8780-4636e0a3efb6_isrForCrosstalkSources_855422_3.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/caad7119-da14-43df-8780-4636e0a3efb6_isrForCrosstalkSources_855422_3.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac16e74e831458b11cc7d1bb499c39a2 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/772d07ba-9301-4545-9f3d-8424ef4f5b09_isrForCrosstalkSources_855451_58.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/772d07ba-9301-4545-9f3d-8424ef4f5b09_isrForCrosstalkSources_855451_58.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba05976ed6f56a9b8bb506a7ca2eec9 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3a596a89-33af-4acf-827c-1a96ccc938a5_isrForCrosstalkSources_855416_35.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3a596a89-33af-4acf-827c-1a96ccc938a5_isrForCrosstalkSources_855416_35.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a56d47cf554ebf0f27388bcd44149b6 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/9eab1ec7-0a41-44ec-a161-ba6a981662df_isrForCrosstalkSources_855442_61.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/9eab1ec7-0a41-44ec-a161-ba6a981662df_isrForCrosstalkSources_855442_61.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1245875b14f94908849b5f96ecbaa1 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e41b34b0-7ebf-48c8-903b-a0ada95ac75c_isrForCrosstalkSources_855431_31.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e41b34b0-7ebf-48c8-903b-a0ada95ac75c_isrForCrosstalkSources_855431_31.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a886434f52a6e9485ca5a22275b6229 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6705a3b5-ade9-4ed4-97d6-9d4c714997b8_isrForCrosstalkSources_855429_58.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6705a3b5-ade9-4ed4-97d6-9d4c714997b8_isrForCrosstalkSources_855429_58.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c62e0f003406000842581055953f9c6 with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0076bd41-c4a1-4ce5-bbd3-e112beed301d_isrForCrosstalkSources_855430_62.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0076bd41-c4a1-4ce5-bbd3-e112beed301d_isrForCrosstalkSources_855430_62.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79d010714ebd8988ca1453e2503196a8 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b6a923a3-7ee6-48b5-8ae2-3eb6259b4d4a_isrForCrosstalkSources_855380_30.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/b6a923a3-7ee6-48b5-8ae2-3eb6259b4d4a_isrForCrosstalkSources_855380_30.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0450b7c9b23c221f07c7764e95f7a530 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0b8e6bfc-8acd-4052-bc17-920a7c9d8da6_isrForCrosstalkSources_855438_47.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0b8e6bfc-8acd-4052-bc17-920a7c9d8da6_isrForCrosstalkSources_855438_47.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2a8fcae35182cb716afb85a9a7b641 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3e79a0f8-e824-44b2-9201-38bd2ed6ef31_isrForCrosstalkSources_855387_60.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/3e79a0f8-e824-44b2-9201-38bd2ed6ef31_isrForCrosstalkSources_855387_60.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ff3f3a328e5717e8ac2b7059777cc84 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7b301cb5-60e9-4bd6-b7ce-f8b6e4b53265_isrForCrosstalkSources_855458_52.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7b301cb5-60e9-4bd6-b7ce-f8b6e4b53265_isrForCrosstalkSources_855458_52.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dfa2b31ba22c650cc81d8315b7f61d4 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3e6618a-db6d-4deb-8db3-49b25607a382_isrForCrosstalkSources_855431_35.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3e6618a-db6d-4deb-8db3-49b25607a382_isrForCrosstalkSources_855431_35.stderr 2024-03-09 14:52:23 proc_lsst.multi:170 [INFO] [multi] job_id 16964265 proc_lsst.multi INFO: [multi] job_id 16964265 2024-03-09 14:52:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-09 14:52:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964265 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964265 parsl.executors.status_handling DEBUG: Launched block 236 on executor multi with job ID 16964265 parsl.executors.status_handling INFO: Allocated block ID 237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-237 2024-03-09 14:52:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-09 14:52:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43c7cf3abcce388bb4b0a415b0dcb7b1 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e94ced9-3551-4f6e-bb6f-a0c0a8139714_isrForCrosstalkSources_855423_18.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e94ced9-3551-4f6e-bb6f-a0c0a8139714_isrForCrosstalkSources_855423_18.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8984fa2be9ed633eca4f216fa60de0d with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/71ced3a8-ca9d-4061-827f-787840fd0dbf_isrForCrosstalkSources_855412_31.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/71ced3a8-ca9d-4061-827f-787840fd0dbf_isrForCrosstalkSources_855412_31.stderr 2024-03-09 14:52:24 proc_lsst.multi:170 [INFO] [multi] job_id 16964266 proc_lsst.multi INFO: [multi] job_id 16964266 2024-03-09 14:52:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-09 14:52:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964266 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964266 parsl.executors.status_handling DEBUG: Launched block 237 on executor multi with job ID 16964266 parsl.executors.status_handling INFO: Allocated block ID 238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-238 2024-03-09 14:52:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb72d71a51a0c1cd9b37beae8991852 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccf3d496-b326-4382-8a5f-b9f1189ff645_isrForCrosstalkSources_855403_39.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccf3d496-b326-4382-8a5f-b9f1189ff645_isrForCrosstalkSources_855403_39.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf21f9676e89fcb94d230f2c8f6ff58a with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9931004-9fbb-4740-a5a0-c30a217adce9_isrForCrosstalkSources_855449_37.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9931004-9fbb-4740-a5a0-c30a217adce9_isrForCrosstalkSources_855449_37.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f27a57280d1c9ae0733a428f8d0872c7 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/adf05dae-16a4-4258-8731-68d7dffa7f9b_isrForCrosstalkSources_855426_14.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/adf05dae-16a4-4258-8731-68d7dffa7f9b_isrForCrosstalkSources_855426_14.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 410c84dc341deea920df501f7c2a73d9 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/7dd1aaf8-917c-45a4-99fc-7ad9a5d563d5_isrForCrosstalkSources_855427_51.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/7dd1aaf8-917c-45a4-99fc-7ad9a5d563d5_isrForCrosstalkSources_855427_51.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2a155d7cdd69b8cae6894a0eb92b0d with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/d10c3232-d338-4559-9666-5ba60bbf65c0_isrForCrosstalkSources_855440_35.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/d10c3232-d338-4559-9666-5ba60bbf65c0_isrForCrosstalkSources_855440_35.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 274ceb8ca21f3a6d8ab9d058abc0ceb8 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/582bc923-4781-46a6-9ea1-92163c4cd6cf_isrForCrosstalkSources_855390_42.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/582bc923-4781-46a6-9ea1-92163c4cd6cf_isrForCrosstalkSources_855390_42.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1880ce00599873654e17d4efd67b33e1 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f2af03ba-9b3b-4b3f-a4aa-6fed0a753705_isrForCrosstalkSources_855407_53.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f2af03ba-9b3b-4b3f-a4aa-6fed0a753705_isrForCrosstalkSources_855407_53.stderr 2024-03-09 14:52:24 proc_lsst.multi:170 [INFO] [multi] job_id 16964267 proc_lsst.multi INFO: [multi] job_id 16964267 2024-03-09 14:52:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-09 14:52:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964267 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964267 parsl.executors.status_handling DEBUG: Launched block 238 on executor multi with job ID 16964267 parsl.executors.status_handling INFO: Allocated block ID 239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-239 2024-03-09 14:52:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-09 14:52:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e26ea0b61729a358279c838bbdf2f9d with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/39ff4820-2423-47cd-b0aa-3307d99e53c8_isrForCrosstalkSources_855423_37.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/39ff4820-2423-47cd-b0aa-3307d99e53c8_isrForCrosstalkSources_855423_37.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39670e679680b40ac79b94ba4be36881 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8b006288-6030-441f-93dc-b930cbb9ed62_isrForCrosstalkSources_855407_57.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8b006288-6030-441f-93dc-b930cbb9ed62_isrForCrosstalkSources_855407_57.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b785110913c5ae3e771096ae2b69eed with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/acb75461-03b8-4ec4-9106-6c2361159adf_isrForCrosstalkSources_855437_37.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/acb75461-03b8-4ec4-9106-6c2361159adf_isrForCrosstalkSources_855437_37.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86c9b10175f18b531a22fd3e043ccc23 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/353db4d8-4054-46e7-8426-5079152559b9_isrForCrosstalkSources_855433_7.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/353db4d8-4054-46e7-8426-5079152559b9_isrForCrosstalkSources_855433_7.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57490dd68fc9244186d85495ac97b2a1 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/2da167f2-8cf4-494a-81d4-2d4b1e5e480b_isrForCrosstalkSources_855394_40.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/2da167f2-8cf4-494a-81d4-2d4b1e5e480b_isrForCrosstalkSources_855394_40.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab7c0bc34a6cc3aa252c89242a466d5 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ebb7d6ee-a17d-4270-8beb-e421191801d6_isrForCrosstalkSources_855432_39.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ebb7d6ee-a17d-4270-8beb-e421191801d6_isrForCrosstalkSources_855432_39.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1542eec74f72813579e1023f8cc79099 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3cc3cbda-b54f-44a9-b84c-ff5146aa6cef_isrForCrosstalkSources_855383_1.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3cc3cbda-b54f-44a9-b84c-ff5146aa6cef_isrForCrosstalkSources_855383_1.stderr 2024-03-09 14:52:25 proc_lsst.multi:170 [INFO] [multi] job_id 16964268 proc_lsst.multi INFO: [multi] job_id 16964268 2024-03-09 14:52:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-09 14:52:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964268 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964268 parsl.executors.status_handling DEBUG: Launched block 239 on executor multi with job ID 16964268 parsl.executors.status_handling INFO: Allocated block ID 240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-240 2024-03-09 14:52:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cec1fc323fffdc1c296c60d8cc8919d with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f0463a44-1e7e-4711-ab5c-3eae6e4cd93a_isrForCrosstalkSources_855459_31.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f0463a44-1e7e-4711-ab5c-3eae6e4cd93a_isrForCrosstalkSources_855459_31.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f40cb051793a88fa966f6c45df5f09e with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b1bef419-9ec6-4363-833b-95c66e6478b1_isrForCrosstalkSources_855440_28.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b1bef419-9ec6-4363-833b-95c66e6478b1_isrForCrosstalkSources_855440_28.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 025f4af5bd70ad9f69d2a7b49be22c07 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0a93ab36-d53c-44b2-9c74-28e7b0586aa4_isrForCrosstalkSources_855380_43.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0a93ab36-d53c-44b2-9c74-28e7b0586aa4_isrForCrosstalkSources_855380_43.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8b4f01d9c120696c960afc0361f50d2 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/23792329-2a57-459e-a538-3851434165e3_isrForCrosstalkSources_855418_23.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/23792329-2a57-459e-a538-3851434165e3_isrForCrosstalkSources_855418_23.stderr 2024-03-09 14:52:25 proc_lsst.multi:170 [INFO] [multi] job_id 16964269 proc_lsst.multi INFO: [multi] job_id 16964269 2024-03-09 14:52:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-09 14:52:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964269 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964269 parsl.executors.status_handling DEBUG: Launched block 240 on executor multi with job ID 16964269 parsl.executors.status_handling INFO: Allocated block ID 241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-241 2024-03-09 14:52:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-09 14:52:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fef3cae623976360c4e8a47e660c9a91 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1ce5ecd6-8095-4746-8f99-2a80c88fd89e_isrForCrosstalkSources_855437_18.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1ce5ecd6-8095-4746-8f99-2a80c88fd89e_isrForCrosstalkSources_855437_18.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b362238dab5180d75382689b0b799821 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/235fb2d0-d60b-4986-82c8-b39f485a44cb_isrForCrosstalkSources_855447_57.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/235fb2d0-d60b-4986-82c8-b39f485a44cb_isrForCrosstalkSources_855447_57.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 241dc42a47c944c8a8f14580040fbc1b with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8212d46e-f162-473e-a728-11a31b6e545c_isrForCrosstalkSources_855447_59.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8212d46e-f162-473e-a728-11a31b6e545c_isrForCrosstalkSources_855447_59.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2dbd552593b825b5d1df890799be66f with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b4e4f868-3b79-4aa9-968e-e7329ef13d2f_isrForCrosstalkSources_855447_18.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/b4e4f868-3b79-4aa9-968e-e7329ef13d2f_isrForCrosstalkSources_855447_18.stderr parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8bfac9268e5191f3f61bb2fd40c4f6a with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/86be32e1-3057-4ef3-a2b2-1a5fe7d59be3_isrForCrosstalkSources_855404_25.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/86be32e1-3057-4ef3-a2b2-1a5fe7d59be3_isrForCrosstalkSources_855404_25.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd123ba388b9de74d5cf773ffea028db with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f57e8385-6449-49b7-9745-5330bbd2c4bb_isrForCrosstalkSources_855383_31.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f57e8385-6449-49b7-9745-5330bbd2c4bb_isrForCrosstalkSources_855383_31.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 551d99d8a9cf806891c1a0af544322d4 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/45d63b97-0953-465e-a4aa-0e0f9b84df9f_isrForCrosstalkSources_855432_15.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/45d63b97-0953-465e-a4aa-0e0f9b84df9f_isrForCrosstalkSources_855432_15.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f8b1b23ee29a9d9017cfbad82ac95a3 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d45f891f-5ba8-435e-a0ec-aba327aa74fe_isrForCrosstalkSources_855388_60.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d45f891f-5ba8-435e-a0ec-aba327aa74fe_isrForCrosstalkSources_855388_60.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b2b92b0bca94ae678ce92b4d041cdc with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e9b49380-f465-49ef-82c0-0f2bb8edbe62_isrForCrosstalkSources_855391_62.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e9b49380-f465-49ef-82c0-0f2bb8edbe62_isrForCrosstalkSources_855391_62.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ed477bc0a41f48b9410fc25834bc55 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/782f1aed-7754-44e3-9396-987b3e9feae9_isrForCrosstalkSources_855408_49.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/782f1aed-7754-44e3-9396-987b3e9feae9_isrForCrosstalkSources_855408_49.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8571a2b0db74bc3d3ab0dba2856a5290 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8e49db95-240a-4685-a940-6f20eeb3dc9b_isrForCrosstalkSources_855384_28.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8e49db95-240a-4685-a940-6f20eeb3dc9b_isrForCrosstalkSources_855384_28.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dec22420f1920e9ecf55a9ca1493fcc with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/84bad68c-fc94-4562-ae85-865a48b3677c_isrForCrosstalkSources_855431_1.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/84bad68c-fc94-4562-ae85-865a48b3677c_isrForCrosstalkSources_855431_1.stderr 2024-03-09 14:52:26 proc_lsst.multi:170 [INFO] [multi] job_id 16964270 proc_lsst.multi INFO: [multi] job_id 16964270 2024-03-09 14:52:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 186 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 186 2024-03-09 14:52:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964270 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964270 parsl.executors.status_handling DEBUG: Launched block 241 on executor multi with job ID 16964270 parsl.executors.status_handling INFO: Allocated block ID 242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-242 2024-03-09 14:52:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 2024-03-09 14:52:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 186 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 186 2024-03-09 14:52:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5046bd768f4c85d1dd79db60f403ce4 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e9da9976-6214-4a2f-9c65-2b83e0dc3f1d_isrForCrosstalkSources_855441_29.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e9da9976-6214-4a2f-9c65-2b83e0dc3f1d_isrForCrosstalkSources_855441_29.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3f850d65714701e3404e2d651202a7 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/38bd0c46-f376-43f2-ba18-a464615af381_isrForCrosstalkSources_855394_27.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/38bd0c46-f376-43f2-ba18-a464615af381_isrForCrosstalkSources_855394_27.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a9715c1206916dd43dff3455664c272 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3c92488-ec2f-4b32-ab63-bd4acd9538e1_isrForCrosstalkSources_855414_56.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3c92488-ec2f-4b32-ab63-bd4acd9538e1_isrForCrosstalkSources_855414_56.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13772b8cf33bf2cddc4be3c288666568 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c78c7297-f931-4bbf-a333-cb6cb0852ddc_isrForCrosstalkSources_855421_54.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c78c7297-f931-4bbf-a333-cb6cb0852ddc_isrForCrosstalkSources_855421_54.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01a2de9c60c6585bf7c4c40e5d0aa642 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31999c57-6a28-46a9-b34c-30866d7de038_isrForCrosstalkSources_855401_34.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31999c57-6a28-46a9-b34c-30866d7de038_isrForCrosstalkSources_855401_34.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 084064fe3dfbaa988d72d2f6625a40b2 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/fd1038fa-9a73-4199-8741-a92191e685a1_isrForCrosstalkSources_855433_54.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/fd1038fa-9a73-4199-8741-a92191e685a1_isrForCrosstalkSources_855433_54.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2d1738dac571496975b6e0dff78c3e2 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ec47a10d-5d61-45a8-a5d8-c597c22a67f2_isrForCrosstalkSources_855444_26.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ec47a10d-5d61-45a8-a5d8-c597c22a67f2_isrForCrosstalkSources_855444_26.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 036ac81cd9dc251ab0077ced1be921ac with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7effbb11-823e-4f0d-b3f2-09fa6e7d19f4_isrForCrosstalkSources_855436_2.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7effbb11-823e-4f0d-b3f2-09fa6e7d19f4_isrForCrosstalkSources_855436_2.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ed9d1d6940e81426de02ef3c7af6089 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1d6b84fa-65c6-455d-a79d-6b860bb16daa_isrForCrosstalkSources_855419_47.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/1d6b84fa-65c6-455d-a79d-6b860bb16daa_isrForCrosstalkSources_855419_47.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5df8f3dbcecd723e532310fe59d9879 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/da9c17d9-d595-4f8d-bd37-391f76eb1ae1_isrForCrosstalkSources_855384_26.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/da9c17d9-d595-4f8d-bd37-391f76eb1ae1_isrForCrosstalkSources_855384_26.stderr parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 223e6d9a961744462c399d95126b3c35 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1923de1e-ddcc-4cd5-9e22-164859d9313a_isrForCrosstalkSources_855431_40.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1923de1e-ddcc-4cd5-9e22-164859d9313a_isrForCrosstalkSources_855431_40.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bdb3bdbe886ebd495ed11de365ae547 with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6e14014f-d7c8-475a-bace-5fe3c4fc5bd5_isrForCrosstalkSources_855413_32.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6e14014f-d7c8-475a-bace-5fe3c4fc5bd5_isrForCrosstalkSources_855413_32.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2d7fe6b9c2cdb2604edb12043629748 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c3ff0a66-bb65-4e23-bc11-7b9b09829d9b_isrForCrosstalkSources_855446_59.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c3ff0a66-bb65-4e23-bc11-7b9b09829d9b_isrForCrosstalkSources_855446_59.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7185c97adaa65a8e814ab05c04f3ce3 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b2f27ae0-928c-4387-b78b-edf249caf4e2_isrForCrosstalkSources_855451_57.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b2f27ae0-928c-4387-b78b-edf249caf4e2_isrForCrosstalkSources_855451_57.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6c83571304134f663f458b8cbfcb27d with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/31ae4494-99d0-48b4-bed8-6c0772dd4b0b_isrForCrosstalkSources_855402_60.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/31ae4494-99d0-48b4-bed8-6c0772dd4b0b_isrForCrosstalkSources_855402_60.stderr 2024-03-09 14:52:27 proc_lsst.multi:170 [INFO] [multi] job_id 16964271 proc_lsst.multi INFO: [multi] job_id 16964271 2024-03-09 14:52:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 187 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 187 2024-03-09 14:52:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964271 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964271 parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d73a67fb59ecb5fa990a283c3c75b4d5 with result from task 874 parsl.executors.status_handling DEBUG: Launched block 242 on executor multi with job ID 16964271 parsl.executors.status_handling INFO: Allocated block ID 243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-243 2024-03-09 14:52:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 2024-03-09 14:52:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 187 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 187 2024-03-09 14:52:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e9552b12-6573-4a8e-b29c-029338684195_isrForCrosstalkSources_855443_16.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e9552b12-6573-4a8e-b29c-029338684195_isrForCrosstalkSources_855443_16.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e66789a9bf6b6b744580b60a753fb617 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31b98e7a-4e0e-484d-837d-086ab79dd0b9_isrForCrosstalkSources_855401_60.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/31b98e7a-4e0e-484d-837d-086ab79dd0b9_isrForCrosstalkSources_855401_60.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dde838332b672591b90e05e75a770637 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c696583d-8797-4f2b-bd46-6f970f041d16_isrForCrosstalkSources_855441_51.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c696583d-8797-4f2b-bd46-6f970f041d16_isrForCrosstalkSources_855441_51.stderr parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 440128e003734869e9eb152b32586063 with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf646472-5e99-4942-be84-8aa7ea484c7b_isrForCrosstalkSources_855402_17.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf646472-5e99-4942-be84-8aa7ea484c7b_isrForCrosstalkSources_855402_17.stderr parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f536c47f580ac935872157c39ada5ad with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f3d8f57b-858e-40a2-adff-b6516a6613c9_isrForCrosstalkSources_855453_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f3d8f57b-858e-40a2-adff-b6516a6613c9_isrForCrosstalkSources_855453_40.stderr parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 208a3b063aab8700d6e8a0dc439b8eed with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bdea1492-f43e-4567-9edc-467aa3cdeda8_isrForCrosstalkSources_855381_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bdea1492-f43e-4567-9edc-467aa3cdeda8_isrForCrosstalkSources_855381_36.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcea4e19a5b842d2db8706f644567d2a with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/2c97c79f-bf31-4122-b70d-10616e738c19_isrForCrosstalkSources_855390_27.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/2c97c79f-bf31-4122-b70d-10616e738c19_isrForCrosstalkSources_855390_27.stderr parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55bc27256b714013c916a75ad946deea with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3489c778-eb49-46ba-823d-1f992b5be100_isrForCrosstalkSources_855382_22.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3489c778-eb49-46ba-823d-1f992b5be100_isrForCrosstalkSources_855382_22.stderr parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b5db8523290458d0174081b30ff36a6 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9fe28-a082-4f25-b7b8-bb58712aa0a8_isrForCrosstalkSources_855429_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/e3e9fe28-a082-4f25-b7b8-bb58712aa0a8_isrForCrosstalkSources_855429_25.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbf035899cec50cf0b485decda337f7d with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f94ad932-9544-4442-a653-2e11b312f288_isrForCrosstalkSources_855459_19.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f94ad932-9544-4442-a653-2e11b312f288_isrForCrosstalkSources_855459_19.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15ac2abed5a8f8da6f9cf844961200e3 with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8d51b0e1-9341-48d7-8e1d-b875846fce03_isrForCrosstalkSources_855384_9.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8d51b0e1-9341-48d7-8e1d-b875846fce03_isrForCrosstalkSources_855384_9.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 456489b176c6b9e6ba5c51b7a16706df with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0f551a5a-0782-48a0-b411-eaf4aa75cf9b_isrForCrosstalkSources_855459_34.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0f551a5a-0782-48a0-b411-eaf4aa75cf9b_isrForCrosstalkSources_855459_34.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81c7980a8e8a5e60acd148f28dbc912e with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/14d3c164-2130-49b1-9a75-9acd41e1451e_isrForCrosstalkSources_855428_52.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/14d3c164-2130-49b1-9a75-9acd41e1451e_isrForCrosstalkSources_855428_52.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b0edad1a22e23e3d77c851b28838186 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/4073da7f-6f45-4068-96cc-33edfc9595e3_isrForCrosstalkSources_855423_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/4073da7f-6f45-4068-96cc-33edfc9595e3_isrForCrosstalkSources_855423_15.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 066e523aa154e2aa5affbb4f039ce72f with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bae434fa-bbcf-4126-b535-e9ecf2e48984_isrForCrosstalkSources_855430_22.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bae434fa-bbcf-4126-b535-e9ecf2e48984_isrForCrosstalkSources_855430_22.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05b294867334d051582ea43076e49de3 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d0a4a43e-ba3d-4db7-8a1c-8b05efe824dd_isrForCrosstalkSources_855430_31.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d0a4a43e-ba3d-4db7-8a1c-8b05efe824dd_isrForCrosstalkSources_855430_31.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3554b7f05ab507c0239a8e5ae2978b16 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cfcae953-b652-4954-b40c-94a737cb477b_isrForCrosstalkSources_855445_61.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cfcae953-b652-4954-b40c-94a737cb477b_isrForCrosstalkSources_855445_61.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 303c72fa28075959dccc52198516a5c1 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2c7611f7-2596-443c-afaf-5bbdae65f3ac_isrForCrosstalkSources_855429_43.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2c7611f7-2596-443c-afaf-5bbdae65f3ac_isrForCrosstalkSources_855429_43.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d45cab23771a9edfe483208ac714746 with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7463069e-43d1-45d4-a6e7-32eb92f46a61_isrForCrosstalkSources_855429_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7463069e-43d1-45d4-a6e7-32eb92f46a61_isrForCrosstalkSources_855429_16.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574ece74f4ba0a61d57e78138e7ff990 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ecf76e-202f-4374-879c-539f6aa87723_isrForCrosstalkSources_855437_5.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ecf76e-202f-4374-879c-539f6aa87723_isrForCrosstalkSources_855437_5.stderr parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6b1523cd1cb63f58daca3a4608f9626 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/580efdd1-ed1a-4fa5-8d2c-23f6b3a61b71_isrForCrosstalkSources_855424_19.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/580efdd1-ed1a-4fa5-8d2c-23f6b3a61b71_isrForCrosstalkSources_855424_19.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dcb46d80d5dfef25da929f391eb04e2 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4beca878-5e8a-4800-b49e-78f85050588b_isrForCrosstalkSources_855382_26.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4beca878-5e8a-4800-b49e-78f85050588b_isrForCrosstalkSources_855382_26.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfac1c92571fbdca5f62d17127903e83 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0dcb8e5c-c3f3-427d-be4e-89eb386f7f08_isrForCrosstalkSources_855438_29.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/0dcb8e5c-c3f3-427d-be4e-89eb386f7f08_isrForCrosstalkSources_855438_29.stderr 2024-03-09 14:52:29 proc_lsst.multi:170 [INFO] [multi] job_id 16964272 proc_lsst.multi INFO: [multi] job_id 16964272 2024-03-09 14:52:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 188 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 188 2024-03-09 14:52:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964272 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964272 parsl.executors.status_handling DEBUG: Launched block 243 on executor multi with job ID 16964272 parsl.executors.status_handling INFO: Allocated block ID 244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-244 2024-03-09 14:52:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 188 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 188 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17280e6104c66414cacac6813e372677 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ed983a27-7c32-480a-b4bc-f8145e3ed628_isrForCrosstalkSources_855391_46.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ed983a27-7c32-480a-b4bc-f8145e3ed628_isrForCrosstalkSources_855391_46.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f41f19b90bb86a27920a3f2ebea1d18 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/36537fe4-6c78-49b4-8c1b-9293e0484886_isrForCrosstalkSources_855445_22.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/36537fe4-6c78-49b4-8c1b-9293e0484886_isrForCrosstalkSources_855445_22.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3ffc93f12b66ca8a17bbb1df2a55cdc with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/234aa698-449c-492f-897b-ce98ff2dcd3c_isrForCrosstalkSources_855416_5.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/234aa698-449c-492f-897b-ce98ff2dcd3c_isrForCrosstalkSources_855416_5.stderr parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdd13d2943704112890cfa17c3d4a5e4 with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/05d6a862-74fe-4add-ae70-2de782d0bff7_isrForCrosstalkSources_855423_17.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/05d6a862-74fe-4add-ae70-2de782d0bff7_isrForCrosstalkSources_855423_17.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7403bf44577e77f3e7826a41245c2112 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/08252cbb-7f44-4b2f-9cdf-6ae6f9bdb469_isrForCrosstalkSources_855420_60.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/08252cbb-7f44-4b2f-9cdf-6ae6f9bdb469_isrForCrosstalkSources_855420_60.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7284506ffd7809b8fc8745fd6989682b with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4a513950-6a7e-498d-8ea7-395b639b4db3_isrForCrosstalkSources_855409_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4a513950-6a7e-498d-8ea7-395b639b4db3_isrForCrosstalkSources_855409_24.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b92f3396fb71ab2e7c0a070f28582b8c with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ef8f681c-dc55-430b-8a52-4bca83acfa5e_isrForCrosstalkSources_855453_21.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ef8f681c-dc55-430b-8a52-4bca83acfa5e_isrForCrosstalkSources_855453_21.stderr parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4af181788a856a78ee820e8a14249a39 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c9b8c76d-7dca-4277-a237-af2178858396_isrForCrosstalkSources_855430_21.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c9b8c76d-7dca-4277-a237-af2178858396_isrForCrosstalkSources_855430_21.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bbec0a6d68f864902235df40f9cc059 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fd0e242-a6bf-43a7-a3ef-9a26a518f5f0_isrForCrosstalkSources_855387_47.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fd0e242-a6bf-43a7-a3ef-9a26a518f5f0_isrForCrosstalkSources_855387_47.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca16453e32e4b12aad103bb784e4003 with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fc7ea35c-cde6-4c16-aa9c-48352ec6369b_isrForCrosstalkSources_855401_3.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fc7ea35c-cde6-4c16-aa9c-48352ec6369b_isrForCrosstalkSources_855401_3.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fa3c7c4ab6a1809b4c439081a0ee429 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/521deab0-5edf-47e5-9aa6-b3bf66255377_isrForCrosstalkSources_855418_4.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/521deab0-5edf-47e5-9aa6-b3bf66255377_isrForCrosstalkSources_855418_4.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05b6259df19911021b351ea45fd412e9 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/463f321c-ce7a-4432-9c50-589bd9e9ec21_isrForCrosstalkSources_855444_59.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/463f321c-ce7a-4432-9c50-589bd9e9ec21_isrForCrosstalkSources_855444_59.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b234645b9ab2c085420979166fd2c2c2 with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7dc87fc2-503f-4879-a323-4cff797fd630_isrForCrosstalkSources_855455_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7dc87fc2-503f-4879-a323-4cff797fd630_isrForCrosstalkSources_855455_52.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6376c9981d082954edcd80eec3265b3 with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ef9df52d-7b94-4606-9f91-e41014b2e458_isrForCrosstalkSources_855455_11.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ef9df52d-7b94-4606-9f91-e41014b2e458_isrForCrosstalkSources_855455_11.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13b05f548df41b4c12ad20333c8a5eca with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f3a5bbd7-8fd3-4a81-8dab-ffa16d5441ba_isrForCrosstalkSources_855445_12.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f3a5bbd7-8fd3-4a81-8dab-ffa16d5441ba_isrForCrosstalkSources_855445_12.stderr 2024-03-09 14:52:29 proc_lsst.multi:170 [INFO] [multi] job_id 16964273 proc_lsst.multi INFO: [multi] job_id 16964273 2024-03-09 14:52:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 189 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 189 2024-03-09 14:52:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964273 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964273 parsl.executors.status_handling DEBUG: Launched block 244 on executor multi with job ID 16964273 parsl.executors.status_handling INFO: Allocated block ID 245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-245 2024-03-09 14:52:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 189 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 189 2024-03-09 14:52:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2c56fff2e8131ec74d21b5e2efb379b with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ee35429a-31e9-4eaa-aac9-cd67ae452315_isrForCrosstalkSources_855454_50.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ee35429a-31e9-4eaa-aac9-cd67ae452315_isrForCrosstalkSources_855454_50.stderr 2024-03-09 14:52:30 proc_lsst.multi:170 [INFO] [multi] job_id 16964274 proc_lsst.multi INFO: [multi] job_id 16964274 2024-03-09 14:52:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 190 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 190 2024-03-09 14:52:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964274 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964274 parsl.executors.status_handling DEBUG: Launched block 245 on executor multi with job ID 16964274 parsl.executors.status_handling INFO: Allocated block ID 246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-246 2024-03-09 14:52:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 2024-03-09 14:52:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 190 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 190 2024-03-09 14:52:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f722382aca022b4e5d2a30d4eb55a1c with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/29ca02bf-d462-4c29-8de2-ff8b1759b99e_isrForCrosstalkSources_855454_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/29ca02bf-d462-4c29-8de2-ff8b1759b99e_isrForCrosstalkSources_855454_40.stderr parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eed7b4ccf4386b9911ad242084a8fa0 with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9b2007a0-8ff6-4528-9585-115b8534f156_isrForCrosstalkSources_855408_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9b2007a0-8ff6-4528-9585-115b8534f156_isrForCrosstalkSources_855408_33.stderr parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f08242278aa4c4e1dd7887dc80dc0b4 with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6f5c9e97-5f3f-45b0-9521-1bd51892ed31_isrForCrosstalkSources_855385_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6f5c9e97-5f3f-45b0-9521-1bd51892ed31_isrForCrosstalkSources_855385_39.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6fcf0aafde3e4ddc76afc2697321ed with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7b76203c-6b18-4799-8a4d-8827bb4ad7ee_isrForCrosstalkSources_855405_10.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7b76203c-6b18-4799-8a4d-8827bb4ad7ee_isrForCrosstalkSources_855405_10.stderr parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f831d41e1578ee331d05eac76ed93d with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0a34c0bb-3a02-40be-ae12-d37bab388cb2_isrForCrosstalkSources_855389_61.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0a34c0bb-3a02-40be-ae12-d37bab388cb2_isrForCrosstalkSources_855389_61.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75cc1205d3fbd79cce83581df4131abc with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8d1c4f5e-6008-44ef-8c38-d07aa00f6b65_isrForCrosstalkSources_855431_50.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8d1c4f5e-6008-44ef-8c38-d07aa00f6b65_isrForCrosstalkSources_855431_50.stderr parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3db0abfa4a7fd40e50409342675e5d82 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/121ebf3b-3849-40aa-9302-587cebd88fc9_isrForCrosstalkSources_855399_5.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/121ebf3b-3849-40aa-9302-587cebd88fc9_isrForCrosstalkSources_855399_5.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edfe31528eb6272cb6b97fb19e551518 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c980437d-e7fb-4f10-b38b-6d61373f9a16_isrForCrosstalkSources_855420_10.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c980437d-e7fb-4f10-b38b-6d61373f9a16_isrForCrosstalkSources_855420_10.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d2771948097072edd4582fc062b337 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f13bcc2f-d973-4e56-88aa-b8fdde591f95_isrForCrosstalkSources_855426_58.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f13bcc2f-d973-4e56-88aa-b8fdde591f95_isrForCrosstalkSources_855426_58.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb7ff246fb080d37ab4ddfab11ec7e00 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ea1da352-bc89-48d1-b795-ebce2a1e4f67_isrForCrosstalkSources_855401_2.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ea1da352-bc89-48d1-b795-ebce2a1e4f67_isrForCrosstalkSources_855401_2.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed24c34e6582a8846681d6efc859972f with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8a783725-7adf-4362-98ea-c9ea91dd79ba_isrForCrosstalkSources_855384_44.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8a783725-7adf-4362-98ea-c9ea91dd79ba_isrForCrosstalkSources_855384_44.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6ac34237c341a6524d977fc1b3e86be with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e39d796c-d475-47b8-97e8-4ea2bbfa5459_isrForCrosstalkSources_855413_36.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e39d796c-d475-47b8-97e8-4ea2bbfa5459_isrForCrosstalkSources_855413_36.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf55acabdbbd83a8cb815d6e08d728b with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/531137ee-3e7d-4b22-83ea-14add9143058_isrForCrosstalkSources_855443_49.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/531137ee-3e7d-4b22-83ea-14add9143058_isrForCrosstalkSources_855443_49.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba6bbc0824eea9885e6023b83ebd9551 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/be6f1ec2-182e-40b6-b098-9da8560d839e_isrForCrosstalkSources_855408_15.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/be6f1ec2-182e-40b6-b098-9da8560d839e_isrForCrosstalkSources_855408_15.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f01a5111397547aa3baa9692c0bf76b with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8e25f42f-892a-4dcf-8fdf-0bca894c26c8_isrForCrosstalkSources_855431_33.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8e25f42f-892a-4dcf-8fdf-0bca894c26c8_isrForCrosstalkSources_855431_33.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1ba9b0472c514587ef1ab63a992e000 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/aa1554c4-1e58-48e7-b845-fe9e2ca05072_isrForCrosstalkSources_855431_10.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/aa1554c4-1e58-48e7-b845-fe9e2ca05072_isrForCrosstalkSources_855431_10.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79e3bb4982cb1ec04cddd67973e7f351 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/869c1a48-3b3c-4a5c-81ee-5f06d17bb690_isrForCrosstalkSources_855388_8.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/869c1a48-3b3c-4a5c-81ee-5f06d17bb690_isrForCrosstalkSources_855388_8.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8fadb304f065225a30b321f7896bc1 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032f1fbd-4781-41d2-9ee1-b1473b1c7aee_isrForCrosstalkSources_855414_30.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032f1fbd-4781-41d2-9ee1-b1473b1c7aee_isrForCrosstalkSources_855414_30.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e38ec961dff4b30db488e1e278a8151 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/2ed1df3e-7474-4a2d-a853-5d098f38381b_isrForCrosstalkSources_855402_4.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/2ed1df3e-7474-4a2d-a853-5d098f38381b_isrForCrosstalkSources_855402_4.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3f2879a5eef89228302444c59092403 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/c40875d3-1bc5-4167-903d-03377f8a7dd8_isrForCrosstalkSources_855432_4.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/c40875d3-1bc5-4167-903d-03377f8a7dd8_isrForCrosstalkSources_855432_4.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cb3d4e79b207fc85d5170759893dd3c with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/14ad085b-3874-454e-ad7e-5cd5217bdee7_isrForCrosstalkSources_855446_45.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/14ad085b-3874-454e-ad7e-5cd5217bdee7_isrForCrosstalkSources_855446_45.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 594c3b2f16758d3084a587bb9367cddb with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c37ee41c-721e-44c3-9815-0907752c6f28_isrForCrosstalkSources_855430_18.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/c37ee41c-721e-44c3-9815-0907752c6f28_isrForCrosstalkSources_855430_18.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98139a98ec93715794f070687dbec35c with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e5c8368c-520c-4428-bb0c-248a62dee7f3_isrForCrosstalkSources_855412_52.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/e5c8368c-520c-4428-bb0c-248a62dee7f3_isrForCrosstalkSources_855412_52.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20de5cf8d14c507a693751ef544aec00 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/57f32464-29e3-4bbe-94df-2b266295ab5b_isrForCrosstalkSources_855389_22.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/57f32464-29e3-4bbe-94df-2b266295ab5b_isrForCrosstalkSources_855389_22.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84480d7ac5308cf443dc906dbbb4866d with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f3420479-8c87-4e17-afb1-40894225986b_isrForCrosstalkSources_855455_26.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f3420479-8c87-4e17-afb1-40894225986b_isrForCrosstalkSources_855455_26.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7790b466921808dc55b3b376907f20b2 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/21605bcd-c832-424f-b65f-f7758b1bf6bf_isrForCrosstalkSources_855442_22.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/21605bcd-c832-424f-b65f-f7758b1bf6bf_isrForCrosstalkSources_855442_22.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 028ead258ecd0cff95ffadd55917bbd9 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2ee48f71-9365-48e9-8be2-9121c56441f2_isrForCrosstalkSources_855456_24.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2ee48f71-9365-48e9-8be2-9121c56441f2_isrForCrosstalkSources_855456_24.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ff89e617bf81ad574596d1fbbfe9c69 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d292ccb3-4e86-419e-b3a2-a9206638fc45_isrForCrosstalkSources_855401_23.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/d292ccb3-4e86-419e-b3a2-a9206638fc45_isrForCrosstalkSources_855401_23.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff49e44efddc67966f64d2f2d2c5cfcb with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0731c929-fff3-46b1-ad43-3f7e2e8153fb_isrForCrosstalkSources_855457_7.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0731c929-fff3-46b1-ad43-3f7e2e8153fb_isrForCrosstalkSources_855457_7.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6365863928f12e09a5efc1a0f416130e with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/80e58bc6-5d86-4ef8-ae5b-c7aa7bf3915f_isrForCrosstalkSources_855451_54.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/80e58bc6-5d86-4ef8-ae5b-c7aa7bf3915f_isrForCrosstalkSources_855451_54.stderr parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd748764c0563b62fe94c3d1b009eff5 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/129c15a2-6e9f-4f6e-86c2-e53967438ded_isrForCrosstalkSources_855447_41.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/129c15a2-6e9f-4f6e-86c2-e53967438ded_isrForCrosstalkSources_855447_41.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2365e5ac0df7c59db2dbf4901f89a2f9 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/45858fd8-515f-47c1-972c-9b620b84d7d1_isrForCrosstalkSources_855450_1.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/45858fd8-515f-47c1-972c-9b620b84d7d1_isrForCrosstalkSources_855450_1.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 873bbb5d243d196b873fa6f314917056 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/506fb34d-8206-4211-b383-db79b5ba854d_isrForCrosstalkSources_855385_54.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/506fb34d-8206-4211-b383-db79b5ba854d_isrForCrosstalkSources_855385_54.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 680b58fcac61110b65cda2575d5a7751 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f5bcb64-f4b7-40ed-b248-e79645de131a_isrForCrosstalkSources_855456_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f5bcb64-f4b7-40ed-b248-e79645de131a_isrForCrosstalkSources_855456_61.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f386f2fa2d50273adef8ef09e2ee7d2 with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/538f88e9-83df-4374-ba70-0f4d62de7f19_isrForCrosstalkSources_855455_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/538f88e9-83df-4374-ba70-0f4d62de7f19_isrForCrosstalkSources_855455_47.stderr 2024-03-09 14:52:31 proc_lsst.multi:170 [INFO] [multi] job_id 16964275 proc_lsst.multi INFO: [multi] job_id 16964275 2024-03-09 14:52:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 191 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 191 2024-03-09 14:52:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964275 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964275 parsl.executors.status_handling DEBUG: Launched block 246 on executor multi with job ID 16964275 parsl.executors.status_handling INFO: Allocated block ID 247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-247 2024-03-09 14:52:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 2024-03-09 14:52:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 191 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 191 2024-03-09 14:52:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 850dac28314b91ec1539d46327d2f479 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f10da8e1-a104-4cdb-893d-736408bc2f99_isrForCrosstalkSources_855399_59.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f10da8e1-a104-4cdb-893d-736408bc2f99_isrForCrosstalkSources_855399_59.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d52065f1471569d4ae7578bf817db38c with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/099404e0-dc83-45af-900b-ad76573a69d1_isrForCrosstalkSources_855387_40.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/099404e0-dc83-45af-900b-ad76573a69d1_isrForCrosstalkSources_855387_40.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb4f48c8e4e814fbd7c7b6d3d931d9a4 with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/123b546c-1daf-4a33-8a36-689709da44f5_isrForCrosstalkSources_855409_6.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/123b546c-1daf-4a33-8a36-689709da44f5_isrForCrosstalkSources_855409_6.stderr parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b6f1adf97e330d490dd5b7b3ebc539 with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e2773-04a2-49fb-ae30-a8ad4aa74a96_isrForCrosstalkSources_855448_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e2773-04a2-49fb-ae30-a8ad4aa74a96_isrForCrosstalkSources_855448_34.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f34c93ef20200f24ee1860016fe11814 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/25358229-3199-48f3-924a-7aa01718cf94_isrForCrosstalkSources_855447_27.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/25358229-3199-48f3-924a-7aa01718cf94_isrForCrosstalkSources_855447_27.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0475255aad292936338c18f711ead941 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/432f5ace-673f-4f2e-ab4b-9b7eb9829d51_isrForCrosstalkSources_855457_27.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/432f5ace-673f-4f2e-ab4b-9b7eb9829d51_isrForCrosstalkSources_855457_27.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd481b97568287aececf209b1c691ce6 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c748a012-c0c0-47e0-b6a6-18a64ccfd849_isrForCrosstalkSources_855433_13.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c748a012-c0c0-47e0-b6a6-18a64ccfd849_isrForCrosstalkSources_855433_13.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac631e10e650e44791ecf1491b74b0ea with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/768a61ee-32fa-4b2b-8d0c-2042c2313f2a_isrForCrosstalkSources_855416_20.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/768a61ee-32fa-4b2b-8d0c-2042c2313f2a_isrForCrosstalkSources_855416_20.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8d964249c8bb19df887840872ae77a1 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66bafbea-660f-4546-8b0c-a04694c0ed1d_isrForCrosstalkSources_855390_46.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66bafbea-660f-4546-8b0c-a04694c0ed1d_isrForCrosstalkSources_855390_46.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5108f8f50cced2c11089047c18baf550 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0f7cb022-e69c-456f-96b8-3fe44eb5e868_isrForCrosstalkSources_855406_30.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0f7cb022-e69c-456f-96b8-3fe44eb5e868_isrForCrosstalkSources_855406_30.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23f96d850d97811b962fbc482bb1503f with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d3e87a54-f143-4785-927c-ad530d565ae8_isrForCrosstalkSources_855419_36.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d3e87a54-f143-4785-927c-ad530d565ae8_isrForCrosstalkSources_855419_36.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 375980ba3c8475c73056e4fb21a784a6 with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/96e0eeba-27d6-4b5c-b8ba-8dac803ebd13_isrForCrosstalkSources_855449_47.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/96e0eeba-27d6-4b5c-b8ba-8dac803ebd13_isrForCrosstalkSources_855449_47.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a1ee052bdae081430893485de36f332 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4ccadd59-95c0-433e-96f1-6cdae7dcfc8a_isrForCrosstalkSources_855430_59.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4ccadd59-95c0-433e-96f1-6cdae7dcfc8a_isrForCrosstalkSources_855430_59.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f981fc77b92b7d83e9a576b608f417 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a8d6b9c-fea8-4942-a09d-e670727a493a_isrForCrosstalkSources_855410_18.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a8d6b9c-fea8-4942-a09d-e670727a493a_isrForCrosstalkSources_855410_18.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64029facfd338593c3021c9e405713ab with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ebeaa0b2-6c91-4538-a3b8-491f5468ed7e_isrForCrosstalkSources_855435_62.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/ebeaa0b2-6c91-4538-a3b8-491f5468ed7e_isrForCrosstalkSources_855435_62.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63551dd6407035802babc7c5077daa92 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/06bf3718-9e6b-45fc-a321-56369f6965a3_isrForCrosstalkSources_855418_35.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/06bf3718-9e6b-45fc-a321-56369f6965a3_isrForCrosstalkSources_855418_35.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b53e5ff6010bc7a2a74aa2c800810cf with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1ce523cb-af51-475e-bae7-19fb735ddd9b_isrForCrosstalkSources_855431_34.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1ce523cb-af51-475e-bae7-19fb735ddd9b_isrForCrosstalkSources_855431_34.stderr parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18e413882facab642847e83a3ef90313 with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/825328c4-c3bc-4522-b511-5042d9f7efa8_isrForCrosstalkSources_855380_26.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/825328c4-c3bc-4522-b511-5042d9f7efa8_isrForCrosstalkSources_855380_26.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59bb82c60df328bd696df902ef7c3e9e with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4039e89b-6188-4511-a95e-5aec5a53ee23_isrForCrosstalkSources_855452_9.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/4039e89b-6188-4511-a95e-5aec5a53ee23_isrForCrosstalkSources_855452_9.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae53897be06e10f8354a56c1c287639 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5d45219b-9d98-49bf-b41d-9af66e9bc7d2_isrForCrosstalkSources_855427_36.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5d45219b-9d98-49bf-b41d-9af66e9bc7d2_isrForCrosstalkSources_855427_36.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f1fd9b76b9c29012de8cecebdcab59 with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/82f9f1c2-f793-421a-8f9e-8b3b56f2850e_isrForCrosstalkSources_855381_44.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/82f9f1c2-f793-421a-8f9e-8b3b56f2850e_isrForCrosstalkSources_855381_44.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caaf44c6a10544b4217a980ad2957953 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a8dc8b6d-c42c-432a-b4c5-a08952597e0c_isrForCrosstalkSources_855431_23.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a8dc8b6d-c42c-432a-b4c5-a08952597e0c_isrForCrosstalkSources_855431_23.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc28b61344f892f5614064bdf4b4767c with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/826c5689-64fc-4e0f-99cb-78205e96a783_isrForCrosstalkSources_855456_10.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/826c5689-64fc-4e0f-99cb-78205e96a783_isrForCrosstalkSources_855456_10.stderr parsl.dataflow.dflow INFO: Task 1119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75033a4745517e732529f4f35b620685 with result from task 1119 parsl.dataflow.dflow DEBUG: Task 1119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/18e7f7ca-190e-4988-bc18-d02e25bbcfe8_isrForCrosstalkSources_855440_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/18e7f7ca-190e-4988-bc18-d02e25bbcfe8_isrForCrosstalkSources_855440_4.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1711ad15ead78a0a222e21974d12c57b with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/985a9c48-5cb5-4e4c-ae4a-b8f6536b2755_isrForCrosstalkSources_855457_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/985a9c48-5cb5-4e4c-ae4a-b8f6536b2755_isrForCrosstalkSources_855457_15.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a9791b96a5d345ef4601877d9bfe036 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/99ed4abe-63cc-4b28-8654-01fe10cfbdcb_isrForCrosstalkSources_855456_18.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/99ed4abe-63cc-4b28-8654-01fe10cfbdcb_isrForCrosstalkSources_855456_18.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8f98d718ea50290fb0926435d139fa8 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/38f0a6f7-7022-4fef-9bfe-318eb4d916b0_isrForCrosstalkSources_855440_48.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/38f0a6f7-7022-4fef-9bfe-318eb4d916b0_isrForCrosstalkSources_855440_48.stderr parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78156545d27f16e70c5faec17459cf8a with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10552c0e-c99e-41e9-b92d-c3ec8dd00adb_isrForCrosstalkSources_855412_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10552c0e-c99e-41e9-b92d-c3ec8dd00adb_isrForCrosstalkSources_855412_26.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a85d16f73b13d673b0754aca3d22e13 with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d63f9f8a-0697-4dcd-a23e-4bd3ecf9a555_isrForCrosstalkSources_855397_43.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d63f9f8a-0697-4dcd-a23e-4bd3ecf9a555_isrForCrosstalkSources_855397_43.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36a8f4f6c4a9afd9dfb36105b753c267 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/04cb54b0-7d9b-4dcd-84d1-5820eb3d2c8a_isrForCrosstalkSources_855393_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/04cb54b0-7d9b-4dcd-84d1-5820eb3d2c8a_isrForCrosstalkSources_855393_28.stderr 2024-03-09 14:52:33 proc_lsst.multi:170 [INFO] [multi] job_id 16964276 proc_lsst.multi INFO: [multi] job_id 16964276 2024-03-09 14:52:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 192 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 192 2024-03-09 14:52:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964276 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964276 parsl.executors.status_handling DEBUG: Launched block 247 on executor multi with job ID 16964276 parsl.executors.status_handling INFO: Allocated block ID 248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-248 2024-03-09 14:52:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 2024-03-09 14:52:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 192 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 192 2024-03-09 14:52:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9747deb8631c5fafd11ec6ae9b79449 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5e09a3c2-7266-4b51-85e7-0ad5f7cb4112_isrForCrosstalkSources_855422_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5e09a3c2-7266-4b51-85e7-0ad5f7cb4112_isrForCrosstalkSources_855422_26.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdc25665f14f59928e35697f2e3218ba with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e4241ec4-7596-4d3a-85e9-5e414df92eba_isrForCrosstalkSources_855434_20.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e4241ec4-7596-4d3a-85e9-5e414df92eba_isrForCrosstalkSources_855434_20.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a29ed967c527a2953590d5bbd0c7d2 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/d8bf6efa-c5db-4875-97a9-710d5b31d03e_isrForCrosstalkSources_855398_37.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/d8bf6efa-c5db-4875-97a9-710d5b31d03e_isrForCrosstalkSources_855398_37.stderr parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e69de09236c51d36f99fcdc9f49b219 with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30c19cae-c536-416c-85a1-3dcb96f58cad_isrForCrosstalkSources_855436_34.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30c19cae-c536-416c-85a1-3dcb96f58cad_isrForCrosstalkSources_855436_34.stderr parsl.dataflow.dflow INFO: Task 1136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2203bac86d95e6341afb2dc196178cbc with result from task 1136 parsl.dataflow.dflow DEBUG: Task 1136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5ee66bce-24c1-4988-8730-919f9e7ad03e_isrForCrosstalkSources_855424_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5ee66bce-24c1-4988-8730-919f9e7ad03e_isrForCrosstalkSources_855424_9.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f62af15161db67124efcbf1440d9c36 with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a99c0eec-ff8a-4151-abad-1d9ee1f90db6_isrForCrosstalkSources_855439_33.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a99c0eec-ff8a-4151-abad-1d9ee1f90db6_isrForCrosstalkSources_855439_33.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a884e924730dd0715d5142e0111b9bb with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7d28587b-cc8a-447c-a7bd-d371526a70a8_isrForCrosstalkSources_855439_31.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7d28587b-cc8a-447c-a7bd-d371526a70a8_isrForCrosstalkSources_855439_31.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63dad412335e8dfe8efc6cde31caf518 with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/f220d316-5987-41fd-8746-67c301bf0760_isrForCrosstalkSources_855457_32.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/f220d316-5987-41fd-8746-67c301bf0760_isrForCrosstalkSources_855457_32.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1c81fe7b641a121524170832e002c7 with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ad28aabc-5aeb-436b-8614-1144cc697d29_isrForCrosstalkSources_855432_27.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/ad28aabc-5aeb-436b-8614-1144cc697d29_isrForCrosstalkSources_855432_27.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1ea5b5267b6505c60a905763099713 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d98b1bcc-5cf4-421b-945b-773d2a1d09b0_isrForCrosstalkSources_855444_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d98b1bcc-5cf4-421b-945b-773d2a1d09b0_isrForCrosstalkSources_855444_19.stderr parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38df398c15bf11259777beef29c66fd4 with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/5f467234-694d-4e9e-b8fa-c3816862a4ee_isrForCrosstalkSources_855448_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/5f467234-694d-4e9e-b8fa-c3816862a4ee_isrForCrosstalkSources_855448_8.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a49e62be23ee6154d7035a0b406f2a4 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a66a15c-df8f-4ef3-987e-dba61a81a02b_isrForCrosstalkSources_855454_13.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a66a15c-df8f-4ef3-987e-dba61a81a02b_isrForCrosstalkSources_855454_13.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14a4fb52c8471cca0f9a1327605df080 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0d1523c6-49f7-405f-8d86-fc91de1c924b_isrForCrosstalkSources_855387_41.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0d1523c6-49f7-405f-8d86-fc91de1c924b_isrForCrosstalkSources_855387_41.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08b72751812fed82e67ec50c4968257 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/026d8372-33ca-4a94-a57d-8eafae82bf37_isrForCrosstalkSources_855417_58.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/026d8372-33ca-4a94-a57d-8eafae82bf37_isrForCrosstalkSources_855417_58.stderr parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e40ddea7616439b9ea9f41442128b02 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/92cb41f8-0fe1-4dcf-b106-de1a7eb2299d_isrForCrosstalkSources_855455_25.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/92cb41f8-0fe1-4dcf-b106-de1a7eb2299d_isrForCrosstalkSources_855455_25.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b30d90b4ac90b53f62dd332c8e0b046b with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/419017cb-a421-44c5-9ae0-8bd48d37f221_isrForCrosstalkSources_855455_13.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/419017cb-a421-44c5-9ae0-8bd48d37f221_isrForCrosstalkSources_855455_13.stderr parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4859063c785f47f09804a50d0295398a with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4f2402e0-b368-4c36-94e8-7b415c603659_isrForCrosstalkSources_855457_3.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4f2402e0-b368-4c36-94e8-7b415c603659_isrForCrosstalkSources_855457_3.stderr parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 820e3697fc045c929cbd7843c5e128b3 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5a43cd8d-e5c5-45bc-a3e0-e34d9c706aa9_isrForCrosstalkSources_855398_49.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5a43cd8d-e5c5-45bc-a3e0-e34d9c706aa9_isrForCrosstalkSources_855398_49.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53cb7a490e9e9a4dd32be41e77111bf4 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/449aab7e-5c39-4f8e-aa52-175ea5b599c0_isrForCrosstalkSources_855385_60.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/449aab7e-5c39-4f8e-aa52-175ea5b599c0_isrForCrosstalkSources_855385_60.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83bcaa644f71a7f4660f6776725ca8ab with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/573b230b-f219-4590-a324-f7a519aee8a6_isrForCrosstalkSources_855454_52.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/573b230b-f219-4590-a324-f7a519aee8a6_isrForCrosstalkSources_855454_52.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5398d515ec40857dd70e305c5186fb20 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/45e6356e-001f-4ad6-9609-e4f702d84115_isrForCrosstalkSources_855411_12.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/45e6356e-001f-4ad6-9609-e4f702d84115_isrForCrosstalkSources_855411_12.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e33c3cde7ae2752383b207d548d4fa96 with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/d9090721-b401-4b4c-b721-d40189abbd20_isrForCrosstalkSources_855406_45.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/d9090721-b401-4b4c-b721-d40189abbd20_isrForCrosstalkSources_855406_45.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88f92033f8f0c47086836a84341c4d5f with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a644a031-1e6a-4a3d-ac48-5ba92060c54e_isrForCrosstalkSources_855452_3.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a644a031-1e6a-4a3d-ac48-5ba92060c54e_isrForCrosstalkSources_855452_3.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc22f951b6970984a454c3aeeb77cede with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/23729405-38e8-46ea-b874-39f22bd962f7_isrForCrosstalkSources_855415_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/23729405-38e8-46ea-b874-39f22bd962f7_isrForCrosstalkSources_855415_2.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15dbb5b7899383520f4b437cdd269cf1 with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/725c5331-06b7-4eb1-a930-b2a6eabd228f_isrForCrosstalkSources_855386_15.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/725c5331-06b7-4eb1-a930-b2a6eabd228f_isrForCrosstalkSources_855386_15.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db41771de6bad8f6a9bb80744d262923 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/80520f54-29e7-47f0-ba20-7bcc1e7ad7bb_isrForCrosstalkSources_855405_16.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/80520f54-29e7-47f0-ba20-7bcc1e7ad7bb_isrForCrosstalkSources_855405_16.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06a3fbc1c8346903a5d4013b5ca59309 with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/64b5ec51-6da0-4263-940b-865fab090604_isrForCrosstalkSources_855430_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/64b5ec51-6da0-4263-940b-865fab090604_isrForCrosstalkSources_855430_45.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa4db97e84469f50a36e81424e1e8225 with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/54e5c7d0-ee1e-49bc-95c5-154d2a10c4f7_isrForCrosstalkSources_855428_43.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/54e5c7d0-ee1e-49bc-95c5-154d2a10c4f7_isrForCrosstalkSources_855428_43.stderr parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94ecbdd9319eddd44c48868ae5e7f115 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3696d5a4-b5fa-456a-a527-8335c5e36f6e_isrForCrosstalkSources_855382_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/3696d5a4-b5fa-456a-a527-8335c5e36f6e_isrForCrosstalkSources_855382_5.stderr parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c789fbb81d6b29a6ef78c2e6371394 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8654cdb2-a957-431e-a966-f9b4ada5ff20_isrForCrosstalkSources_855425_54.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8654cdb2-a957-431e-a966-f9b4ada5ff20_isrForCrosstalkSources_855425_54.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 131428d38a5ef82e866dd43988f10cce with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/439120d1-1180-461f-92bc-0f0f88a866d1_isrForCrosstalkSources_855421_7.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/439120d1-1180-461f-92bc-0f0f88a866d1_isrForCrosstalkSources_855421_7.stderr parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f8c415bd3974942226446bb95904e37 with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/244aef86-a315-425f-9910-c7c7a9768563_isrForCrosstalkSources_855454_34.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/244aef86-a315-425f-9910-c7c7a9768563_isrForCrosstalkSources_855454_34.stderr parsl.dataflow.dflow INFO: Task 1131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f75e589ee7d1698cb707bafe14b0299 with result from task 1131 parsl.dataflow.dflow DEBUG: Task 1131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8324c2-c965-453c-8f78-c9e91511f296_isrForCrosstalkSources_855404_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8324c2-c965-453c-8f78-c9e91511f296_isrForCrosstalkSources_855404_20.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba28ee95b04b9885047c8b78c3c6c37 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fcfc4187-7123-4c67-96d8-559e429cb1af_isrForCrosstalkSources_855379_52.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fcfc4187-7123-4c67-96d8-559e429cb1af_isrForCrosstalkSources_855379_52.stderr parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89b90f4a23cc272b98441c380843b66 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/416a4004-1f68-47d0-bffa-4573a051d940_isrForCrosstalkSources_855438_21.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/416a4004-1f68-47d0-bffa-4573a051d940_isrForCrosstalkSources_855438_21.stderr parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7bdad810ecb1ec83f2878a302da3543 with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5f6cd64c-b440-4f8a-bb08-3d5dd08d9f2a_isrForCrosstalkSources_855440_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5f6cd64c-b440-4f8a-bb08-3d5dd08d9f2a_isrForCrosstalkSources_855440_19.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfa2e579aed7782e27b7ad561385f136 with result from task 910 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/52f7bfea-86d2-439a-b12c-3f5913cb7a04_isrForCrosstalkSources_855379_31.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/52f7bfea-86d2-439a-b12c-3f5913cb7a04_isrForCrosstalkSources_855379_31.stderr parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac42ee07de37ae6b922fd45699ca35a with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68eaf8fb-a4c6-42be-aec0-2cdefd608241_isrForCrosstalkSources_855455_54.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68eaf8fb-a4c6-42be-aec0-2cdefd608241_isrForCrosstalkSources_855455_54.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dbfb01f65b77a210aa971e64321549a with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/68a4292f-c83a-4733-a20f-12aeb7d0dce0_isrForCrosstalkSources_855402_62.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/68a4292f-c83a-4733-a20f-12aeb7d0dce0_isrForCrosstalkSources_855402_62.stderr parsl.dataflow.dflow INFO: Task 1179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5edf563e21511dedd5f3c0f744f3742 with result from task 1179 parsl.dataflow.dflow DEBUG: Task 1179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c18b06ad-522f-460a-b717-070b78e1ed2e_isrForCrosstalkSources_855458_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c18b06ad-522f-460a-b717-070b78e1ed2e_isrForCrosstalkSources_855458_39.stderr parsl.dataflow.dflow INFO: Task 1123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7605325d5ddb721cb804d9f845553541 with result from task 1123 parsl.dataflow.dflow DEBUG: Task 1123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c1316e3d-f936-4222-8a40-d482553e0b02_isrForCrosstalkSources_855405_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/c1316e3d-f936-4222-8a40-d482553e0b02_isrForCrosstalkSources_855405_27.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f5cb186afdd7ffedcb4b3a72798e455 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8c4dbe61-d857-4484-9565-5ba30eab7400_isrForCrosstalkSources_855384_17.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8c4dbe61-d857-4484-9565-5ba30eab7400_isrForCrosstalkSources_855384_17.stderr parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 261299525bdb3afef2606d14da6fde0d with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/94d05175-94cf-4396-9da7-4ac451507bf5_isrForCrosstalkSources_855459_7.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/94d05175-94cf-4396-9da7-4ac451507bf5_isrForCrosstalkSources_855459_7.stderr parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b03c803a73a63a15adde7a2ac5dd9d8 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/5aa94319-f443-40f8-9e41-7b88ce631348_isrForCrosstalkSources_855431_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/5aa94319-f443-40f8-9e41-7b88ce631348_isrForCrosstalkSources_855431_52.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1149bd7fd496ec5e469268df085a05d4 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8061285-0e0c-4815-adfe-8321397955e3_isrForCrosstalkSources_855399_60.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b8061285-0e0c-4815-adfe-8321397955e3_isrForCrosstalkSources_855399_60.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5ecdf0d070600325a3167a18f1afac7 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9fd08a27-0612-46a9-9685-0eafb5cde0fb_isrForCrosstalkSources_855437_48.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9fd08a27-0612-46a9-9685-0eafb5cde0fb_isrForCrosstalkSources_855437_48.stderr parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085537abfbba19366ec0a419a1a57985 with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f3b48296-226f-41cc-8f34-06b7113c0bdd_isrForCrosstalkSources_855452_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f3b48296-226f-41cc-8f34-06b7113c0bdd_isrForCrosstalkSources_855452_39.stderr parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 661ab47d693973672978d8e1ced2790c with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6bb6851b-3b77-46a7-a31e-0e45de69e5b1_isrForCrosstalkSources_855392_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6bb6851b-3b77-46a7-a31e-0e45de69e5b1_isrForCrosstalkSources_855392_24.stderr 2024-03-09 14:52:34 proc_lsst.multi:170 [INFO] [multi] job_id 16964277 proc_lsst.multi INFO: [multi] job_id 16964277 2024-03-09 14:52:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 193 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 193 2024-03-09 14:52:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964277 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964277 parsl.executors.status_handling DEBUG: Launched block 248 on executor multi with job ID 16964277 parsl.executors.status_handling INFO: Allocated block ID 249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-249 2024-03-09 14:52:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 2024-03-09 14:52:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:52:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:52:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:52:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:52:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:52:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:52:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:52:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:52:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:52:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:52:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 193 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 193 2024-03-09 14:52:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37dd0a296b7c9b8e9e5d14251c66c26b with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d782ef03-c72f-40cc-9b9e-daa9265511ce_isrForCrosstalkSources_855449_33.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d782ef03-c72f-40cc-9b9e-daa9265511ce_isrForCrosstalkSources_855449_33.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd5c0ca22d011658ab8ed53527e7a50 with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/3da81791-fe1d-491d-bace-60599496cfeb_isrForCrosstalkSources_855431_7.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/3da81791-fe1d-491d-bace-60599496cfeb_isrForCrosstalkSources_855431_7.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74ac0f6233db10adf4a39050cbbf1e4f with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a196b5d0-7d31-484e-8101-e35719fa55c6_isrForCrosstalkSources_855388_62.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a196b5d0-7d31-484e-8101-e35719fa55c6_isrForCrosstalkSources_855388_62.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ad3f8d398ec8222bcf795131aa915e with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5dfb448d-7469-4280-b8e9-20d1248e8421_isrForCrosstalkSources_855398_61.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5dfb448d-7469-4280-b8e9-20d1248e8421_isrForCrosstalkSources_855398_61.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca05c50414044e69591aac256d35506 with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f737ce33-fff4-425e-ad91-94f7c080874e_isrForCrosstalkSources_855453_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f737ce33-fff4-425e-ad91-94f7c080874e_isrForCrosstalkSources_855453_51.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d223b82d62bb92fcf663d1dc35da037 with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/84266b3b-b3d0-44af-9fc7-94a4b371f39a_isrForCrosstalkSources_855403_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/84266b3b-b3d0-44af-9fc7-94a4b371f39a_isrForCrosstalkSources_855403_12.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70d0bf2df625b68e9d2e26b236d7e0e with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/69f6faee-98c6-4493-ad56-73d50022f468_isrForCrosstalkSources_855438_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/69f6faee-98c6-4493-ad56-73d50022f468_isrForCrosstalkSources_855438_38.stderr parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 093cc13a473245f9490bb7f65c21b983 with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/d5272255-b495-42ad-aace-0aea80fb82b9_isrForCrosstalkSources_855437_50.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/d5272255-b495-42ad-aace-0aea80fb82b9_isrForCrosstalkSources_855437_50.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4821f93ea90d6ee4cedd3bc74e1026c with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/b47de29f-2e28-47eb-a96f-50e3c0555ab8_isrForCrosstalkSources_855386_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/b47de29f-2e28-47eb-a96f-50e3c0555ab8_isrForCrosstalkSources_855386_25.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f855fefb4d368595779ab510ed21acd9 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9d3e89c6-802c-415b-b992-5369f5970c01_isrForCrosstalkSources_855431_30.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9d3e89c6-802c-415b-b992-5369f5970c01_isrForCrosstalkSources_855431_30.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83fd81c56123c9c0ace25bfc449470cc with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ef626d3c-0dd4-440c-ae27-d562b1bbf85f_isrForCrosstalkSources_855439_32.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ef626d3c-0dd4-440c-ae27-d562b1bbf85f_isrForCrosstalkSources_855439_32.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17b2e9bb86432c47cc60e95c234b91a5 with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d09803d3-7ccf-42c9-bff0-5c4fd4580062_isrForCrosstalkSources_855450_38.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d09803d3-7ccf-42c9-bff0-5c4fd4580062_isrForCrosstalkSources_855450_38.stderr 2024-03-09 14:52:35 proc_lsst.multi:170 [INFO] [multi] job_id 16964278 proc_lsst.multi INFO: [multi] job_id 16964278 2024-03-09 14:52:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 194 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 194 2024-03-09 14:52:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:52:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964278 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964278 parsl.executors.status_handling DEBUG: Launched block 249 on executor multi with job ID 16964278 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22375328864144 2024-03-09 14:52:35 proc_lsst.multi:146 [INFO] found job 77954 in provider local proc_lsst.multi INFO: found job 77954 in provider local parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1324b9657efad86709358dedd50008 with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1c9beefd-e628-49bc-ab8d-056a85a5cb0f_isrForCrosstalkSources_855439_23.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1c9beefd-e628-49bc-ab8d-056a85a5cb0f_isrForCrosstalkSources_855439_23.stderr 2024-03-09 14:52:35 proc_lsst.multi:146 [INFO] found job 16964026 in provider astro proc_lsst.multi INFO: found job 16964026 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc90de48b5e04d38b3444b5e34c25912 with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/e3837c24-ba35-46ab-879f-6c7b77616571_isrForCrosstalkSources_855393_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/e3837c24-ba35-46ab-879f-6c7b77616571_isrForCrosstalkSources_855393_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:36 proc_lsst.multi:146 [INFO] found job 16964027 in provider astro proc_lsst.multi INFO: found job 16964027 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be8861fcc95c1e5729c1f5b608ed6932 with result from task 1133 parsl.dataflow.dflow DEBUG: Task 1133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9c953f81-bda5-4498-9b21-18d0a2387798_isrForCrosstalkSources_855399_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9c953f81-bda5-4498-9b21-18d0a2387798_isrForCrosstalkSources_855399_20.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085ad517a882e9ddbcccaa6b4c40c8ae with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b0ca8d8f-2cb9-43e5-9044-aa8d5d7a79eb_isrForCrosstalkSources_855397_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/b0ca8d8f-2cb9-43e5-9044-aa8d5d7a79eb_isrForCrosstalkSources_855397_33.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7f37381f4111394b0156223be5c8137 with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e40042b2-aa82-47f7-af0a-0a375f9a589b_isrForCrosstalkSources_855431_16.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e40042b2-aa82-47f7-af0a-0a375f9a589b_isrForCrosstalkSources_855431_16.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c6298c1ef807a420d4be96ba712bf81 with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d485c358-f358-48ff-b62d-2a8525495034_isrForCrosstalkSources_855426_46.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/d485c358-f358-48ff-b62d-2a8525495034_isrForCrosstalkSources_855426_46.stderr parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea649d8858e6b1265db09d0b78901b1 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2a30f6b8-f97e-40e4-9c90-5e69fa4d1019_isrForCrosstalkSources_855436_33.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/2a30f6b8-f97e-40e4-9c90-5e69fa4d1019_isrForCrosstalkSources_855436_33.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b8d288f39b7e33995aff0bedf954c03 with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7445c029-87a4-4c1b-b18f-7d014e6db990_isrForCrosstalkSources_855392_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7445c029-87a4-4c1b-b18f-7d014e6db990_isrForCrosstalkSources_855392_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:36 proc_lsst.multi:146 [INFO] found job 16964028 in provider astro proc_lsst.multi INFO: found job 16964028 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db2aecbe688c9f1093beb693f0bde8d with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d782645f-0769-4552-8131-7e17c7a76d25_isrForCrosstalkSources_855386_21.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d782645f-0769-4552-8131-7e17c7a76d25_isrForCrosstalkSources_855386_21.stderr parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ca255f4e9b9e7bd77f9d4c6a450ecbf with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f00d04f-ffcf-440e-bd19-a7eca80dfa09_isrForCrosstalkSources_855393_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f00d04f-ffcf-440e-bd19-a7eca80dfa09_isrForCrosstalkSources_855393_11.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61df5346a20a83bedaabc9f94047e07e with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a683caaf-ec1b-4fe5-a7e1-a55349a998d8_isrForCrosstalkSources_855438_50.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a683caaf-ec1b-4fe5-a7e1-a55349a998d8_isrForCrosstalkSources_855438_50.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74cef28cfe556967e45d01c307024e2a with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/20720e6d-15fc-4413-9ccd-9d73dcde77ee_isrForCrosstalkSources_855447_42.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/20720e6d-15fc-4413-9ccd-9d73dcde77ee_isrForCrosstalkSources_855447_42.stderr parsl.dataflow.dflow INFO: Task 1203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a7788273652058d448dfd1813cf9285 with result from task 1203 parsl.dataflow.dflow DEBUG: Task 1203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b584af14-bfd2-4bb5-890a-c29d71e2405f_isrForCrosstalkSources_855444_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/b584af14-bfd2-4bb5-890a-c29d71e2405f_isrForCrosstalkSources_855444_35.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0373f1703fe458d4bec0a30b3445b91 with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e64f316b-077e-421b-a90d-060a1ee486e5_isrForCrosstalkSources_855444_16.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e64f316b-077e-421b-a90d-060a1ee486e5_isrForCrosstalkSources_855444_16.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9405f3e20028dc9b96e881bb30c3ad5 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8ec05c50-fadc-475b-a082-63d4b962ddf9_isrForCrosstalkSources_855423_14.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8ec05c50-fadc-475b-a082-63d4b962ddf9_isrForCrosstalkSources_855423_14.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00deadec1a376f96d46821d54f46c336 with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ecc25b4-1bb4-4f4d-94f5-7bc1dd491d26_isrForCrosstalkSources_855437_22.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ecc25b4-1bb4-4f4d-94f5-7bc1dd491d26_isrForCrosstalkSources_855437_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:36 proc_lsst.multi:146 [INFO] found job 16964029 in provider astro proc_lsst.multi INFO: found job 16964029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c16a046db7f4212251fa93343790e696 with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0fe32d31-55e9-4084-ad9f-a4eae2be976e_isrForCrosstalkSources_855417_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0fe32d31-55e9-4084-ad9f-a4eae2be976e_isrForCrosstalkSources_855417_32.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce2e297fcdd899197e520a94e7753a75 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/eccb9ce4-0cc8-4767-abbb-e34770f95ec5_isrForCrosstalkSources_855402_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/eccb9ce4-0cc8-4767-abbb-e34770f95ec5_isrForCrosstalkSources_855402_34.stderr parsl.dataflow.dflow INFO: Task 1130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93f693501f6c98b1d55c4916268a4f59 with result from task 1130 parsl.dataflow.dflow DEBUG: Task 1130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1fa79e63-2459-4b3d-a903-3cb1cfae4283_isrForCrosstalkSources_855454_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1fa79e63-2459-4b3d-a903-3cb1cfae4283_isrForCrosstalkSources_855454_27.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2dd16a080e08b3a9cf7615886968bd5 with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e92ec7d3-4a70-43f7-b930-d5e54ca3f9a0_isrForCrosstalkSources_855401_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e92ec7d3-4a70-43f7-b930-d5e54ca3f9a0_isrForCrosstalkSources_855401_19.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b57efd6c547e07200ba6424cee3fa7c with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d4962b96-ca04-4176-b73a-05c2a5fd8bdc_isrForCrosstalkSources_855423_24.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d4962b96-ca04-4176-b73a-05c2a5fd8bdc_isrForCrosstalkSources_855423_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:36 proc_lsst.multi:146 [INFO] found job 16964030 in provider astro proc_lsst.multi INFO: found job 16964030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c86ca933d2e87b70308c8e9e2c13cf3 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c7a6ecef-f620-48c3-b745-629fc138cf4f_isrForCrosstalkSources_855406_54.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c7a6ecef-f620-48c3-b745-629fc138cf4f_isrForCrosstalkSources_855406_54.stderr parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d72ffe90fe976cbee7363ce260d4f19f with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3303f96c-5cc7-4288-8a7a-1275d6fa0751_isrForCrosstalkSources_855416_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3303f96c-5cc7-4288-8a7a-1275d6fa0751_isrForCrosstalkSources_855416_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:37 proc_lsst.multi:146 [INFO] found job 16964031 in provider astro proc_lsst.multi INFO: found job 16964031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bff5ab1dde24f6c93d2fe6ed82cc662 with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d9ec2eac-bbaf-440d-bcc5-69854579f95b_isrForCrosstalkSources_855417_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d9ec2eac-bbaf-440d-bcc5-69854579f95b_isrForCrosstalkSources_855417_13.stderr parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9ecd3f34fc6eb10fae9c7a4556bfea6 with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/80270615-cc1a-45e1-83ec-b87c3bef6ae1_isrForCrosstalkSources_855402_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/80270615-cc1a-45e1-83ec-b87c3bef6ae1_isrForCrosstalkSources_855402_25.stderr parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b41371cae0e4733cb928361c1f2c5f with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/815b423f-284b-4880-90d2-7f336a815734_isrForCrosstalkSources_855410_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/815b423f-284b-4880-90d2-7f336a815734_isrForCrosstalkSources_855410_32.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c5aacabfa2f9f812b41c738bc413d7 with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ef69e9c3-9e9c-49c2-9ba7-d4565513ccda_isrForCrosstalkSources_855441_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ef69e9c3-9e9c-49c2-9ba7-d4565513ccda_isrForCrosstalkSources_855441_27.stderr parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8c316c261b4538a1ce83df0b5180432 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/22cae9f3-1644-4cf5-b0ac-dd4e0672b45b_isrForCrosstalkSources_855449_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/22cae9f3-1644-4cf5-b0ac-dd4e0672b45b_isrForCrosstalkSources_855449_25.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed0f8515559cafafc469259c2d94d34 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/54140e4a-50bb-48a3-b2af-269029d5d3fc_isrForCrosstalkSources_855397_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/54140e4a-50bb-48a3-b2af-269029d5d3fc_isrForCrosstalkSources_855397_36.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e5f04dd92b6cd546d337cc69bc27de6 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/1c0bb058-7206-49b9-a342-ca59ed41cc4a_isrForCrosstalkSources_855458_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/1c0bb058-7206-49b9-a342-ca59ed41cc4a_isrForCrosstalkSources_855458_15.stderr parsl.dataflow.dflow INFO: Task 1142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4844e2fc9a9ca7aa667469e2f73560e7 with result from task 1142 parsl.dataflow.dflow DEBUG: Task 1142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8534491-d0eb-4138-a490-5e99f18c12c9_isrForCrosstalkSources_855435_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e8534491-d0eb-4138-a490-5e99f18c12c9_isrForCrosstalkSources_855435_54.stderr parsl.dataflow.dflow INFO: Task 1121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 724dc40fdc6dcf535b5e6122272324cb with result from task 1121 parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:37 proc_lsst.multi:146 [INFO] found job 16964032 in provider astro proc_lsst.multi INFO: found job 16964032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow DEBUG: Task 1121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0b16b48f-883e-4827-bc17-81ea736cdafc_isrForCrosstalkSources_855383_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0b16b48f-883e-4827-bc17-81ea736cdafc_isrForCrosstalkSources_855383_17.stderr parsl.dataflow.dflow INFO: Task 1218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aebd0bba9caba884add3b4b662f99962 with result from task 1218 parsl.dataflow.dflow DEBUG: Task 1218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5ff4434c-f6fe-4d10-a5c6-efd8ae013619_isrForCrosstalkSources_855450_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5ff4434c-f6fe-4d10-a5c6-efd8ae013619_isrForCrosstalkSources_855450_27.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77f3c25e470681bf7620ee71f0f3ffce with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a91459ec-226c-43bf-8d58-7491252dfd14_isrForCrosstalkSources_855401_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a91459ec-226c-43bf-8d58-7491252dfd14_isrForCrosstalkSources_855401_4.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eb01c37272ad9274caa4fe56d75b7c3 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/dedbc0af-3918-4f73-a8e6-d2f921691d43_isrForCrosstalkSources_855412_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/dedbc0af-3918-4f73-a8e6-d2f921691d43_isrForCrosstalkSources_855412_62.stderr parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e1febcbf01ee5a7cc97db09b647a3f2 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b4c5a669-e2c3-4cbd-98bd-9b467bc633bb_isrForCrosstalkSources_855384_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b4c5a669-e2c3-4cbd-98bd-9b467bc633bb_isrForCrosstalkSources_855384_45.stderr parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5869d4bab664cd9370ace862d6a04595 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a5b750eb-c06f-46a4-b45e-48777544e5f4_isrForCrosstalkSources_855445_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a5b750eb-c06f-46a4-b45e-48777544e5f4_isrForCrosstalkSources_855445_40.stderr parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 227472478fe7dbc61f3b32882678696b with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7a28f6b6-bf7a-4872-8ed4-d17ceecb581c_isrForCrosstalkSources_855448_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7a28f6b6-bf7a-4872-8ed4-d17ceecb581c_isrForCrosstalkSources_855448_26.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 492aa3a5cef40485ed7753dced20277a with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c6989876-67e9-4ae8-a6c4-bdc714d4c3d1_isrForCrosstalkSources_855394_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c6989876-67e9-4ae8-a6c4-bdc714d4c3d1_isrForCrosstalkSources_855394_14.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ad4c8d58bbc7678ccb9334faffb473e with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/be68e3a4-28ad-4eb3-b850-d91e98f223dc_isrForCrosstalkSources_855386_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/be68e3a4-28ad-4eb3-b850-d91e98f223dc_isrForCrosstalkSources_855386_39.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 986289a25d81da4e37f0138ea0fc3b95 with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2eac81f-7405-4854-bfe0-a2a9a8c458a1_isrForCrosstalkSources_855402_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2eac81f-7405-4854-bfe0-a2a9a8c458a1_isrForCrosstalkSources_855402_46.stderr parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f191b52ea5d07c18b473e9c33a9d1e8 with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/41ad133e-c83b-4d5d-af88-02a089c1ad7b_isrForCrosstalkSources_855437_36.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/41ad133e-c83b-4d5d-af88-02a089c1ad7b_isrForCrosstalkSources_855437_36.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17d3bd7cc36b31435c893bef02e1a839 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/1e584d68-3d4a-4b60-b6ac-68ff856c3d62_isrForCrosstalkSources_855448_17.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/1e584d68-3d4a-4b60-b6ac-68ff856c3d62_isrForCrosstalkSources_855448_17.stderr parsl.dataflow.dflow INFO: Task 1217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de71658815225ee5a6fc365f787c9473 with result from task 1217 parsl.dataflow.dflow DEBUG: Task 1217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5dd04427-c96d-4b8a-823d-7b2f67234240_isrForCrosstalkSources_855400_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5dd04427-c96d-4b8a-823d-7b2f67234240_isrForCrosstalkSources_855400_13.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c79025349af5622437f87bae7f84cbad with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/8f2543fe-65bd-44f5-a853-c9b9bf29ef04_isrForCrosstalkSources_855389_54.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/8f2543fe-65bd-44f5-a853-c9b9bf29ef04_isrForCrosstalkSources_855389_54.stderr parsl.dataflow.dflow INFO: Task 1122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9955058f234eb769192e69f2c04a08 with result from task 1122 parsl.dataflow.dflow DEBUG: Task 1122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f883ab6-2b11-47eb-ad23-7f369abb5cd7_isrForCrosstalkSources_855415_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f883ab6-2b11-47eb-ad23-7f369abb5cd7_isrForCrosstalkSources_855415_13.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98fea163669ba737bbb74a44abe845b6 with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5c3f2e66-4595-44bf-9e28-0cf3dd7e7445_isrForCrosstalkSources_855440_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5c3f2e66-4595-44bf-9e28-0cf3dd7e7445_isrForCrosstalkSources_855440_42.stderr parsl.dataflow.dflow INFO: Task 1132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d9d088b27c025bbac496b56e3bdce6d with result from task 1132 parsl.dataflow.dflow DEBUG: Task 1132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1838ce99-25bf-47f6-b4ca-29fa3087dcc4_isrForCrosstalkSources_855408_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/1838ce99-25bf-47f6-b4ca-29fa3087dcc4_isrForCrosstalkSources_855408_39.stderr parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62483969f63e2bf7d1bd54eb3ecb661e with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/22e489cb-55fa-41e2-a6f1-53e141ed89fd_isrForCrosstalkSources_855448_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/22e489cb-55fa-41e2-a6f1-53e141ed89fd_isrForCrosstalkSources_855448_28.stderr parsl.dataflow.dflow INFO: Task 1246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce330b30e3d9bc6e34bbc337d3ff7ee2 with result from task 1246 parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:38 proc_lsst.multi:146 [INFO] found job 16964033 in provider astro proc_lsst.multi INFO: found job 16964033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow DEBUG: Task 1246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9e8d44ca-693a-4f5b-a5d5-c802c901a4b7_isrForCrosstalkSources_855437_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9e8d44ca-693a-4f5b-a5d5-c802c901a4b7_isrForCrosstalkSources_855437_26.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e94a632edeee8f9d9a3222d619d444 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6012028d-48ad-46ed-b480-81716905e09b_isrForCrosstalkSources_855408_6.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6012028d-48ad-46ed-b480-81716905e09b_isrForCrosstalkSources_855408_6.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c187dd2dfb7cfea5d8c50910e17ddb2f with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e8b4269f-70df-4894-95c9-52f4239bd589_isrForCrosstalkSources_855444_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e8b4269f-70df-4894-95c9-52f4239bd589_isrForCrosstalkSources_855444_55.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30de47ad580322d6ea96d9a7615f8198 with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8b7a5750-72b2-4ae6-912e-824ec01a43af_isrForCrosstalkSources_855427_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8b7a5750-72b2-4ae6-912e-824ec01a43af_isrForCrosstalkSources_855427_1.stderr parsl.dataflow.dflow INFO: Task 1138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c0158cdf885fdcc31673b6d9bb0ccab with result from task 1138 parsl.dataflow.dflow DEBUG: Task 1138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/deaaacd0-0fd2-4ea8-b21d-c712c559a72f_isrForCrosstalkSources_855389_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/deaaacd0-0fd2-4ea8-b21d-c712c559a72f_isrForCrosstalkSources_855389_21.stderr parsl.dataflow.dflow INFO: Task 1135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68c0531972ff9e9c1eff138cd72d9587 with result from task 1135 parsl.dataflow.dflow DEBUG: Task 1135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/aea48172-435c-416f-a598-3de0b1f510ed_isrForCrosstalkSources_855430_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/aea48172-435c-416f-a598-3de0b1f510ed_isrForCrosstalkSources_855430_20.stderr parsl.dataflow.dflow INFO: Task 1173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a18e420695e2b8ee670f36e349d49cb6 with result from task 1173 parsl.dataflow.dflow DEBUG: Task 1173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5e493692-ebd7-434b-b20b-ed83268c8305_isrForCrosstalkSources_855427_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5e493692-ebd7-434b-b20b-ed83268c8305_isrForCrosstalkSources_855427_2.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b211dae0b5869e189b3c430da9d68aa with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0b081061-3ea9-4979-a642-55a132124b65_isrForCrosstalkSources_855396_48.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0b081061-3ea9-4979-a642-55a132124b65_isrForCrosstalkSources_855396_48.stderr parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f6fda9b10296aa2d6c67cf16f6c5f6 with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ee1f1e81-6565-4617-acd9-f0f8a5276a41_isrForCrosstalkSources_855386_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ee1f1e81-6565-4617-acd9-f0f8a5276a41_isrForCrosstalkSources_855386_2.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc279ce8315853bc94c77c69980004e with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/87483b3b-3c29-499b-ab0d-772bf0654bea_isrForCrosstalkSources_855386_41.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/87483b3b-3c29-499b-ab0d-772bf0654bea_isrForCrosstalkSources_855386_41.stderr parsl.dataflow.dflow INFO: Task 1206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 550a47a2aa1fc294b4e09c2df3fe551f with result from task 1206 parsl.dataflow.dflow DEBUG: Task 1206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/2d81d349-c386-487b-8912-43b8aa375a71_isrForCrosstalkSources_855423_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/2d81d349-c386-487b-8912-43b8aa375a71_isrForCrosstalkSources_855423_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:38 proc_lsst.multi:146 [INFO] found job 16964034 in provider astro proc_lsst.multi INFO: found job 16964034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcda995593e9f2b919cdb46748e91645 with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/627b19c7-727a-4d5f-aa1b-211ef026afa7_isrForCrosstalkSources_855428_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/627b19c7-727a-4d5f-aa1b-211ef026afa7_isrForCrosstalkSources_855428_24.stderr parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f94a745dd5b42442d0dffbfac2a9b22 with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/64f9faa2-4ee7-4e57-b677-449176b7292c_isrForCrosstalkSources_855427_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/64f9faa2-4ee7-4e57-b677-449176b7292c_isrForCrosstalkSources_855427_14.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b28e226d0ab96c13c4b8d5e5c9d5c7e4 with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/472abddd-d06a-4cc8-83f4-ff81179a352b_isrForCrosstalkSources_855395_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/472abddd-d06a-4cc8-83f4-ff81179a352b_isrForCrosstalkSources_855395_24.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9534301c2268e1e9ee9d41458e91d42f with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/400914f0-0555-499a-b6dd-49acda4cccd6_isrForCrosstalkSources_855452_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/400914f0-0555-499a-b6dd-49acda4cccd6_isrForCrosstalkSources_855452_30.stderr parsl.dataflow.dflow INFO: Task 1253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20deb78122750428266dfa1a0830478d with result from task 1253 parsl.dataflow.dflow DEBUG: Task 1253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5e5567ac-b4f1-4c90-a7d4-d09bd8c7bb75_isrForCrosstalkSources_855407_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5e5567ac-b4f1-4c90-a7d4-d09bd8c7bb75_isrForCrosstalkSources_855407_34.stderr parsl.dataflow.dflow INFO: Task 1223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65037fba7ec2cd4864c3b36900e8fcf8 with result from task 1223 parsl.dataflow.dflow DEBUG: Task 1223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/1bc03eb4-02bf-4af8-ae49-3b01a5a4c720_isrForCrosstalkSources_855433_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/1bc03eb4-02bf-4af8-ae49-3b01a5a4c720_isrForCrosstalkSources_855433_21.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed086ea6990acb4a6ec8c026cbd18267 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3d663ab8-634c-494b-8624-28d990f75140_isrForCrosstalkSources_855379_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3d663ab8-634c-494b-8624-28d990f75140_isrForCrosstalkSources_855379_11.stderr parsl.dataflow.dflow INFO: Task 1134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24a95af06982687dff66f117e433266 with result from task 1134 parsl.dataflow.dflow DEBUG: Task 1134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a8d749d8-9e29-4cc2-a4a7-64a320d4b652_isrForCrosstalkSources_855416_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a8d749d8-9e29-4cc2-a4a7-64a320d4b652_isrForCrosstalkSources_855416_48.stderr parsl.dataflow.dflow INFO: Task 1140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f14c2d1b14051e0444270d12facd526a with result from task 1140 parsl.dataflow.dflow DEBUG: Task 1140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aec4830f-02d3-4866-9860-af4adb13d960_isrForCrosstalkSources_855449_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aec4830f-02d3-4866-9860-af4adb13d960_isrForCrosstalkSources_855449_56.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fba2fe7edff44b06d1f361c838a70e4a with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/e329ce36-305b-42bb-b401-dc21e2c58560_isrForCrosstalkSources_855397_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/e329ce36-305b-42bb-b401-dc21e2c58560_isrForCrosstalkSources_855397_16.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49be6d4ba7cd48ba20a5c8846e1d668d with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b50d0cea-61dc-4300-90aa-476bdf48030b_isrForCrosstalkSources_855456_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b50d0cea-61dc-4300-90aa-476bdf48030b_isrForCrosstalkSources_855456_6.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e663776a5f332a5cad8ac5ae9c7de93f with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ab5c8e-c125-437f-8932-7e7b6ab87c68_isrForCrosstalkSources_855437_6.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/69ab5c8e-c125-437f-8932-7e7b6ab87c68_isrForCrosstalkSources_855437_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:39 proc_lsst.multi:146 [INFO] found job 16964035 in provider astro proc_lsst.multi INFO: found job 16964035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef8eda19d6623cccbd2c9b3c53e4c53 with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9aefc00f-938a-4689-a081-7a441e06654d_isrForCrosstalkSources_855398_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9aefc00f-938a-4689-a081-7a441e06654d_isrForCrosstalkSources_855398_7.stderr parsl.dataflow.dflow INFO: Task 1211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4410f52ceaa5c731f0c9eb33efb7b09 with result from task 1211 parsl.dataflow.dflow DEBUG: Task 1211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e0fae397-6983-47ac-afc7-15bc45c0e0d1_isrForCrosstalkSources_855401_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/e0fae397-6983-47ac-afc7-15bc45c0e0d1_isrForCrosstalkSources_855401_20.stderr parsl.dataflow.dflow INFO: Task 1215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dd5f3c3da8ca721d4853e38f8df1fd1 with result from task 1215 parsl.dataflow.dflow DEBUG: Task 1215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c12f0398-413f-4d7d-8f65-7eee3bab327c_isrForCrosstalkSources_855458_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c12f0398-413f-4d7d-8f65-7eee3bab327c_isrForCrosstalkSources_855458_40.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cced6e0071fd0a51f6c6a093c4dafad8 with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f38dc340-3ae4-4e3f-9096-a409ab34a825_isrForCrosstalkSources_855387_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f38dc340-3ae4-4e3f-9096-a409ab34a825_isrForCrosstalkSources_855387_29.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ad3d93e1a2b3f48a1ef04d1dc0baf4 with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/83d7af3b-6755-4d4a-b8d3-0dfa0223b197_isrForCrosstalkSources_855428_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/83d7af3b-6755-4d4a-b8d3-0dfa0223b197_isrForCrosstalkSources_855428_33.stderr parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df2336a4962cb895c037539deff0aae4 with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c2b77439-8bce-47f5-ae57-0cfb957edbd8_isrForCrosstalkSources_855418_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c2b77439-8bce-47f5-ae57-0cfb957edbd8_isrForCrosstalkSources_855418_51.stderr parsl.dataflow.dflow INFO: Task 1181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 584368854afa4e8b185723668ea8dc29 with result from task 1181 parsl.dataflow.dflow DEBUG: Task 1181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/81faaf59-1024-40a9-b8c6-e8bdef12984e_isrForCrosstalkSources_855381_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/81faaf59-1024-40a9-b8c6-e8bdef12984e_isrForCrosstalkSources_855381_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:39 proc_lsst.multi:146 [INFO] found job 16964036 in provider astro proc_lsst.multi INFO: found job 16964036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3256e6fe01adf554845f6716a6daa407 with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/7fe35ea8-d553-42db-9d97-ff5c215f84ee_isrForCrosstalkSources_855389_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/7fe35ea8-d553-42db-9d97-ff5c215f84ee_isrForCrosstalkSources_855389_8.stderr parsl.dataflow.dflow INFO: Task 1214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e71c8d515ab0b746bb4fcfcd66e1d0b with result from task 1214 parsl.dataflow.dflow DEBUG: Task 1214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7518cfeb-27cc-4b13-b03a-e87027f3c5d3_isrForCrosstalkSources_855436_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7518cfeb-27cc-4b13-b03a-e87027f3c5d3_isrForCrosstalkSources_855436_57.stderr parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df65ca9769c5c17e1fd4239952bbe9c2 with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d108866b-88be-4877-b1f2-7897ef955691_isrForCrosstalkSources_855429_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d108866b-88be-4877-b1f2-7897ef955691_isrForCrosstalkSources_855429_52.stderr parsl.dataflow.dflow INFO: Task 1236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af9ede8186d17fb2c7fffa5c8add067b with result from task 1236 parsl.dataflow.dflow DEBUG: Task 1236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/caa0b4f8-ec6f-4e66-935f-cdec40b5ac03_isrForCrosstalkSources_855445_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/caa0b4f8-ec6f-4e66-935f-cdec40b5ac03_isrForCrosstalkSources_855445_1.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14119ac812742d42cbaad03a8a15b22c with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/3ac2d5e2-90dd-4c31-99f3-99de29ac36f7_isrForCrosstalkSources_855393_49.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/3ac2d5e2-90dd-4c31-99f3-99de29ac36f7_isrForCrosstalkSources_855393_49.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13d08055da46e7d17d25388ba90c7940 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/78d9b8aa-8f55-4ae0-b31c-ffe39cbcc159_isrForCrosstalkSources_855452_41.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/78d9b8aa-8f55-4ae0-b31c-ffe39cbcc159_isrForCrosstalkSources_855452_41.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 979e2cf3a8e2e1c9395e158c620b2abb with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/889c171b-b589-4e55-9421-d44ac91c25a6_isrForCrosstalkSources_855395_5.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/889c171b-b589-4e55-9421-d44ac91c25a6_isrForCrosstalkSources_855395_5.stderr parsl.dataflow.dflow INFO: Task 1183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1560c7f133a4839605ed500cab247d46 with result from task 1183 parsl.dataflow.dflow DEBUG: Task 1183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6235e50a-ec5a-4b28-8e27-161faff1c473_isrForCrosstalkSources_855425_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6235e50a-ec5a-4b28-8e27-161faff1c473_isrForCrosstalkSources_855425_25.stderr parsl.dataflow.dflow INFO: Task 1194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2e524753ee47c78f642979dcba6350 with result from task 1194 parsl.dataflow.dflow DEBUG: Task 1194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cfa45cba-8a51-4a81-988c-f80eb521c9b2_isrForCrosstalkSources_855418_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/cfa45cba-8a51-4a81-988c-f80eb521c9b2_isrForCrosstalkSources_855418_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:40 proc_lsst.multi:146 [INFO] found job 16964037 in provider astro proc_lsst.multi INFO: found job 16964037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b632be506b2be26b576a6ba810ebce55 with result from task 1224 parsl.dataflow.dflow DEBUG: Task 1224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/82751ce6-69c1-40aa-bce7-84f433669989_isrForCrosstalkSources_855427_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/82751ce6-69c1-40aa-bce7-84f433669989_isrForCrosstalkSources_855427_25.stderr parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0a4e79e7db0da5371c96555103d406e with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5d134dcd-a396-4ec0-b023-32bdb11ca2cd_isrForCrosstalkSources_855401_54.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5d134dcd-a396-4ec0-b023-32bdb11ca2cd_isrForCrosstalkSources_855401_54.stderr parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c7cf26c35b8fd84fd5951861f89e4d1 with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc0d9a60-65f7-4085-a1b7-016c594e16fc_isrForCrosstalkSources_855421_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc0d9a60-65f7-4085-a1b7-016c594e16fc_isrForCrosstalkSources_855421_43.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 449ffc2c91902600bcf0958ba21378b5 with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f828aa24-09ed-4c7c-b071-ab4195ffb726_isrForCrosstalkSources_855405_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f828aa24-09ed-4c7c-b071-ab4195ffb726_isrForCrosstalkSources_855405_44.stderr parsl.dataflow.dflow INFO: Task 1205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e26da2313f6811d11786f91e6d09e53 with result from task 1205 parsl.dataflow.dflow DEBUG: Task 1205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7a0896fa-fe54-4228-a11b-a6401e718cec_isrForCrosstalkSources_855422_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7a0896fa-fe54-4228-a11b-a6401e718cec_isrForCrosstalkSources_855422_12.stderr parsl.dataflow.dflow INFO: Task 1201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70ddbaa6bd21b0d16424e40e6d5d4ad7 with result from task 1201 parsl.dataflow.dflow DEBUG: Task 1201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fea916fe-485c-4a2c-8e02-210bc1e5cc8e_isrForCrosstalkSources_855387_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fea916fe-485c-4a2c-8e02-210bc1e5cc8e_isrForCrosstalkSources_855387_12.stderr parsl.dataflow.dflow INFO: Task 1193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e7cb949215b23671255fb1249415038 with result from task 1193 parsl.dataflow.dflow DEBUG: Task 1193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/bab3f2ea-a0b7-4951-b32f-118b5f9b5334_isrForCrosstalkSources_855409_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/bab3f2ea-a0b7-4951-b32f-118b5f9b5334_isrForCrosstalkSources_855409_17.stderr parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d4698bc7cd76b8949fe91922969b1ae with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d0df42b0-3d85-4691-813e-ec1df406e33d_isrForCrosstalkSources_855443_29.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d0df42b0-3d85-4691-813e-ec1df406e33d_isrForCrosstalkSources_855443_29.stderr parsl.dataflow.dflow INFO: Task 1141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b36d05d220529c8982e47cf8fdbc4a5 with result from task 1141 parsl.dataflow.dflow DEBUG: Task 1141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/706dbfe2-44bf-4875-8e65-c44dda7dd8fd_isrForCrosstalkSources_855405_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/706dbfe2-44bf-4875-8e65-c44dda7dd8fd_isrForCrosstalkSources_855405_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:40 proc_lsst.multi:146 [INFO] found job 16964038 in provider astro proc_lsst.multi INFO: found job 16964038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 750e2cf7eef826c46facb659a6964610 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/49735925-507c-46a6-a6c8-17451bc265f1_isrForCrosstalkSources_855389_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/49735925-507c-46a6-a6c8-17451bc265f1_isrForCrosstalkSources_855389_30.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e598218b9033ffa0cd374bc84611a0a6 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/931a46cc-df00-4b2e-9ac7-05ac181d3eeb_isrForCrosstalkSources_855424_42.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/931a46cc-df00-4b2e-9ac7-05ac181d3eeb_isrForCrosstalkSources_855424_42.stderr parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18b06279aaaa17fedf8fd2e0448cf41b with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/dec4f71c-6b8c-4127-a647-7a7384510efc_isrForCrosstalkSources_855452_60.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/dec4f71c-6b8c-4127-a647-7a7384510efc_isrForCrosstalkSources_855452_60.stderr parsl.dataflow.dflow INFO: Task 1192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69a9abb2b799fecf02e40dce306767e2 with result from task 1192 parsl.dataflow.dflow DEBUG: Task 1192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/79e28332-e8c6-41bf-ae0c-e583bb410fd4_isrForCrosstalkSources_855385_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/79e28332-e8c6-41bf-ae0c-e583bb410fd4_isrForCrosstalkSources_855385_33.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry decdf5b21b9f51f66e884be92651c6b6 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ba848d1b-a638-4b2e-a409-d7fc68914177_isrForCrosstalkSources_855415_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ba848d1b-a638-4b2e-a409-d7fc68914177_isrForCrosstalkSources_855415_34.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8558893146b8ddacf576a833d72a68a with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b151aa41-3705-4240-9e96-37f796e80961_isrForCrosstalkSources_855426_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b151aa41-3705-4240-9e96-37f796e80961_isrForCrosstalkSources_855426_24.stderr parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5481eb2534b79e02a13c1120c9f1dc22 with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e8517d88-6e50-4b8a-8fa5-2da09f27bf10_isrForCrosstalkSources_855455_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e8517d88-6e50-4b8a-8fa5-2da09f27bf10_isrForCrosstalkSources_855455_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:41 proc_lsst.multi:146 [INFO] found job 16964039 in provider astro proc_lsst.multi INFO: found job 16964039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78016b4195e1fa5e3cb8f671fae08fb7 with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f66c4a18-9360-491a-a18c-76fe6985cf58_isrForCrosstalkSources_855384_48.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f66c4a18-9360-491a-a18c-76fe6985cf58_isrForCrosstalkSources_855384_48.stderr parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68988617701e6367cf790b04cc106610 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ee867dab-ef11-4046-ba98-47638f079966_isrForCrosstalkSources_855451_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ee867dab-ef11-4046-ba98-47638f079966_isrForCrosstalkSources_855451_13.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39555b7c204ed8d96e312b981fbaafd3 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c1c08088-92bf-46c5-b0d3-cff1528364cf_isrForCrosstalkSources_855459_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c1c08088-92bf-46c5-b0d3-cff1528364cf_isrForCrosstalkSources_855459_57.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5da03ab37e225d09b51245ccec600f2f with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e8b8871d-1c02-451c-80fe-f62eec14727f_isrForCrosstalkSources_855418_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e8b8871d-1c02-451c-80fe-f62eec14727f_isrForCrosstalkSources_855418_45.stderr parsl.dataflow.dflow INFO: Task 1190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48d1c03b4d11477e1f22e9eebecef10d with result from task 1190 parsl.dataflow.dflow DEBUG: Task 1190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d28d27a7-f59d-4014-abfa-9a243af49d0a_isrForCrosstalkSources_855388_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/d28d27a7-f59d-4014-abfa-9a243af49d0a_isrForCrosstalkSources_855388_21.stderr parsl.dataflow.dflow INFO: Task 1116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df96a937f4e51e48b1995bd250516106 with result from task 1116 parsl.dataflow.dflow DEBUG: Task 1116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/759eb945-049d-43a1-b972-1732c605feed_isrForCrosstalkSources_855398_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/759eb945-049d-43a1-b972-1732c605feed_isrForCrosstalkSources_855398_24.stderr parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6522aa8f5180b96e6a856f2e589d3325 with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/05af7bd4-a040-41fe-ad2a-23b8c3190c20_isrForCrosstalkSources_855399_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/05af7bd4-a040-41fe-ad2a-23b8c3190c20_isrForCrosstalkSources_855399_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:41 proc_lsst.multi:146 [INFO] found job 16964040 in provider astro proc_lsst.multi INFO: found job 16964040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3beb8edda1df68d3794eed7134b8af97 with result from task 1189 parsl.dataflow.dflow DEBUG: Task 1189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8982692a-4a36-4902-9e91-36b00d56d5a9_isrForCrosstalkSources_855412_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8982692a-4a36-4902-9e91-36b00d56d5a9_isrForCrosstalkSources_855412_20.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f456015a40cc6aca8e4fda0d536d0cfa with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8b878db-defd-4f5d-aafe-3640fd649d4e_isrForCrosstalkSources_855449_58.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8b878db-defd-4f5d-aafe-3640fd649d4e_isrForCrosstalkSources_855449_58.stderr parsl.dataflow.dflow INFO: Task 1210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862c1ed4d08b971fc6e877a5e3ba5d44 with result from task 1210 parsl.dataflow.dflow DEBUG: Task 1210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f4744bcf-b883-4bde-a77b-be1f55433f2f_isrForCrosstalkSources_855434_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f4744bcf-b883-4bde-a77b-be1f55433f2f_isrForCrosstalkSources_855434_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:41 proc_lsst.multi:146 [INFO] found job 16964041 in provider astro proc_lsst.multi INFO: found job 16964041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0a413487dace40068931c8a0e7a8596 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0cab1a3-1b54-4ec2-ad12-db799382a378_isrForCrosstalkSources_855384_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0cab1a3-1b54-4ec2-ad12-db799382a378_isrForCrosstalkSources_855384_49.stderr parsl.dataflow.dflow INFO: Task 1213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db2df1abb3c3ff0a1b29d5465f3fdb7c with result from task 1213 parsl.dataflow.dflow DEBUG: Task 1213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/84b6eeea-257b-4946-858d-f36cde3bd42d_isrForCrosstalkSources_855417_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/84b6eeea-257b-4946-858d-f36cde3bd42d_isrForCrosstalkSources_855417_6.stderr parsl.dataflow.dflow INFO: Task 1321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 952fd26600d659852eba2b0555428f40 with result from task 1321 parsl.dataflow.dflow DEBUG: Task 1321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/66258046-064e-45d3-834a-cc3974227351_isrForCrosstalkSources_855413_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/66258046-064e-45d3-834a-cc3974227351_isrForCrosstalkSources_855413_8.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bce0dc07fee54f2990dd9ede525a283 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5b335c02-955e-46d7-9364-a7023074e310_isrForCrosstalkSources_855383_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5b335c02-955e-46d7-9364-a7023074e310_isrForCrosstalkSources_855383_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:42 proc_lsst.multi:146 [INFO] found job 16964042 in provider astro proc_lsst.multi INFO: found job 16964042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb01295f4bfe35c4a300ef42aeb693f2 with result from task 1191 parsl.dataflow.dflow DEBUG: Task 1191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/868d5c9f-f70c-4ac7-a4ae-963041d7478b_isrForCrosstalkSources_855395_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/868d5c9f-f70c-4ac7-a4ae-963041d7478b_isrForCrosstalkSources_855395_36.stderr parsl.dataflow.dflow INFO: Task 1175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06dda96dc385b8e774334110e4704a41 with result from task 1175 parsl.dataflow.dflow DEBUG: Task 1175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/be647352-b0f6-495f-bd20-f795d5c3d258_isrForCrosstalkSources_855383_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/be647352-b0f6-495f-bd20-f795d5c3d258_isrForCrosstalkSources_855383_14.stderr parsl.dataflow.dflow INFO: Task 1182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c13c7811c36e632acf066e70b179651d with result from task 1182 parsl.dataflow.dflow DEBUG: Task 1182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c3c87755-a3b8-4f8e-925f-9e21d8c4a121_isrForCrosstalkSources_855439_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c3c87755-a3b8-4f8e-925f-9e21d8c4a121_isrForCrosstalkSources_855439_14.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d3bf3e932060de2c6d9aea034eb97b1 with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a8a3181-9a55-4e88-9639-d15d33bca52d_isrForCrosstalkSources_855454_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/4a8a3181-9a55-4e88-9639-d15d33bca52d_isrForCrosstalkSources_855454_9.stderr parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e1af3babc5f842085440e15395417f with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/75b7fbf7-c25f-409f-8d19-b6020a6f9460_isrForCrosstalkSources_855456_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/75b7fbf7-c25f-409f-8d19-b6020a6f9460_isrForCrosstalkSources_855456_1.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1f34552c6bab50e5f2e7b82b9713899 with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3731a6c2-5435-4413-80a8-a5d49a58f801_isrForCrosstalkSources_855443_15.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3731a6c2-5435-4413-80a8-a5d49a58f801_isrForCrosstalkSources_855443_15.stderr parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 358adf50f34560504eef102eab6c14f8 with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fe259fb4-16ad-4963-9da4-95ebc7decd26_isrForCrosstalkSources_855454_62.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fe259fb4-16ad-4963-9da4-95ebc7decd26_isrForCrosstalkSources_855454_62.stderr parsl.dataflow.dflow INFO: Task 1184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1c80f16e9ecd744df141042008bec23 with result from task 1184 parsl.dataflow.dflow DEBUG: Task 1184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d20d862f-31a0-4727-82d4-b9e92c59e45f_isrForCrosstalkSources_855402_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d20d862f-31a0-4727-82d4-b9e92c59e45f_isrForCrosstalkSources_855402_11.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41d2f8673ec8e3f230a288bae00c3372 with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/27d364a7-0461-4238-bec5-aa4f53230a48_isrForCrosstalkSources_855383_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/27d364a7-0461-4238-bec5-aa4f53230a48_isrForCrosstalkSources_855383_24.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b604b8a39b73d4c219e38bb593934a23 with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6de70b17-a552-4f25-ae54-9c37d944226b_isrForCrosstalkSources_855432_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6de70b17-a552-4f25-ae54-9c37d944226b_isrForCrosstalkSources_855432_11.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d496ec53b763f1d7f747613e8baff56 with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/aca1156e-b77f-424f-9238-ed4f20d988dc_isrForCrosstalkSources_855395_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/aca1156e-b77f-424f-9238-ed4f20d988dc_isrForCrosstalkSources_855395_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:43 proc_lsst.multi:146 [INFO] found job 16964043 in provider astro proc_lsst.multi INFO: found job 16964043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6370d85165c10af27a53ed2b2ad2dd with result from task 1212 parsl.dataflow.dflow DEBUG: Task 1212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/cabc9533-9833-400a-92df-7079a9599de9_isrForCrosstalkSources_855397_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/cabc9533-9833-400a-92df-7079a9599de9_isrForCrosstalkSources_855397_54.stderr parsl.dataflow.dflow INFO: Task 1198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8addfd2fe864ced18bc36a77d5984130 with result from task 1198 parsl.dataflow.dflow DEBUG: Task 1198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f6b3e8ea-e1e3-44b8-a4fc-c603c1487823_isrForCrosstalkSources_855424_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f6b3e8ea-e1e3-44b8-a4fc-c603c1487823_isrForCrosstalkSources_855424_39.stderr parsl.dataflow.dflow INFO: Task 1127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a54dafa123c35803c75816849ce338ff with result from task 1127 parsl.dataflow.dflow DEBUG: Task 1127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d17a0f09-f881-4f86-b4b4-6052f8dd1ff7_isrForCrosstalkSources_855385_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d17a0f09-f881-4f86-b4b4-6052f8dd1ff7_isrForCrosstalkSources_855385_37.stderr parsl.dataflow.dflow INFO: Task 1177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb8ff660a3d90fadda77fa97cf34f5d9 with result from task 1177 parsl.dataflow.dflow DEBUG: Task 1177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fab20f97-03cd-41e1-936c-1a8730238e42_isrForCrosstalkSources_855439_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fab20f97-03cd-41e1-936c-1a8730238e42_isrForCrosstalkSources_855439_51.stderr parsl.dataflow.dflow INFO: Task 1199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1a14a223f82f686d19401120d19ed36 with result from task 1199 parsl.dataflow.dflow DEBUG: Task 1199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/425ccc48-9d68-41c6-8738-801192920fe9_isrForCrosstalkSources_855458_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/425ccc48-9d68-41c6-8738-801192920fe9_isrForCrosstalkSources_855458_50.stderr parsl.dataflow.dflow INFO: Task 1117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a888bdb2224ec0a7a104a5c697829fcf with result from task 1117 parsl.dataflow.dflow DEBUG: Task 1117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/87cf0a73-be29-4263-af15-95055e9588e6_isrForCrosstalkSources_855423_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/87cf0a73-be29-4263-af15-95055e9588e6_isrForCrosstalkSources_855423_10.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13d41b70ddfff4680d5d037febe3ada9 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fe56cd1f-9530-45a1-ac8a-a576e7576d88_isrForCrosstalkSources_855415_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fe56cd1f-9530-45a1-ac8a-a576e7576d88_isrForCrosstalkSources_855415_14.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eb360727bfa0cb460e0ef9f612fe2c6 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9232741-4cae-49bb-9c2c-5b5a1d68be54_isrForCrosstalkSources_855458_21.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9232741-4cae-49bb-9c2c-5b5a1d68be54_isrForCrosstalkSources_855458_21.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc314c5316625692121eb03cc0ab2daf with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/43260daa-8b8d-4487-8eb1-f43ec10ee80d_isrForCrosstalkSources_855445_42.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/43260daa-8b8d-4487-8eb1-f43ec10ee80d_isrForCrosstalkSources_855445_42.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eca5584ec8cd9ee8f9a036c837420f10 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/eea61c08-6922-4003-8650-a767a9558bf1_isrForCrosstalkSources_855403_35.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/eea61c08-6922-4003-8650-a767a9558bf1_isrForCrosstalkSources_855403_35.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3a1fb65e83ba1b20edf9b0ea6a24e23 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c01be294-e5ec-45b3-bc96-6ee9f3ac5e67_isrForCrosstalkSources_855410_17.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c01be294-e5ec-45b3-bc96-6ee9f3ac5e67_isrForCrosstalkSources_855410_17.stderr parsl.dataflow.dflow INFO: Task 1180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f287408744b0deac110176cbfb0dabf with result from task 1180 parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:43 proc_lsst.multi:146 [INFO] found job 16964044 in provider astro proc_lsst.multi INFO: found job 16964044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow DEBUG: Task 1180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4a8be49b-23f7-4ff8-a3d0-0cbe5c009de1_isrForCrosstalkSources_855404_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4a8be49b-23f7-4ff8-a3d0-0cbe5c009de1_isrForCrosstalkSources_855404_50.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aa3b26aef5912b52d2b613067b723eb with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/47d87d7c-9972-4cd6-9dbc-04eaa846a59e_isrForCrosstalkSources_855433_33.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/47d87d7c-9972-4cd6-9dbc-04eaa846a59e_isrForCrosstalkSources_855433_33.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1829e36d50304f77a5af274bf402c87 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/666d9687-4adf-406a-bfd9-192d7dc988d1_isrForCrosstalkSources_855432_57.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/666d9687-4adf-406a-bfd9-192d7dc988d1_isrForCrosstalkSources_855432_57.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e07b810ad3345019171e2002056567e with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0327a516-3af2-41be-9340-bc30c24bfe6a_isrForCrosstalkSources_855454_14.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0327a516-3af2-41be-9340-bc30c24bfe6a_isrForCrosstalkSources_855454_14.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9b4dce7349c67a57ed38538c8b8431c with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c9816fe7-c0a5-43f9-8ec8-7b3b278d79f4_isrForCrosstalkSources_855414_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c9816fe7-c0a5-43f9-8ec8-7b3b278d79f4_isrForCrosstalkSources_855414_31.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e5fad7bc0632ebf2ad41fe7d9a17cd1 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e406454c-883c-4d73-bbf0-73a05ba3a07f_isrForCrosstalkSources_855431_51.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e406454c-883c-4d73-bbf0-73a05ba3a07f_isrForCrosstalkSources_855431_51.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e444aaae71fa6afc0a29bdb265906cad with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/07436a77-6852-43ec-bc79-32ea8d34fa1a_isrForCrosstalkSources_855420_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/07436a77-6852-43ec-bc79-32ea8d34fa1a_isrForCrosstalkSources_855420_3.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 669be3fdfddda4d4197ab0beab89cf9d with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/85810983-268b-4b1d-bd17-db567dd8c2ab_isrForCrosstalkSources_855432_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/85810983-268b-4b1d-bd17-db567dd8c2ab_isrForCrosstalkSources_855432_29.stderr parsl.dataflow.dflow INFO: Task 1126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065df532442a6d0b1c42e35c0b7707a0 with result from task 1126 parsl.dataflow.dflow DEBUG: Task 1126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/0c38afec-96c6-4dd1-bc5c-d9a0ec2e2dd1_isrForCrosstalkSources_855455_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/0c38afec-96c6-4dd1-bc5c-d9a0ec2e2dd1_isrForCrosstalkSources_855455_56.stderr parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6261e80d357160d4214f2541fd574c6d with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c26e2a8c-a401-495f-8150-b6b54eabcd6b_isrForCrosstalkSources_855422_59.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c26e2a8c-a401-495f-8150-b6b54eabcd6b_isrForCrosstalkSources_855422_59.stderr parsl.dataflow.dflow INFO: Task 1128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da36ce8169262ccae04ad21b9136c059 with result from task 1128 parsl.dataflow.dflow DEBUG: Task 1128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/66fad533-283d-4790-b9af-c2370326dfcf_isrForCrosstalkSources_855398_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/66fad533-283d-4790-b9af-c2370326dfcf_isrForCrosstalkSources_855398_54.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6397e00c590c909938430d8b402ea6e6 with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fc95917d-8a9a-417e-8f32-125706e88bb9_isrForCrosstalkSources_855458_41.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fc95917d-8a9a-417e-8f32-125706e88bb9_isrForCrosstalkSources_855458_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:43 proc_lsst.multi:146 [INFO] found job 16964045 in provider astro proc_lsst.multi INFO: found job 16964045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 713e1c9215c272781eb100c686f69c02 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/9d909381-dd66-4523-b306-7590cfab7025_isrForCrosstalkSources_855459_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/9d909381-dd66-4523-b306-7590cfab7025_isrForCrosstalkSources_855459_39.stderr parsl.dataflow.dflow INFO: Task 1144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 335758e25517741cc73291d98f32505a with result from task 1144 parsl.dataflow.dflow DEBUG: Task 1144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1323489a-957c-4bf0-9312-dbfbaaf5bd53_isrForCrosstalkSources_855425_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1323489a-957c-4bf0-9312-dbfbaaf5bd53_isrForCrosstalkSources_855425_24.stderr parsl.dataflow.dflow INFO: Task 1238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3f2fad297082ebce7d64c300f4642b5 with result from task 1238 parsl.dataflow.dflow DEBUG: Task 1238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3989f1cc-6bc6-4d77-b871-6cc215c4b3af_isrForCrosstalkSources_855391_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3989f1cc-6bc6-4d77-b871-6cc215c4b3af_isrForCrosstalkSources_855391_26.stderr parsl.dataflow.dflow INFO: Task 1195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1ccf371eaf725bb49b6f69843bea56 with result from task 1195 parsl.dataflow.dflow DEBUG: Task 1195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/10bd851f-d5d4-46b3-9de8-0920fe0fe860_isrForCrosstalkSources_855386_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/10bd851f-d5d4-46b3-9de8-0920fe0fe860_isrForCrosstalkSources_855386_31.stderr parsl.dataflow.dflow INFO: Task 1227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed33819195f57dbd8a558d3ca857d64f with result from task 1227 parsl.dataflow.dflow DEBUG: Task 1227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/878a2737-0235-427b-9d5c-372b0c718b28_isrForCrosstalkSources_855433_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/878a2737-0235-427b-9d5c-372b0c718b28_isrForCrosstalkSources_855433_23.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a91b5afb8e7ba6760bc9009364ca2f6 with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/9724a414-697c-4ed0-949a-d20d730a2238_isrForCrosstalkSources_855413_59.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/9724a414-697c-4ed0-949a-d20d730a2238_isrForCrosstalkSources_855413_59.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b9c64b8c0a3395466ab65f283210e56 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0afe2387-7330-4e7b-bdad-4c6f900d2555_isrForCrosstalkSources_855382_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0afe2387-7330-4e7b-bdad-4c6f900d2555_isrForCrosstalkSources_855382_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:44 proc_lsst.multi:146 [INFO] found job 16964046 in provider astro proc_lsst.multi INFO: found job 16964046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a505e3c5c20280465076be8f3505cbdd with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/3c347239-bea0-47e8-9509-aa9e0b43da1f_isrForCrosstalkSources_855435_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/3c347239-bea0-47e8-9509-aa9e0b43da1f_isrForCrosstalkSources_855435_2.stderr parsl.dataflow.dflow INFO: Task 1120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2a2965b877a7fabe8dae212e70d2bfe with result from task 1120 parsl.dataflow.dflow DEBUG: Task 1120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c69be069-5463-4635-af75-8f171a176a5a_isrForCrosstalkSources_855423_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c69be069-5463-4635-af75-8f171a176a5a_isrForCrosstalkSources_855423_52.stderr parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd8ade43539aa8433c34610045a897d with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/1796020a-831c-47a9-9c47-de1dfee55cee_isrForCrosstalkSources_855432_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/1796020a-831c-47a9-9c47-de1dfee55cee_isrForCrosstalkSources_855432_18.stderr parsl.dataflow.dflow INFO: Task 1209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42fed2058d186953d3946fd5cdb2cc0d with result from task 1209 parsl.dataflow.dflow DEBUG: Task 1209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d683273d-47e1-4111-8d50-6c1e5045959d_isrForCrosstalkSources_855432_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d683273d-47e1-4111-8d50-6c1e5045959d_isrForCrosstalkSources_855432_55.stderr parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17829e454b4ff80560728486828a6256 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d0012d71-bc90-48bf-baf0-46c42a585549_isrForCrosstalkSources_855454_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d0012d71-bc90-48bf-baf0-46c42a585549_isrForCrosstalkSources_855454_1.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21462bf78e9e59d035138eb1533f73d7 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e07aae58-85b2-4b4b-afbe-60d9e5f73f34_isrForCrosstalkSources_855386_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e07aae58-85b2-4b4b-afbe-60d9e5f73f34_isrForCrosstalkSources_855386_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:45 proc_lsst.multi:146 [INFO] found job 16964047 in provider astro proc_lsst.multi INFO: found job 16964047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e065be288de888d003671c43e792013 with result from task 1188 parsl.dataflow.dflow DEBUG: Task 1188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a497bf4d-79b5-47d1-a69d-2bc2a246df28_isrForCrosstalkSources_855405_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a497bf4d-79b5-47d1-a69d-2bc2a246df28_isrForCrosstalkSources_855405_37.stderr parsl.dataflow.dflow INFO: Task 1225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e200266c29d916ad6103656ccb0bf7ff with result from task 1225 parsl.dataflow.dflow DEBUG: Task 1225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a5db1e0c-4f3a-4260-a680-9d6ee9338bd3_isrForCrosstalkSources_855457_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/a5db1e0c-4f3a-4260-a680-9d6ee9338bd3_isrForCrosstalkSources_855457_22.stderr parsl.dataflow.dflow INFO: Task 1174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73282b2470ef17254375996306fb7a2c with result from task 1174 parsl.dataflow.dflow DEBUG: Task 1174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3595cb71-35f4-4b7b-bf04-7b80baa24194_isrForCrosstalkSources_855425_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3595cb71-35f4-4b7b-bf04-7b80baa24194_isrForCrosstalkSources_855425_18.stderr parsl.dataflow.dflow INFO: Task 1185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f75e82f5cf4f854da1e2abee7dd9b922 with result from task 1185 parsl.dataflow.dflow DEBUG: Task 1185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9d93a122-c6f5-4c30-8fc4-8881fa70bc26_isrForCrosstalkSources_855400_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9d93a122-c6f5-4c30-8fc4-8881fa70bc26_isrForCrosstalkSources_855400_16.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9abf96fdf329e316ea6d35dc66119517 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/23a3d736-2fe4-40ef-a0d3-8384d2461371_isrForCrosstalkSources_855411_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/23a3d736-2fe4-40ef-a0d3-8384d2461371_isrForCrosstalkSources_855411_23.stderr parsl.dataflow.dflow INFO: Task 1176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fd48f5f6869dfa81b5c1c572f5c89ef with result from task 1176 parsl.dataflow.dflow DEBUG: Task 1176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7b3424ee-145a-4d68-af9f-cbcbd0dc3228_isrForCrosstalkSources_855380_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7b3424ee-145a-4d68-af9f-cbcbd0dc3228_isrForCrosstalkSources_855380_6.stderr parsl.dataflow.dflow INFO: Task 1240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 476551efaf90a4d00c66f0cd8052613c with result from task 1240 parsl.dataflow.dflow DEBUG: Task 1240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/fb9c569a-fb8f-4806-90c6-20d02f4fdf13_isrForCrosstalkSources_855380_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/fb9c569a-fb8f-4806-90c6-20d02f4fdf13_isrForCrosstalkSources_855380_18.stderr parsl.dataflow.dflow INFO: Task 1290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4e47769de0327c9bb34101b1c2e6b47 with result from task 1290 parsl.dataflow.dflow DEBUG: Task 1290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/69c811f6-5c4f-409e-b85f-999edfc51c1e_isrForCrosstalkSources_855388_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/69c811f6-5c4f-409e-b85f-999edfc51c1e_isrForCrosstalkSources_855388_17.stderr parsl.dataflow.dflow INFO: Task 1291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f922797f58e4bff9570c044dccd1eb98 with result from task 1291 parsl.dataflow.dflow DEBUG: Task 1291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/66d4495f-f947-45cd-b601-25f8e60772a8_isrForCrosstalkSources_855423_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/66d4495f-f947-45cd-b601-25f8e60772a8_isrForCrosstalkSources_855423_6.stderr parsl.dataflow.dflow INFO: Task 1259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 590877f57967da9bab95515ea9e28962 with result from task 1259 parsl.dataflow.dflow DEBUG: Task 1259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ef3779b7-9f46-46f0-aadb-66770c6e689a_isrForCrosstalkSources_855386_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ef3779b7-9f46-46f0-aadb-66770c6e689a_isrForCrosstalkSources_855386_51.stderr parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 198f4c72bdbd3101c6975590b8004e36 with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/eaa24ccb-c435-4b8b-be58-1e9d46e5bee2_isrForCrosstalkSources_855459_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/eaa24ccb-c435-4b8b-be58-1e9d46e5bee2_isrForCrosstalkSources_855459_12.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c29fca88d1290f0a52dcde151b53aba8 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4794b0c9-372f-49ed-870c-80a8cc27c852_isrForCrosstalkSources_855457_54.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/4794b0c9-372f-49ed-870c-80a8cc27c852_isrForCrosstalkSources_855457_54.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5771aaa339b5d242d86a4bae37feaf08 with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/897087bf-9ffa-4950-9c34-3113d3204a4c_isrForCrosstalkSources_855381_40.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/897087bf-9ffa-4950-9c34-3113d3204a4c_isrForCrosstalkSources_855381_40.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575ea955f8aaa94d692a15bf8b36c57e with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dcb50532-0c8a-4824-8bd9-75124288d3b4_isrForCrosstalkSources_855387_44.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dcb50532-0c8a-4824-8bd9-75124288d3b4_isrForCrosstalkSources_855387_44.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b7579879bbee6e5a1d643fdd8afaff with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6554d286-eb4b-4e3c-8961-62f63576a2fd_isrForCrosstalkSources_855452_33.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6554d286-eb4b-4e3c-8961-62f63576a2fd_isrForCrosstalkSources_855452_33.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 187f660dcf7f4aebfd912abf9ad89ae7 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/cc1b6ddd-7074-4752-b17b-dd79649c5a41_isrForCrosstalkSources_855423_30.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/cc1b6ddd-7074-4752-b17b-dd79649c5a41_isrForCrosstalkSources_855423_30.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11b5db86479e9b6bb8e6430982120898 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8c1eed94-7ee6-47d1-9660-946d82be5665_isrForCrosstalkSources_855379_60.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8c1eed94-7ee6-47d1-9660-946d82be5665_isrForCrosstalkSources_855379_60.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3121c58f8f7a90a6473d9d14f44ecac6 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/50fdfd62-f6bf-4928-b5f2-b6a9bc734db9_isrForCrosstalkSources_855452_53.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/50fdfd62-f6bf-4928-b5f2-b6a9bc734db9_isrForCrosstalkSources_855452_53.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2fe8c88d66564a12e61f0993a3caea9 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c05f929b-0925-44bf-a835-62eaff6e6ed0_isrForCrosstalkSources_855417_12.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c05f929b-0925-44bf-a835-62eaff6e6ed0_isrForCrosstalkSources_855417_12.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1a10abd21bd14ec7de1f7c6c6285c0 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/28c8a0c6-45f5-4353-8210-8c0261927a5d_isrForCrosstalkSources_855452_44.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/28c8a0c6-45f5-4353-8210-8c0261927a5d_isrForCrosstalkSources_855452_44.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e23cc21d3ceeee4256839f85286c3c with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4eca5b12-fa24-4509-97d5-1b9efe3b6b74_isrForCrosstalkSources_855440_26.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4eca5b12-fa24-4509-97d5-1b9efe3b6b74_isrForCrosstalkSources_855440_26.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2b3cb7e980f75dbf38951dbc8083322 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/93cecb44-fb90-46c5-99d1-374323a2dd69_isrForCrosstalkSources_855436_26.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/93cecb44-fb90-46c5-99d1-374323a2dd69_isrForCrosstalkSources_855436_26.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8dbdc7a5b3434397451c68f1df181f7 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f6c121fb-9999-4892-b2c9-1a9eb2294e9e_isrForCrosstalkSources_855421_32.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f6c121fb-9999-4892-b2c9-1a9eb2294e9e_isrForCrosstalkSources_855421_32.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7f05605c67e15ee45337303b553cb77 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/65926a1c-8ccd-42dc-87d7-8f8e990bfcc9_isrForCrosstalkSources_855403_8.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/65926a1c-8ccd-42dc-87d7-8f8e990bfcc9_isrForCrosstalkSources_855403_8.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e546cde3b351086b9bb9d4f19783dbb with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/6c8163dd-68df-43c1-9d83-2abd6f6af59f_isrForCrosstalkSources_855423_23.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/6c8163dd-68df-43c1-9d83-2abd6f6af59f_isrForCrosstalkSources_855423_23.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31ad507c4cb18974a9595d42cbde4052 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/538b97cd-52ab-42c0-8956-7ff0944dfd6c_isrForCrosstalkSources_855381_51.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/538b97cd-52ab-42c0-8956-7ff0944dfd6c_isrForCrosstalkSources_855381_51.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e38b216ddfc70c58ac595094d0d9ba4 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4b869a4f-f288-49ef-bd7d-255b1a5fc436_isrForCrosstalkSources_855439_17.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4b869a4f-f288-49ef-bd7d-255b1a5fc436_isrForCrosstalkSources_855439_17.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd149253911e77206f8bf4f99771326d with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c3e7f722-2877-4b03-b780-d8b3ff56bb23_isrForCrosstalkSources_855423_29.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c3e7f722-2877-4b03-b780-d8b3ff56bb23_isrForCrosstalkSources_855423_29.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a994c33ef3ae34036a588196caeb972b with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6a791e39-583e-4ee1-bf01-21dfbeec9c4f_isrForCrosstalkSources_855415_12.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6a791e39-583e-4ee1-bf01-21dfbeec9c4f_isrForCrosstalkSources_855415_12.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e96cbdbb75fe80b05b624e7805eb59a with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/23afdc9a-b056-4326-8da5-9deec398016d_isrForCrosstalkSources_855439_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/23afdc9a-b056-4326-8da5-9deec398016d_isrForCrosstalkSources_855439_59.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e3de34d5aaa91c108534005e8e14736 with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/921ea495-ced7-412a-9077-9bd594360a30_isrForCrosstalkSources_855442_10.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/921ea495-ced7-412a-9077-9bd594360a30_isrForCrosstalkSources_855442_10.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee0a77bd9f43c5d64b72a99b206f0fb7 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/986369e1-0403-4bf2-b3a9-82d3b5c23986_isrForCrosstalkSources_855391_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/986369e1-0403-4bf2-b3a9-82d3b5c23986_isrForCrosstalkSources_855391_31.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d51655d1cd8c3ff4fdcdadd8a8c3671 with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afdba5bf-6b4d-4652-9643-a83d5a5e68d1_isrForCrosstalkSources_855452_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afdba5bf-6b4d-4652-9643-a83d5a5e68d1_isrForCrosstalkSources_855452_43.stderr parsl.dataflow.dflow INFO: Task 1147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15a046f467531d98fc5916ffd7715115 with result from task 1147 parsl.dataflow.dflow DEBUG: Task 1147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/373a00b9-a18b-4cbc-bd99-2126c83db6f0_isrForCrosstalkSources_855391_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/373a00b9-a18b-4cbc-bd99-2126c83db6f0_isrForCrosstalkSources_855391_54.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ab92ee6b150d12512bc1df595bf8768 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/af954b42-e94b-4e79-940c-4b89478c1f74_isrForCrosstalkSources_855456_38.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/af954b42-e94b-4e79-940c-4b89478c1f74_isrForCrosstalkSources_855456_38.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24e82faddd17aa2ab3c2ace78843e010 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3cf31f25-dd00-4dcc-b0ba-9d2aa849ad7e_isrForCrosstalkSources_855413_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3cf31f25-dd00-4dcc-b0ba-9d2aa849ad7e_isrForCrosstalkSources_855413_50.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5834dbc9b78b3eda4ec2f2a7815efdd with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/26af3ad5-f2b9-4441-a660-f9746383f2e1_isrForCrosstalkSources_855431_48.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/26af3ad5-f2b9-4441-a660-f9746383f2e1_isrForCrosstalkSources_855431_48.stderr parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5410676942865e68c85a2eccc6e9000c with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a209234d-0e16-4976-9aea-9c0e04c44f16_isrForCrosstalkSources_855455_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a209234d-0e16-4976-9aea-9c0e04c44f16_isrForCrosstalkSources_855455_48.stderr parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e630123b05c2305d57dc88330cb43829 with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5154d4ec-3f27-4e01-aa8a-af0be50caa1c_isrForCrosstalkSources_855384_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5154d4ec-3f27-4e01-aa8a-af0be50caa1c_isrForCrosstalkSources_855384_11.stderr parsl.dataflow.dflow INFO: Task 1118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5a1a883ba54fa954625d7ff5b8f291 with result from task 1118 parsl.dataflow.dflow DEBUG: Task 1118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/80f88bf2-b828-4777-a467-03e892576052_isrForCrosstalkSources_855411_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/80f88bf2-b828-4777-a467-03e892576052_isrForCrosstalkSources_855411_58.stderr parsl.dataflow.dflow INFO: Task 1239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d20c5265a55511dcddae5fc1ea21964 with result from task 1239 parsl.dataflow.dflow DEBUG: Task 1239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/87fba31a-90c1-4a64-bd24-0f06218bcc3b_isrForCrosstalkSources_855428_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/87fba31a-90c1-4a64-bd24-0f06218bcc3b_isrForCrosstalkSources_855428_29.stderr parsl.dataflow.dflow INFO: Task 1124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2449637782c2cd97d5d9c935559585d3 with result from task 1124 parsl.dataflow.dflow DEBUG: Task 1124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0b54c141-8372-4726-a29f-94fd20a5631e_isrForCrosstalkSources_855427_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0b54c141-8372-4726-a29f-94fd20a5631e_isrForCrosstalkSources_855427_47.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44c735e76ed99b0438d279539ef7a341 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/55991eaa-8e87-4583-98ac-1c38cb33675f_isrForCrosstalkSources_855394_43.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/55991eaa-8e87-4583-98ac-1c38cb33675f_isrForCrosstalkSources_855394_43.stderr parsl.dataflow.dflow INFO: Task 1447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc8db8fe8df21449092e9f3823996a1 with result from task 1447 parsl.dataflow.dflow DEBUG: Task 1447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3f95a577-8ca7-4760-b556-53782ba8ea44_isrForCrosstalkSources_855457_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3f95a577-8ca7-4760-b556-53782ba8ea44_isrForCrosstalkSources_855457_33.stderr parsl.dataflow.dflow INFO: Task 1256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df93cd285f0b722cb8cde74e7833e01a with result from task 1256 parsl.dataflow.dflow DEBUG: Task 1256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aab08b9-d77b-472a-85ae-688c221b66a7_isrForCrosstalkSources_855407_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aab08b9-d77b-472a-85ae-688c221b66a7_isrForCrosstalkSources_855407_62.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc8222849a3db2df1f981cd1a47a8334 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cf622e73-c8ae-4b83-aed0-8b9da427bf93_isrForCrosstalkSources_855422_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cf622e73-c8ae-4b83-aed0-8b9da427bf93_isrForCrosstalkSources_855422_58.stderr parsl.dataflow.dflow INFO: Task 1257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92246350372cb78aa278c828d3acfcf7 with result from task 1257 parsl.dataflow.dflow DEBUG: Task 1257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32436abd-1f2b-4e0b-9ad9-5d7c11e52fea_isrForCrosstalkSources_855448_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32436abd-1f2b-4e0b-9ad9-5d7c11e52fea_isrForCrosstalkSources_855448_42.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b84d7e4dfba3368e8df0ce8da3c7e3b1 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5485db8c-1a4e-434f-8dbd-9711f16bf8f9_isrForCrosstalkSources_855402_54.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5485db8c-1a4e-434f-8dbd-9711f16bf8f9_isrForCrosstalkSources_855402_54.stderr parsl.dataflow.dflow INFO: Task 1414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fc459cc93ece955470aeb23b96d982e with result from task 1414 parsl.dataflow.dflow DEBUG: Task 1414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9baa78df-503a-4ae8-8662-e201e750ccfe_isrForCrosstalkSources_855449_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9baa78df-503a-4ae8-8662-e201e750ccfe_isrForCrosstalkSources_855449_13.stderr parsl.dataflow.dflow INFO: Task 1226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63b756e5fb953db9c356f5ba493e44a9 with result from task 1226 parsl.dataflow.dflow DEBUG: Task 1226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0a36c45d-f532-44d5-8cd8-8d84afdd7872_isrForCrosstalkSources_855387_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0a36c45d-f532-44d5-8cd8-8d84afdd7872_isrForCrosstalkSources_855387_56.stderr parsl.dataflow.dflow INFO: Task 1251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a91564580602ba592b4595ad9118792 with result from task 1251 parsl.dataflow.dflow DEBUG: Task 1251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2e7dd9a7-ed06-4bf7-8e31-63c5860d4031_isrForCrosstalkSources_855408_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2e7dd9a7-ed06-4bf7-8e31-63c5860d4031_isrForCrosstalkSources_855408_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:47 proc_lsst.multi:146 [INFO] found job 16964048 in provider astro proc_lsst.multi INFO: found job 16964048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4af98d32b13db471be23b8980173f1ee with result from task 1202 parsl.dataflow.dflow DEBUG: Task 1202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/97ff951f-02b1-46be-9bc9-f8042a70e81d_isrForCrosstalkSources_855436_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/97ff951f-02b1-46be-9bc9-f8042a70e81d_isrForCrosstalkSources_855436_11.stderr parsl.dataflow.dflow INFO: Task 1338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e12341fbaff432f035649708ee31045 with result from task 1338 parsl.dataflow.dflow DEBUG: Task 1338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/450f2d9a-3fc4-45a2-b381-dcac0725f567_isrForCrosstalkSources_855411_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/450f2d9a-3fc4-45a2-b381-dcac0725f567_isrForCrosstalkSources_855411_26.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 006182ed6abee01d7da2dbaf78e0d041 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/246d087b-6a35-4eb9-ade6-a6dd5750a346_isrForCrosstalkSources_855394_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/246d087b-6a35-4eb9-ade6-a6dd5750a346_isrForCrosstalkSources_855394_44.stderr parsl.dataflow.dflow INFO: Task 1258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca8354383ea2d50d5de6b68221551b5 with result from task 1258 parsl.dataflow.dflow DEBUG: Task 1258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/435b7919-834e-4815-8b81-06de05c3df97_isrForCrosstalkSources_855406_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/435b7919-834e-4815-8b81-06de05c3df97_isrForCrosstalkSources_855406_62.stderr parsl.dataflow.dflow INFO: Task 1365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e294bc1bce9043ed3ffdff81fa1bcd7f with result from task 1365 parsl.dataflow.dflow DEBUG: Task 1365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a0f2c5e-403d-434d-82f9-fe023a5488c8_isrForCrosstalkSources_855447_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a0f2c5e-403d-434d-82f9-fe023a5488c8_isrForCrosstalkSources_855447_14.stderr parsl.dataflow.dflow INFO: Task 1357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73082f0226473e4e7421184fe3fd8954 with result from task 1357 parsl.dataflow.dflow DEBUG: Task 1357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce4930e6-40aa-46c2-9794-a442c009ecec_isrForCrosstalkSources_855445_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce4930e6-40aa-46c2-9794-a442c009ecec_isrForCrosstalkSources_855445_30.stderr parsl.dataflow.dflow INFO: Task 1446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00189ad4317562cd7487e0e62c7df7ab with result from task 1446 parsl.dataflow.dflow DEBUG: Task 1446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bb54b1ea-8904-4286-a9f5-08062e5590f4_isrForCrosstalkSources_855450_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bb54b1ea-8904-4286-a9f5-08062e5590f4_isrForCrosstalkSources_855450_15.stderr parsl.dataflow.dflow INFO: Task 1345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a42649c415d41ba6447f08c604e82c34 with result from task 1345 parsl.dataflow.dflow DEBUG: Task 1345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b8e94b91-557e-4253-8892-0b05dd3d0ff2_isrForCrosstalkSources_855459_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b8e94b91-557e-4253-8892-0b05dd3d0ff2_isrForCrosstalkSources_855459_38.stderr parsl.dataflow.dflow INFO: Task 1350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 403e5387278958e2c63057135a2231f2 with result from task 1350 parsl.dataflow.dflow DEBUG: Task 1350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7a51e99e-84d9-42fc-8f64-b2c260b90ae1_isrForCrosstalkSources_855410_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7a51e99e-84d9-42fc-8f64-b2c260b90ae1_isrForCrosstalkSources_855410_21.stderr parsl.dataflow.dflow INFO: Task 1145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d4493f2e6c256235f409f3985c7447 with result from task 1145 parsl.dataflow.dflow DEBUG: Task 1145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c7b2b421-bf8f-4774-a193-b8385a1c6108_isrForCrosstalkSources_855426_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/c7b2b421-bf8f-4774-a193-b8385a1c6108_isrForCrosstalkSources_855426_49.stderr parsl.dataflow.dflow INFO: Task 1431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119b5588753bd15df0d3fc7f05232383 with result from task 1431 parsl.dataflow.dflow DEBUG: Task 1431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/983ad80a-7048-442a-98d7-cd0a18558365_isrForCrosstalkSources_855457_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/983ad80a-7048-442a-98d7-cd0a18558365_isrForCrosstalkSources_855457_9.stderr parsl.dataflow.dflow INFO: Task 1186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02b0b790cbc1168f60cc51815c3cb959 with result from task 1186 parsl.dataflow.dflow DEBUG: Task 1186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3af24d6e-bb65-4423-9ddd-97543fb87ca8_isrForCrosstalkSources_855433_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3af24d6e-bb65-4423-9ddd-97543fb87ca8_isrForCrosstalkSources_855433_47.stderr parsl.dataflow.dflow INFO: Task 1361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58b34d30c9c6de2f3748c4b79787443 with result from task 1361 parsl.dataflow.dflow DEBUG: Task 1361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2d4d7d9e-1457-4548-81ff-b5ff576278fb_isrForCrosstalkSources_855379_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2d4d7d9e-1457-4548-81ff-b5ff576278fb_isrForCrosstalkSources_855379_32.stderr parsl.dataflow.dflow INFO: Task 1129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69e9aca02c05248a3fd87046996afa6e with result from task 1129 parsl.dataflow.dflow DEBUG: Task 1129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4dd62509-55cd-4279-8f13-b28f2ebf8894_isrForCrosstalkSources_855390_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4dd62509-55cd-4279-8f13-b28f2ebf8894_isrForCrosstalkSources_855390_44.stderr parsl.dataflow.dflow INFO: Task 1362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7ee2741fabf885ec65ecec2b63d7685 with result from task 1362 parsl.dataflow.dflow DEBUG: Task 1362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/35fc9e0a-95d6-42dd-9eba-186d09ed0c51_isrForCrosstalkSources_855386_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/35fc9e0a-95d6-42dd-9eba-186d09ed0c51_isrForCrosstalkSources_855386_29.stderr parsl.dataflow.dflow INFO: Task 1326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c5ca5d1ff0629fa0e992ce0c50ce648 with result from task 1326 parsl.dataflow.dflow DEBUG: Task 1326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36360b6b-5220-44ca-ad27-1f524aff7bc5_isrForCrosstalkSources_855388_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/36360b6b-5220-44ca-ad27-1f524aff7bc5_isrForCrosstalkSources_855388_12.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34c70c85923476c34351f7440c4d3587 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/68e3bea7-ed25-4f25-b043-e855fc921a37_isrForCrosstalkSources_855447_30.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/68e3bea7-ed25-4f25-b043-e855fc921a37_isrForCrosstalkSources_855447_30.stderr parsl.dataflow.dflow INFO: Task 1394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 114960dae12685aab9762ebd54ab1b1e with result from task 1394 parsl.dataflow.dflow DEBUG: Task 1394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/dfe31f11-c25a-472a-8f18-ab2f69edea6c_isrForCrosstalkSources_855386_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/dfe31f11-c25a-472a-8f18-ab2f69edea6c_isrForCrosstalkSources_855386_62.stderr parsl.dataflow.dflow INFO: Task 1323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa7a9053d5733e8fd79efe67127be21b with result from task 1323 parsl.dataflow.dflow DEBUG: Task 1323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53e0f64b-f39b-4fdb-81cc-654da276a321_isrForCrosstalkSources_855389_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/53e0f64b-f39b-4fdb-81cc-654da276a321_isrForCrosstalkSources_855389_4.stderr parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 289b743d8bacfaa5639ec60955394830 with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a2d4c2c7-232d-43be-b975-5d29aaaafa88_isrForCrosstalkSources_855390_53.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a2d4c2c7-232d-43be-b975-5d29aaaafa88_isrForCrosstalkSources_855390_53.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc40ef06ca8160f8a9c5cd9510ca1662 with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ec524951-1239-4d76-b901-c757e6e42f8f_isrForCrosstalkSources_855438_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ec524951-1239-4d76-b901-c757e6e42f8f_isrForCrosstalkSources_855438_43.stderr parsl.dataflow.dflow INFO: Task 1237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d57e80e3b3e062d7cd2d12c02ae1514 with result from task 1237 parsl.dataflow.dflow DEBUG: Task 1237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/9a283e5b-17dd-47fa-b159-044bd9995f77_isrForCrosstalkSources_855417_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/9a283e5b-17dd-47fa-b159-044bd9995f77_isrForCrosstalkSources_855417_37.stderr parsl.dataflow.dflow INFO: Task 1456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b3e17614fca04deeda2a6970434cd5 with result from task 1456 parsl.dataflow.dflow DEBUG: Task 1456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a03d39b8-26ce-4462-8e2f-51b4721f0239_isrForCrosstalkSources_855396_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a03d39b8-26ce-4462-8e2f-51b4721f0239_isrForCrosstalkSources_855396_40.stderr parsl.dataflow.dflow INFO: Task 1308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f569bf108948972212f022c3986e631 with result from task 1308 parsl.dataflow.dflow DEBUG: Task 1308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b347679f-030f-409c-b40f-f987a3839c93_isrForCrosstalkSources_855385_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b347679f-030f-409c-b40f-f987a3839c93_isrForCrosstalkSources_855385_17.stderr parsl.dataflow.dflow INFO: Task 1327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d27ebf4f6bb2ce6d8b006cef96d7fd23 with result from task 1327 parsl.dataflow.dflow DEBUG: Task 1327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5e296d09-2400-4317-81b0-38927d6b8965_isrForCrosstalkSources_855403_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5e296d09-2400-4317-81b0-38927d6b8965_isrForCrosstalkSources_855403_22.stderr parsl.dataflow.dflow INFO: Task 1139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1aae9cfab05d2f3836da634beb30bd1 with result from task 1139 parsl.dataflow.dflow DEBUG: Task 1139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2c6991cb-711f-4dba-b934-8399f9d7aedb_isrForCrosstalkSources_855456_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2c6991cb-711f-4dba-b934-8399f9d7aedb_isrForCrosstalkSources_855456_52.stderr parsl.dataflow.dflow INFO: Task 1137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9dc3df4bde4b436b015fdc96d21a557 with result from task 1137 parsl.dataflow.dflow DEBUG: Task 1137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/041752d6-a314-4bb4-8d67-c19bfbf13efe_isrForCrosstalkSources_855414_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/041752d6-a314-4bb4-8d67-c19bfbf13efe_isrForCrosstalkSources_855414_57.stderr parsl.dataflow.dflow INFO: Task 1449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73fb52b0319f9bd4fe1685e21145bd4c with result from task 1449 parsl.dataflow.dflow DEBUG: Task 1449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a4133cd7-e61b-441a-86b5-ab667ecf0460_isrForCrosstalkSources_855409_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a4133cd7-e61b-441a-86b5-ab667ecf0460_isrForCrosstalkSources_855409_19.stderr parsl.dataflow.dflow INFO: Task 1348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e745900141c44ec7aa500fd2fbf8d80f with result from task 1348 parsl.dataflow.dflow DEBUG: Task 1348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d634ac82-289c-4924-a14b-212fdd20d983_isrForCrosstalkSources_855389_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d634ac82-289c-4924-a14b-212fdd20d983_isrForCrosstalkSources_855389_51.stderr parsl.dataflow.dflow INFO: Task 1143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b437a9c9b85d8899f142c3e3b8446ba with result from task 1143 parsl.dataflow.dflow DEBUG: Task 1143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/cc1f4ad7-bebc-41e3-a8e7-91b0d0fc0d78_isrForCrosstalkSources_855433_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/cc1f4ad7-bebc-41e3-a8e7-91b0d0fc0d78_isrForCrosstalkSources_855433_53.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3adeb7db68ff3d05ebf35a1e1eceebc with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9728a97f-ddc2-40ea-8a4f-97e2318f118e_isrForCrosstalkSources_855431_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9728a97f-ddc2-40ea-8a4f-97e2318f118e_isrForCrosstalkSources_855431_62.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfdebae938b1f3b57590797a0e3f220e with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c966fa5f-56f0-45a3-83d6-7c1f3d08ea40_isrForCrosstalkSources_855411_59.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c966fa5f-56f0-45a3-83d6-7c1f3d08ea40_isrForCrosstalkSources_855411_59.stderr parsl.dataflow.dflow INFO: Task 1222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d333e2a0e1369ae23a40ee1505986d0 with result from task 1222 parsl.dataflow.dflow DEBUG: Task 1222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3b87263d-ed4f-4e33-906e-8819f0622d3c_isrForCrosstalkSources_855450_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3b87263d-ed4f-4e33-906e-8819f0622d3c_isrForCrosstalkSources_855450_44.stderr parsl.dataflow.dflow INFO: Task 1319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beae336106f4114d4e6afbbb5dee4fce with result from task 1319 parsl.dataflow.dflow DEBUG: Task 1319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c96de887-7906-416a-8580-0064b3231660_isrForCrosstalkSources_855425_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/c96de887-7906-416a-8580-0064b3231660_isrForCrosstalkSources_855425_30.stderr parsl.dataflow.dflow INFO: Task 1268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fbaded75a2fa5d4a21079d02b439d81 with result from task 1268 parsl.dataflow.dflow DEBUG: Task 1268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cbb8fb59-c55a-4a0e-b46b-02ff60d65002_isrForCrosstalkSources_855411_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/cbb8fb59-c55a-4a0e-b46b-02ff60d65002_isrForCrosstalkSources_855411_6.stderr parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03b837fb516532b379608b09944ec222 with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/78c1ac3b-60b7-48e1-ac58-59177eca5308_isrForCrosstalkSources_855422_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/78c1ac3b-60b7-48e1-ac58-59177eca5308_isrForCrosstalkSources_855422_42.stderr parsl.dataflow.dflow INFO: Task 1438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeab1534aea73e34fa361e8c85c1dade with result from task 1438 parsl.dataflow.dflow DEBUG: Task 1438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eae4c1d2-35ef-454a-8bf6-f0bf4b51fa98_isrForCrosstalkSources_855396_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eae4c1d2-35ef-454a-8bf6-f0bf4b51fa98_isrForCrosstalkSources_855396_13.stderr parsl.dataflow.dflow INFO: Task 1364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1453f334805afb56c4bdfb26aa421f7c with result from task 1364 parsl.dataflow.dflow DEBUG: Task 1364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/d09e157e-6c81-49c2-aa64-b2765fb8550f_isrForCrosstalkSources_855411_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/d09e157e-6c81-49c2-aa64-b2765fb8550f_isrForCrosstalkSources_855411_1.stderr parsl.dataflow.dflow INFO: Task 1178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5812165e70eeeeeddac31a0f2af2d670 with result from task 1178 parsl.dataflow.dflow DEBUG: Task 1178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ce26efc5-5cbd-48d3-8afb-53f9cd8e56c9_isrForCrosstalkSources_855395_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/ce26efc5-5cbd-48d3-8afb-53f9cd8e56c9_isrForCrosstalkSources_855395_6.stderr parsl.dataflow.dflow INFO: Task 1292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e41f72b033614310a51989418da064a with result from task 1292 parsl.dataflow.dflow DEBUG: Task 1292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e700dd17-da2d-4931-a63c-419dc8e9d769_isrForCrosstalkSources_855413_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e700dd17-da2d-4931-a63c-419dc8e9d769_isrForCrosstalkSources_855413_46.stderr parsl.dataflow.dflow INFO: Task 1294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b73cb6c92e3a6ee614fa87fa4f8f7703 with result from task 1294 parsl.dataflow.dflow DEBUG: Task 1294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a7e87389-be4c-462f-8cad-bbae3ab9ef96_isrForCrosstalkSources_855453_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a7e87389-be4c-462f-8cad-bbae3ab9ef96_isrForCrosstalkSources_855453_19.stderr parsl.dataflow.dflow INFO: Task 1384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a51e039eb28229999d7c5c6609e02f5e with result from task 1384 parsl.dataflow.dflow DEBUG: Task 1384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ab4ffbb6-0541-485d-939c-8677e00ee08e_isrForCrosstalkSources_855431_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ab4ffbb6-0541-485d-939c-8677e00ee08e_isrForCrosstalkSources_855431_41.stderr parsl.dataflow.dflow INFO: Task 1265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1ef4d928b98aa29fc8799c31eeb9d89 with result from task 1265 parsl.dataflow.dflow DEBUG: Task 1265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e2013fcb-b0d8-4fd7-898d-2caba0fd2192_isrForCrosstalkSources_855402_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e2013fcb-b0d8-4fd7-898d-2caba0fd2192_isrForCrosstalkSources_855402_42.stderr parsl.dataflow.dflow INFO: Task 1358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d2c3eb7a7728adf974a88d97629dc6a with result from task 1358 parsl.dataflow.dflow DEBUG: Task 1358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8ece8090-74a7-4c73-a4b6-6ff6c7abbf75_isrForCrosstalkSources_855453_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8ece8090-74a7-4c73-a4b6-6ff6c7abbf75_isrForCrosstalkSources_855453_14.stderr parsl.dataflow.dflow INFO: Task 1296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68846659de6255d445a7893baa67e2b1 with result from task 1296 parsl.dataflow.dflow DEBUG: Task 1296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b855dccf-8243-406c-ab3d-0632aa139dbf_isrForCrosstalkSources_855453_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b855dccf-8243-406c-ab3d-0632aa139dbf_isrForCrosstalkSources_855453_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:49 proc_lsst.multi:146 [INFO] found job 16964049 in provider astro proc_lsst.multi INFO: found job 16964049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ba81aa59e74133efc810069a12471f with result from task 1455 parsl.dataflow.dflow DEBUG: Task 1455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/aaf9f718-1e71-464c-9a3f-dd5e84ff72e4_isrForCrosstalkSources_855400_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/aaf9f718-1e71-464c-9a3f-dd5e84ff72e4_isrForCrosstalkSources_855400_11.stderr parsl.dataflow.dflow INFO: Task 1464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 130570862c65fbba3c1303bb7f6fcfc2 with result from task 1464 parsl.dataflow.dflow DEBUG: Task 1464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/502bcdce-8158-4a21-b841-13c02f26d487_isrForCrosstalkSources_855412_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/502bcdce-8158-4a21-b841-13c02f26d487_isrForCrosstalkSources_855412_36.stderr parsl.dataflow.dflow INFO: Task 1339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a7cb1ed1c4ec7bf297908ff7354692 with result from task 1339 parsl.dataflow.dflow DEBUG: Task 1339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/75a5e783-2258-43c0-ac84-ad827e5d847a_isrForCrosstalkSources_855430_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/75a5e783-2258-43c0-ac84-ad827e5d847a_isrForCrosstalkSources_855430_54.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91826ff4640e4e9b420b47ec7ffd8e38 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9fdfd991-5c23-4736-9bda-5e3cc0ece19f_isrForCrosstalkSources_855408_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9fdfd991-5c23-4736-9bda-5e3cc0ece19f_isrForCrosstalkSources_855408_57.stderr parsl.dataflow.dflow INFO: Task 1219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41a201d0a3b81222cb4c743f2d0487df with result from task 1219 parsl.dataflow.dflow DEBUG: Task 1219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/36802aad-889d-4f8f-86d5-45143952a23c_isrForCrosstalkSources_855451_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/36802aad-889d-4f8f-86d5-45143952a23c_isrForCrosstalkSources_855451_21.stderr parsl.dataflow.dflow INFO: Task 1293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da09a8f95b40b92c787737af7187af4b with result from task 1293 parsl.dataflow.dflow DEBUG: Task 1293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/febf0934-de7f-4dd6-9047-02fd4cb13244_isrForCrosstalkSources_855453_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/febf0934-de7f-4dd6-9047-02fd4cb13244_isrForCrosstalkSources_855453_15.stderr parsl.dataflow.dflow INFO: Task 1462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7b09ca49ac8e11ba9addda4bb61dd4 with result from task 1462 parsl.dataflow.dflow DEBUG: Task 1462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/71392155-85fa-494d-8cbf-4494203ddb20_isrForCrosstalkSources_855443_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/71392155-85fa-494d-8cbf-4494203ddb20_isrForCrosstalkSources_855443_61.stderr parsl.dataflow.dflow INFO: Task 1125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5f99aa25ee607e4b84a9dff6a0d8e8 with result from task 1125 parsl.dataflow.dflow DEBUG: Task 1125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf661f1f-b3e6-4550-bc94-004aaa75719c_isrForCrosstalkSources_855401_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf661f1f-b3e6-4550-bc94-004aaa75719c_isrForCrosstalkSources_855401_50.stderr parsl.dataflow.dflow INFO: Task 1316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ab7bb3f7fc25d2156635ee65031ef2 with result from task 1316 parsl.dataflow.dflow DEBUG: Task 1316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/86c94024-e976-4d32-8d0a-0048321b3825_isrForCrosstalkSources_855381_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/86c94024-e976-4d32-8d0a-0048321b3825_isrForCrosstalkSources_855381_38.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 673f9acedec57660448dd6f2da2fdbe2 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b4d9fa0e-b7df-40bb-85e1-7a24aa680dc6_isrForCrosstalkSources_855428_54.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b4d9fa0e-b7df-40bb-85e1-7a24aa680dc6_isrForCrosstalkSources_855428_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:49 proc_lsst.multi:146 [INFO] found job 16964050 in provider astro proc_lsst.multi INFO: found job 16964050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb507540bbb0a93d57ba5c4ae140f54f with result from task 1309 parsl.dataflow.dflow DEBUG: Task 1309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/87003133-5e0d-49f4-be9e-f775dddd51e2_isrForCrosstalkSources_855395_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/87003133-5e0d-49f4-be9e-f775dddd51e2_isrForCrosstalkSources_855395_50.stderr parsl.dataflow.dflow INFO: Task 1299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46bba9df017a529be602e73e2eabaeb9 with result from task 1299 parsl.dataflow.dflow DEBUG: Task 1299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e678a845-4e12-4bd2-9704-7bc59f58429b_isrForCrosstalkSources_855451_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e678a845-4e12-4bd2-9704-7bc59f58429b_isrForCrosstalkSources_855451_43.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c5c4db5660d3adb3753811565053324 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a14c869a-676a-4ecb-b5f5-97ba36502296_isrForCrosstalkSources_855412_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a14c869a-676a-4ecb-b5f5-97ba36502296_isrForCrosstalkSources_855412_55.stderr parsl.dataflow.dflow INFO: Task 1441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db5355f8ad97b40d11e0d970c6965260 with result from task 1441 parsl.dataflow.dflow DEBUG: Task 1441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/140f7a41-1471-44fd-9014-37e161c8fd4b_isrForCrosstalkSources_855458_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/140f7a41-1471-44fd-9014-37e161c8fd4b_isrForCrosstalkSources_855458_18.stderr parsl.dataflow.dflow INFO: Task 1261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd0549cf0aac57d572f3a74e2b455e68 with result from task 1261 parsl.dataflow.dflow DEBUG: Task 1261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/abe0bc68-b1da-46a5-a17a-558e9c27eb13_isrForCrosstalkSources_855388_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/abe0bc68-b1da-46a5-a17a-558e9c27eb13_isrForCrosstalkSources_855388_61.stderr parsl.dataflow.dflow INFO: Task 1156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dec89b0b2929b7edb8bce791ea26b20d with result from task 1156 parsl.dataflow.dflow DEBUG: Task 1156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a7ff794-7428-4c73-9b54-90700e07396f_isrForCrosstalkSources_855449_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/4a7ff794-7428-4c73-9b54-90700e07396f_isrForCrosstalkSources_855449_54.stderr parsl.dataflow.dflow INFO: Task 1303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d67d20a818874eb265bf97950c389c04 with result from task 1303 parsl.dataflow.dflow DEBUG: Task 1303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/557bcb82-074d-4c98-8cba-b66d1165bcae_isrForCrosstalkSources_855425_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/557bcb82-074d-4c98-8cba-b66d1165bcae_isrForCrosstalkSources_855425_47.stderr parsl.dataflow.dflow INFO: Task 1330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5f12f7f3f7456f37c077eadb382b1e8 with result from task 1330 parsl.dataflow.dflow DEBUG: Task 1330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2a7a15f1-d2f9-44c7-bd2a-611dceccfc0e_isrForCrosstalkSources_855403_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2a7a15f1-d2f9-44c7-bd2a-611dceccfc0e_isrForCrosstalkSources_855403_58.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dada14a5a51a1dba1ac9747d36fa0b50 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b521c2cd-a5a3-4d57-bd29-057cfce2af62_isrForCrosstalkSources_855394_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b521c2cd-a5a3-4d57-bd29-057cfce2af62_isrForCrosstalkSources_855394_62.stderr parsl.dataflow.dflow INFO: Task 1297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f2b45d455e0a402631a22d5a3c603c8 with result from task 1297 parsl.dataflow.dflow DEBUG: Task 1297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/20146c4b-1c2a-4229-a9d4-9f3c121e9003_isrForCrosstalkSources_855453_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/20146c4b-1c2a-4229-a9d4-9f3c121e9003_isrForCrosstalkSources_855453_28.stderr parsl.dataflow.dflow INFO: Task 1376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2b2b886b7cf694c21430adb8ae0693 with result from task 1376 parsl.dataflow.dflow DEBUG: Task 1376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f40266f1-d0f1-430c-90d6-058063de0ad9_isrForCrosstalkSources_855380_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f40266f1-d0f1-430c-90d6-058063de0ad9_isrForCrosstalkSources_855380_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:50 proc_lsst.multi:146 [INFO] found job 16964051 in provider astro proc_lsst.multi INFO: found job 16964051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 541c68c9473e81e1516c41b30c25c74c with result from task 1197 parsl.dataflow.dflow DEBUG: Task 1197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a61ff73a-d552-4e2e-9b88-69a4b3ea2c73_isrForCrosstalkSources_855391_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a61ff73a-d552-4e2e-9b88-69a4b3ea2c73_isrForCrosstalkSources_855391_50.stderr parsl.dataflow.dflow INFO: Task 1200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd00c7b5e21416e52a951e86c921bccb with result from task 1200 parsl.dataflow.dflow DEBUG: Task 1200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72b5e654-d85e-4acd-862a-a9fcdf0bc7a8_isrForCrosstalkSources_855437_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72b5e654-d85e-4acd-862a-a9fcdf0bc7a8_isrForCrosstalkSources_855437_53.stderr parsl.dataflow.dflow INFO: Task 1295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91315702fa3e2d29cfa82c52243fc01 with result from task 1295 parsl.dataflow.dflow DEBUG: Task 1295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fc8b863c-f4b3-4a19-91a8-27d1085bd03f_isrForCrosstalkSources_855425_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fc8b863c-f4b3-4a19-91a8-27d1085bd03f_isrForCrosstalkSources_855425_58.stderr parsl.dataflow.dflow INFO: Task 1254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 467b91f74e91e974ac99d37ef49c1407 with result from task 1254 parsl.dataflow.dflow DEBUG: Task 1254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/91e30c68-cba3-4211-9c30-863901ef8f65_isrForCrosstalkSources_855423_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/91e30c68-cba3-4211-9c30-863901ef8f65_isrForCrosstalkSources_855423_20.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f4e7ff8985d29e020fe2de1187003b8 with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d39055aa-a547-44ab-a735-302ca7f86e24_isrForCrosstalkSources_855419_55.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/d39055aa-a547-44ab-a735-302ca7f86e24_isrForCrosstalkSources_855419_55.stderr parsl.dataflow.dflow INFO: Task 1315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce203baccf08f176e62989baf4bcf53d with result from task 1315 parsl.dataflow.dflow DEBUG: Task 1315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da3ecc14-1e9a-4b80-b5ae-d243d38fc598_isrForCrosstalkSources_855404_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da3ecc14-1e9a-4b80-b5ae-d243d38fc598_isrForCrosstalkSources_855404_62.stderr parsl.dataflow.dflow INFO: Task 1207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c16d216fce3dc2a39e6842f58ccc0078 with result from task 1207 parsl.dataflow.dflow DEBUG: Task 1207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/192735ff-a08b-43e1-b9ee-a1150b687f47_isrForCrosstalkSources_855451_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/192735ff-a08b-43e1-b9ee-a1150b687f47_isrForCrosstalkSources_855451_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:50 proc_lsst.multi:146 [INFO] found job 16964052 in provider astro proc_lsst.multi INFO: found job 16964052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 900ca1f7e2b935c87534a796d8c96497 with result from task 1331 parsl.dataflow.dflow DEBUG: Task 1331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/c1c93ab4-95f3-4cec-b833-013f84663acc_isrForCrosstalkSources_855407_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/c1c93ab4-95f3-4cec-b833-013f84663acc_isrForCrosstalkSources_855407_61.stderr parsl.dataflow.dflow INFO: Task 1495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 863bab34a716bec640a6a93f0ff01c6a with result from task 1495 parsl.dataflow.dflow DEBUG: Task 1495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/81dd28fd-865e-4b7a-ab28-55e5cd6c4a81_isrForCrosstalkSources_855406_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/81dd28fd-865e-4b7a-ab28-55e5cd6c4a81_isrForCrosstalkSources_855406_8.stderr parsl.dataflow.dflow INFO: Task 1243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270f80c5e5e820816bf9288ee8d1264a with result from task 1243 parsl.dataflow.dflow DEBUG: Task 1243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/134b8198-6e1c-458b-9c45-d816ac1b7346_isrForCrosstalkSources_855441_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/134b8198-6e1c-458b-9c45-d816ac1b7346_isrForCrosstalkSources_855441_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:50 proc_lsst.multi:146 [INFO] found job 16964053 in provider astro proc_lsst.multi INFO: found job 16964053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca0526ed7cba5e54e33aec9c5d91830 with result from task 1244 parsl.dataflow.dflow DEBUG: Task 1244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4d64aee1-cc78-44c1-9c7d-3706f0857989_isrForCrosstalkSources_855400_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4d64aee1-cc78-44c1-9c7d-3706f0857989_isrForCrosstalkSources_855400_33.stderr parsl.dataflow.dflow INFO: Task 1208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f249e6fb2eb82a454f7c60e25d3a312 with result from task 1208 parsl.dataflow.dflow DEBUG: Task 1208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8613e3a1-c37b-4514-a38a-020de96ef597_isrForCrosstalkSources_855410_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8613e3a1-c37b-4514-a38a-020de96ef597_isrForCrosstalkSources_855410_44.stderr parsl.dataflow.dflow INFO: Task 1260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 760bd98144ec8327f2d830ce93ff6cfd with result from task 1260 parsl.dataflow.dflow DEBUG: Task 1260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5dba5c03-1145-4e30-810b-b5cf7630aec9_isrForCrosstalkSources_855389_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/5dba5c03-1145-4e30-810b-b5cf7630aec9_isrForCrosstalkSources_855389_33.stderr parsl.dataflow.dflow INFO: Task 1374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c74743f7b5bd776b9441d5dd497710db with result from task 1374 parsl.dataflow.dflow DEBUG: Task 1374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/4f5f74dc-2d7a-497f-95ca-72cf50425764_isrForCrosstalkSources_855437_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/4f5f74dc-2d7a-497f-95ca-72cf50425764_isrForCrosstalkSources_855437_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:51 proc_lsst.multi:146 [INFO] found job 16964054 in provider astro proc_lsst.multi INFO: found job 16964054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39203d0b8486e887f01d21a40072898a with result from task 1312 parsl.dataflow.dflow DEBUG: Task 1312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e02e7ba3-8aa4-4ae2-a99e-93dab5feca81_isrForCrosstalkSources_855407_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e02e7ba3-8aa4-4ae2-a99e-93dab5feca81_isrForCrosstalkSources_855407_26.stderr parsl.dataflow.dflow INFO: Task 1412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f12f37ff06b6fab9856cff04131af120 with result from task 1412 parsl.dataflow.dflow DEBUG: Task 1412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2356c3a5-39e7-40f0-b1a4-e7f6993e1cd4_isrForCrosstalkSources_855441_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2356c3a5-39e7-40f0-b1a4-e7f6993e1cd4_isrForCrosstalkSources_855441_35.stderr parsl.dataflow.dflow INFO: Task 1324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0475895b4f99d3b0d1ae28f2a99552f1 with result from task 1324 parsl.dataflow.dflow DEBUG: Task 1324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8be47b2b-715a-45f2-af8d-8f78dd25f25e_isrForCrosstalkSources_855399_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8be47b2b-715a-45f2-af8d-8f78dd25f25e_isrForCrosstalkSources_855399_17.stderr parsl.dataflow.dflow INFO: Task 1249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac8d07e425f5f7e3d0992727fb3ed67 with result from task 1249 parsl.dataflow.dflow DEBUG: Task 1249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b8778b36-6991-4f90-8f98-f25641afa329_isrForCrosstalkSources_855421_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b8778b36-6991-4f90-8f98-f25641afa329_isrForCrosstalkSources_855421_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:51 proc_lsst.multi:146 [INFO] found job 16964055 in provider astro proc_lsst.multi INFO: found job 16964055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 1391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e89b0c7a5e871891892ffc3c660066 with result from task 1391 parsl.dataflow.dflow DEBUG: Task 1391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/171081e6-b651-4257-a18b-f1fbe6481126_isrForCrosstalkSources_855452_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/171081e6-b651-4257-a18b-f1fbe6481126_isrForCrosstalkSources_855452_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:51 proc_lsst.multi:146 [INFO] found job 16964056 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964056 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 1310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab0f689152f923b711c39c0cc867aba8 with result from task 1310 parsl.dataflow.dflow DEBUG: Task 1310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1c19a13e-e764-4cab-bf82-0120b9bd3c53_isrForCrosstalkSources_855455_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1c19a13e-e764-4cab-bf82-0120b9bd3c53_isrForCrosstalkSources_855455_19.stderr parsl.dataflow.dflow INFO: Task 1334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccab38fa119d1eb31244571c4372b047 with result from task 1334 parsl.dataflow.dflow DEBUG: Task 1334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2e0cb3ff-e5a4-4e29-8a9e-3b1f7c9e3116_isrForCrosstalkSources_855447_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2e0cb3ff-e5a4-4e29-8a9e-3b1f7c9e3116_isrForCrosstalkSources_855447_45.stderr parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d4f41450bf137ac8b4e7bafc8bac08c with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ac104252-390f-489c-b9af-2fbd99e5d647_isrForCrosstalkSources_855416_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ac104252-390f-489c-b9af-2fbd99e5d647_isrForCrosstalkSources_855416_54.stderr parsl.dataflow.dflow INFO: Task 1366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f67b123c01eac931a6330877581021bd with result from task 1366 parsl.dataflow.dflow DEBUG: Task 1366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/024dac83-2a09-4146-910c-d122fbf29926_isrForCrosstalkSources_855435_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/024dac83-2a09-4146-910c-d122fbf29926_isrForCrosstalkSources_855435_40.stderr parsl.dataflow.dflow INFO: Task 1428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3006386aae5da76f3c83243ec7812e33 with result from task 1428 parsl.dataflow.dflow DEBUG: Task 1428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c115c1eb-43c4-483e-9e9a-5fc2139de29d_isrForCrosstalkSources_855451_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c115c1eb-43c4-483e-9e9a-5fc2139de29d_isrForCrosstalkSources_855451_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:51 proc_lsst.multi:146 [INFO] found job 16964057 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964057 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 1146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81000f2986489ada7a9d3028a30aca80 with result from task 1146 parsl.dataflow.dflow DEBUG: Task 1146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d29abc-4cc2-4f82-96f8-c02fff4da0a5_isrForCrosstalkSources_855448_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d29abc-4cc2-4f82-96f8-c02fff4da0a5_isrForCrosstalkSources_855448_55.stderr parsl.dataflow.dflow INFO: Task 1301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e54c16bce6dc06d06a931f479c440f19 with result from task 1301 parsl.dataflow.dflow DEBUG: Task 1301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08bb7590-ada6-4aac-bb8a-f28688484c0c_isrForCrosstalkSources_855437_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08bb7590-ada6-4aac-bb8a-f28688484c0c_isrForCrosstalkSources_855437_1.stderr parsl.dataflow.dflow INFO: Task 1272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 224bb6feccc05cea1699a5dd5d969c34 with result from task 1272 parsl.dataflow.dflow DEBUG: Task 1272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c8bd9403-a3b1-4f7e-9e0b-346dd96f248e_isrForCrosstalkSources_855391_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c8bd9403-a3b1-4f7e-9e0b-346dd96f248e_isrForCrosstalkSources_855391_17.stderr parsl.dataflow.dflow INFO: Task 1375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74144739686562911e675a4b15234cd4 with result from task 1375 parsl.dataflow.dflow DEBUG: Task 1375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46f900e5-e42c-44cb-870d-2c5bcf107d08_isrForCrosstalkSources_855448_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/46f900e5-e42c-44cb-870d-2c5bcf107d08_isrForCrosstalkSources_855448_19.stderr parsl.dataflow.dflow INFO: Task 1204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 489e670cac84a44cb72e45e34ef99c19 with result from task 1204 parsl.dataflow.dflow DEBUG: Task 1204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/63e030b0-ecce-4ada-a602-91eee322c74d_isrForCrosstalkSources_855414_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/63e030b0-ecce-4ada-a602-91eee322c74d_isrForCrosstalkSources_855414_16.stderr parsl.dataflow.dflow INFO: Task 1342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a40d651e09ad2deac00410a7822691e with result from task 1342 parsl.dataflow.dflow DEBUG: Task 1342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c7b1fb3f-8b0f-471c-bb43-31c2da8ee99a_isrForCrosstalkSources_855448_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c7b1fb3f-8b0f-471c-bb43-31c2da8ee99a_isrForCrosstalkSources_855448_21.stderr parsl.dataflow.dflow INFO: Task 1304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 589c9b4ae4ad357dbbf8bf88a41d19ba with result from task 1304 parsl.dataflow.dflow DEBUG: Task 1304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/62c95f0e-802c-4ee0-a27a-858301a32a7d_isrForCrosstalkSources_855430_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/62c95f0e-802c-4ee0-a27a-858301a32a7d_isrForCrosstalkSources_855430_41.stderr parsl.dataflow.dflow INFO: Task 1263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5cc42f481d207ff92af7730af958eb7 with result from task 1263 parsl.dataflow.dflow DEBUG: Task 1263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/65b79be9-f467-4450-9eea-7e284bf2fde2_isrForCrosstalkSources_855448_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/65b79be9-f467-4450-9eea-7e284bf2fde2_isrForCrosstalkSources_855448_29.stderr parsl.dataflow.dflow INFO: Task 1442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa345f00d82f17e05eddbe9be95df09 with result from task 1442 parsl.dataflow.dflow DEBUG: Task 1442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/888130f5-e922-48a3-84fe-cc92d9266001_isrForCrosstalkSources_855385_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/888130f5-e922-48a3-84fe-cc92d9266001_isrForCrosstalkSources_855385_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:52 proc_lsst.multi:146 [INFO] found job 16964058 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964058 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 1314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81699f1ed8cb86b0f0c5dead88a758c5 with result from task 1314 parsl.dataflow.dflow DEBUG: Task 1314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5cfd9b92-3f22-4387-89de-cb30b0b2057e_isrForCrosstalkSources_855427_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5cfd9b92-3f22-4387-89de-cb30b0b2057e_isrForCrosstalkSources_855427_39.stderr parsl.dataflow.dflow INFO: Task 1349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 932f60fc74ace2ef895cb9cef8edc49e with result from task 1349 parsl.dataflow.dflow DEBUG: Task 1349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/4ed6a71c-ecc9-4314-b28d-32dd516f0a28_isrForCrosstalkSources_855407_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/4ed6a71c-ecc9-4314-b28d-32dd516f0a28_isrForCrosstalkSources_855407_10.stderr parsl.dataflow.dflow INFO: Task 1351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45e4b2bf3847b14b61b3954084532c6b with result from task 1351 parsl.dataflow.dflow DEBUG: Task 1351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bc0b0985-388d-416d-b211-184dc00d975d_isrForCrosstalkSources_855438_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/bc0b0985-388d-416d-b211-184dc00d975d_isrForCrosstalkSources_855438_34.stderr parsl.dataflow.dflow INFO: Task 1320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0083101376a9cbe547325a809ac55cd0 with result from task 1320 parsl.dataflow.dflow DEBUG: Task 1320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/a60e20fc-f449-4489-b8fa-990b53f07d74_isrForCrosstalkSources_855389_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/a60e20fc-f449-4489-b8fa-990b53f07d74_isrForCrosstalkSources_855389_14.stderr parsl.dataflow.dflow INFO: Task 1515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1e5cbdaeb6d4fbef8822b56e37a9d9c with result from task 1515 parsl.dataflow.dflow DEBUG: Task 1515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/99924625-b0c6-4890-8304-b145ffc16266_isrForCrosstalkSources_855432_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/99924625-b0c6-4890-8304-b145ffc16266_isrForCrosstalkSources_855432_20.stderr parsl.dataflow.dflow INFO: Task 1266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62f498704092d656a3535bc8b6850e97 with result from task 1266 parsl.dataflow.dflow DEBUG: Task 1266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3ef72b83-df55-4511-ab74-b9c9866af7f3_isrForCrosstalkSources_855427_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3ef72b83-df55-4511-ab74-b9c9866af7f3_isrForCrosstalkSources_855427_11.stderr parsl.dataflow.dflow INFO: Task 1383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4dadea399f90882acd8f243e80809f with result from task 1383 parsl.dataflow.dflow DEBUG: Task 1383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/34e690e0-9ac8-4371-9840-ffa047bf8809_isrForCrosstalkSources_855407_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/34e690e0-9ac8-4371-9840-ffa047bf8809_isrForCrosstalkSources_855407_40.stderr parsl.dataflow.dflow INFO: Task 1381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e185057692941c61e1aa0091530198a5 with result from task 1381 parsl.dataflow.dflow DEBUG: Task 1381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a8a51654-0177-4d7a-99b8-795615b237a2_isrForCrosstalkSources_855387_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a8a51654-0177-4d7a-99b8-795615b237a2_isrForCrosstalkSources_855387_35.stderr parsl.dataflow.dflow INFO: Task 1220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d6aba6bfbe97e21569913209bc0eaf with result from task 1220 parsl.dataflow.dflow DEBUG: Task 1220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/572795c9-8ba3-4066-845b-cdd714519b7f_isrForCrosstalkSources_855408_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/572795c9-8ba3-4066-845b-cdd714519b7f_isrForCrosstalkSources_855408_48.stderr parsl.dataflow.dflow INFO: Task 1347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085c119dc573374cea21541ef449f98c with result from task 1347 parsl.dataflow.dflow DEBUG: Task 1347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3fdf4877-f0b2-48d9-98d2-88de3b89dbf4_isrForCrosstalkSources_855385_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3fdf4877-f0b2-48d9-98d2-88de3b89dbf4_isrForCrosstalkSources_855385_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:52 proc_lsst.multi:146 [INFO] found job 16964059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d76973ad477dce9c036d6de5ae68dc28 with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/450f629e-7f7e-44fa-8e0f-0832e94473a8_isrForCrosstalkSources_855398_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/450f629e-7f7e-44fa-8e0f-0832e94473a8_isrForCrosstalkSources_855398_58.stderr parsl.dataflow.dflow INFO: Task 1302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 690b565e799c2213d5f69b221667b829 with result from task 1302 parsl.dataflow.dflow DEBUG: Task 1302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2225b674-ad77-4c5f-ad33-1337d06c3e71_isrForCrosstalkSources_855449_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2225b674-ad77-4c5f-ad33-1337d06c3e71_isrForCrosstalkSources_855449_18.stderr parsl.dataflow.dflow INFO: Task 1497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 146e5fdbb6e2631baae6d13d90a2b1d2 with result from task 1497 parsl.dataflow.dflow DEBUG: Task 1497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/11e4b955-ced1-4c30-9e81-2818f1b5c4f8_isrForCrosstalkSources_855393_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/11e4b955-ced1-4c30-9e81-2818f1b5c4f8_isrForCrosstalkSources_855393_16.stderr parsl.dataflow.dflow INFO: Task 1440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2f379f91017a2112cbfe2fd5c5412b7 with result from task 1440 parsl.dataflow.dflow DEBUG: Task 1440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8a608126-3bfa-46b0-a30a-180fa0366f83_isrForCrosstalkSources_855440_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8a608126-3bfa-46b0-a30a-180fa0366f83_isrForCrosstalkSources_855440_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:53 proc_lsst.multi:146 [INFO] found job 16964060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 1354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8bcca8fc5eb4690addef791b4d87792 with result from task 1354 parsl.dataflow.dflow DEBUG: Task 1354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a3dd926f-6197-46ee-86ad-03a39fb76b6d_isrForCrosstalkSources_855434_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a3dd926f-6197-46ee-86ad-03a39fb76b6d_isrForCrosstalkSources_855434_56.stderr parsl.dataflow.dflow INFO: Task 1498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa7e2005a07dcf2fe0adef23434393a with result from task 1498 parsl.dataflow.dflow DEBUG: Task 1498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7e7b343c-d761-4ef5-8a79-206949752ebf_isrForCrosstalkSources_855406_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7e7b343c-d761-4ef5-8a79-206949752ebf_isrForCrosstalkSources_855406_16.stderr parsl.dataflow.dflow INFO: Task 1322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b666a375291914178646b5fb1e55ca82 with result from task 1322 parsl.dataflow.dflow DEBUG: Task 1322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b4f326f9-37af-41a3-bd7c-9b8ead0294c7_isrForCrosstalkSources_855446_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b4f326f9-37af-41a3-bd7c-9b8ead0294c7_isrForCrosstalkSources_855446_56.stderr parsl.dataflow.dflow INFO: Task 1317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 223039c26ef68c93d58c5905fb981df3 with result from task 1317 parsl.dataflow.dflow DEBUG: Task 1317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/0b30c9b2-3269-487c-bc31-196ae62addf8_isrForCrosstalkSources_855421_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/0b30c9b2-3269-487c-bc31-196ae62addf8_isrForCrosstalkSources_855421_20.stderr parsl.dataflow.dflow INFO: Task 1533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce7c86eaf34facc13ee2645eb7571919 with result from task 1533 parsl.dataflow.dflow DEBUG: Task 1533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a1d5cef5-c5a2-46e9-8e2b-b1b1bda67021_isrForCrosstalkSources_855407_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a1d5cef5-c5a2-46e9-8e2b-b1b1bda67021_isrForCrosstalkSources_855407_19.stderr parsl.dataflow.dflow INFO: Task 1463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbbc4d1fac633bcecfba440f95655444 with result from task 1463 parsl.dataflow.dflow DEBUG: Task 1463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8ca21f6-9695-46d9-8a65-9d00e305eea3_isrForCrosstalkSources_855449_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/e8ca21f6-9695-46d9-8a65-9d00e305eea3_isrForCrosstalkSources_855449_21.stderr parsl.dataflow.dflow INFO: Task 1241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691d75fe48d11fe8304e55b7f1da3292 with result from task 1241 parsl.dataflow.dflow DEBUG: Task 1241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ee7204a3-5ccc-45ef-b4dd-52e1b1216fda_isrForCrosstalkSources_855414_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ee7204a3-5ccc-45ef-b4dd-52e1b1216fda_isrForCrosstalkSources_855414_11.stderr parsl.dataflow.dflow INFO: Task 1271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d7aaecaf4086014e83fcb14a40baa3f with result from task 1271 parsl.dataflow.dflow DEBUG: Task 1271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a350a02b-c7bf-4e4a-a892-43335800ae36_isrForCrosstalkSources_855452_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a350a02b-c7bf-4e4a-a892-43335800ae36_isrForCrosstalkSources_855452_35.stderr parsl.dataflow.dflow INFO: Task 1425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8595ff973276d4a2e0ec848f35c35295 with result from task 1425 parsl.dataflow.dflow DEBUG: Task 1425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7f309441-ad53-4424-a701-48f407a9036e_isrForCrosstalkSources_855437_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7f309441-ad53-4424-a701-48f407a9036e_isrForCrosstalkSources_855437_31.stderr parsl.dataflow.dflow INFO: Task 1196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8c1910b3d58d72e51180b2caff4b40d with result from task 1196 parsl.dataflow.dflow DEBUG: Task 1196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/aa4ac66e-9bae-494e-a39b-594004925371_isrForCrosstalkSources_855405_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/aa4ac66e-9bae-494e-a39b-594004925371_isrForCrosstalkSources_855405_61.stderr parsl.dataflow.dflow INFO: Task 1411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3d16f3547f809adc28713df7aceeb54 with result from task 1411 parsl.dataflow.dflow DEBUG: Task 1411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/93ab04a7-da49-442c-aa83-4a20284602d7_isrForCrosstalkSources_855425_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/93ab04a7-da49-442c-aa83-4a20284602d7_isrForCrosstalkSources_855425_40.stderr parsl.dataflow.dflow INFO: Task 1262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7653fd5bf0bb77212d968b8fdaaebbea with result from task 1262 parsl.dataflow.dflow DEBUG: Task 1262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4719d837-3dbb-4df0-be0f-a99fe1a9b0d9_isrForCrosstalkSources_855443_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4719d837-3dbb-4df0-be0f-a99fe1a9b0d9_isrForCrosstalkSources_855443_53.stderr parsl.dataflow.dflow INFO: Task 1278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bccde0956314ca1c2a7e8faa9d1ffc1 with result from task 1278 parsl.dataflow.dflow DEBUG: Task 1278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d6cc6044-2a7e-4a01-b02e-ddaa17aed0a0_isrForCrosstalkSources_855384_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d6cc6044-2a7e-4a01-b02e-ddaa17aed0a0_isrForCrosstalkSources_855384_59.stderr parsl.dataflow.dflow INFO: Task 1392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db179ffa4fcaf1a759361a2282de67a7 with result from task 1392 parsl.dataflow.dflow DEBUG: Task 1392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c2f3024b-59e8-46bb-ac91-0d140ca1ede1_isrForCrosstalkSources_855395_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c2f3024b-59e8-46bb-ac91-0d140ca1ede1_isrForCrosstalkSources_855395_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:54 proc_lsst.multi:146 [INFO] found job 16964061 in provider escience_a40 proc_lsst.multi INFO: found job 16964061 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03a5249d1b06ad6073d076a3de62081f with result from task 1343 parsl.dataflow.dflow DEBUG: Task 1343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/32327c0e-04f0-4d3e-a0ab-db0ae59538b2_isrForCrosstalkSources_855439_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/32327c0e-04f0-4d3e-a0ab-db0ae59538b2_isrForCrosstalkSources_855439_20.stderr parsl.dataflow.dflow INFO: Task 1474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cee0b70a1b8f72d02032892112585d2a with result from task 1474 parsl.dataflow.dflow DEBUG: Task 1474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/62d54c10-6479-42d8-817c-a197d0618efb_isrForCrosstalkSources_855440_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/62d54c10-6479-42d8-817c-a197d0618efb_isrForCrosstalkSources_855440_31.stderr parsl.dataflow.dflow INFO: Task 1511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58a75ebc20db701dbd28e0fa597b213b with result from task 1511 parsl.dataflow.dflow DEBUG: Task 1511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c504f64-62fc-4220-a553-99cc6ab695cb_isrForCrosstalkSources_855400_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2c504f64-62fc-4220-a553-99cc6ab695cb_isrForCrosstalkSources_855400_53.stderr parsl.dataflow.dflow INFO: Task 1332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9299b5d22bbf5c7d604915b8d7e939e with result from task 1332 parsl.dataflow.dflow DEBUG: Task 1332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ac6e2c7a-55bf-4c62-aaa6-a9c329fbbf25_isrForCrosstalkSources_855454_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ac6e2c7a-55bf-4c62-aaa6-a9c329fbbf25_isrForCrosstalkSources_855454_51.stderr parsl.dataflow.dflow INFO: Task 1471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf528a8b8ccfcece26351c6b5e86a1b9 with result from task 1471 parsl.dataflow.dflow DEBUG: Task 1471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c806cd76-eaf4-449b-bb6d-5db845ba00b7_isrForCrosstalkSources_855452_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c806cd76-eaf4-449b-bb6d-5db845ba00b7_isrForCrosstalkSources_855452_21.stderr parsl.dataflow.dflow INFO: Task 1380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa69480c85f4d6a2f58595d16e3217f with result from task 1380 parsl.dataflow.dflow DEBUG: Task 1380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/32c8a22b-2d08-4da6-8bf2-d2e7a5b13653_isrForCrosstalkSources_855436_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/32c8a22b-2d08-4da6-8bf2-d2e7a5b13653_isrForCrosstalkSources_855436_18.stderr parsl.dataflow.dflow INFO: Task 1475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420d6632698a98f852b021fbfd6cf46f with result from task 1475 parsl.dataflow.dflow DEBUG: Task 1475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4b5989dc-79cb-41b6-9eac-f92a95dc9826_isrForCrosstalkSources_855390_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/4b5989dc-79cb-41b6-9eac-f92a95dc9826_isrForCrosstalkSources_855390_3.stderr parsl.dataflow.dflow INFO: Task 1333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56a9b79e6829280839a6a0cc19eacb73 with result from task 1333 parsl.dataflow.dflow DEBUG: Task 1333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/a7f6d45f-f725-43c0-ab9a-f35d26368438_isrForCrosstalkSources_855446_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/a7f6d45f-f725-43c0-ab9a-f35d26368438_isrForCrosstalkSources_855446_16.stderr parsl.dataflow.dflow INFO: Task 1490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c42c503475344d01286317a43794d1ce with result from task 1490 parsl.dataflow.dflow DEBUG: Task 1490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9c03fd4e-d4f5-40ef-9ed8-6a4f81bb9eac_isrForCrosstalkSources_855387_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9c03fd4e-d4f5-40ef-9ed8-6a4f81bb9eac_isrForCrosstalkSources_855387_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:54 proc_lsst.multi:146 [INFO] found job 16964062 in provider escience_a40 proc_lsst.multi INFO: found job 16964062 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d115ad7fd2cb36cae56d3cf5d36146a3 with result from task 1426 parsl.dataflow.dflow DEBUG: Task 1426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3d4d3f2d-12d2-4e7e-837d-cde10da9b0b6_isrForCrosstalkSources_855381_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3d4d3f2d-12d2-4e7e-837d-cde10da9b0b6_isrForCrosstalkSources_855381_57.stderr parsl.dataflow.dflow INFO: Task 1313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70e920b5a72763613ea5d57a983273a with result from task 1313 parsl.dataflow.dflow DEBUG: Task 1313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c7574d87-d8ed-4555-9b6a-79a4a782f6f3_isrForCrosstalkSources_855392_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c7574d87-d8ed-4555-9b6a-79a4a782f6f3_isrForCrosstalkSources_855392_7.stderr parsl.dataflow.dflow INFO: Task 1264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 034d355e74ed77f113593ee881f29e29 with result from task 1264 parsl.dataflow.dflow DEBUG: Task 1264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3907cb54-d165-4e79-b5ba-d61788582ca8_isrForCrosstalkSources_855411_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3907cb54-d165-4e79-b5ba-d61788582ca8_isrForCrosstalkSources_855411_9.stderr parsl.dataflow.dflow INFO: Task 1369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de93359db71808c262083334cb2603d6 with result from task 1369 parsl.dataflow.dflow DEBUG: Task 1369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/ce1bc7e1-801a-46b2-b9db-33f603c56d9a_isrForCrosstalkSources_855458_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/ce1bc7e1-801a-46b2-b9db-33f603c56d9a_isrForCrosstalkSources_855458_46.stderr parsl.dataflow.dflow INFO: Task 1306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de4c4d5cf0e559648f68155ff7db3930 with result from task 1306 parsl.dataflow.dflow DEBUG: Task 1306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e479013a-0584-48da-a843-208637635a8e_isrForCrosstalkSources_855394_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e479013a-0584-48da-a843-208637635a8e_isrForCrosstalkSources_855394_42.stderr parsl.dataflow.dflow INFO: Task 1267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10a6122df700ff1af1357b1cd28c5d55 with result from task 1267 parsl.dataflow.dflow DEBUG: Task 1267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/50e6e099-4753-4014-90b1-5588a4b81fde_isrForCrosstalkSources_855410_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/50e6e099-4753-4014-90b1-5588a4b81fde_isrForCrosstalkSources_855410_53.stderr parsl.dataflow.dflow INFO: Task 1311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5990672721d51aa64e71c8fc8a5575ac with result from task 1311 parsl.dataflow.dflow DEBUG: Task 1311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b976d34a-cd82-420c-8404-1f68db4f912a_isrForCrosstalkSources_855425_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b976d34a-cd82-420c-8404-1f68db4f912a_isrForCrosstalkSources_855425_5.stderr parsl.dataflow.dflow INFO: Task 1187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 567449b1339431d577b18a2bcfddd46f with result from task 1187 parsl.dataflow.dflow DEBUG: Task 1187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b65b6b3e-aa32-4945-a847-9a5627525381_isrForCrosstalkSources_855422_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b65b6b3e-aa32-4945-a847-9a5627525381_isrForCrosstalkSources_855422_57.stderr parsl.dataflow.dflow INFO: Task 1359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ed88b6ee918f8692e5740a61ad1ea68 with result from task 1359 parsl.dataflow.dflow DEBUG: Task 1359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9110c907-f64f-4025-bf22-cddef1f9a1f1_isrForCrosstalkSources_855418_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9110c907-f64f-4025-bf22-cddef1f9a1f1_isrForCrosstalkSources_855418_25.stderr parsl.dataflow.dflow INFO: Task 1372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f140adb3cb847e3788325153ebfd946 with result from task 1372 parsl.dataflow.dflow DEBUG: Task 1372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ae3ac075-a6d6-4f80-8bc6-4a7e9e1d273d_isrForCrosstalkSources_855399_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ae3ac075-a6d6-4f80-8bc6-4a7e9e1d273d_isrForCrosstalkSources_855399_26.stderr parsl.dataflow.dflow INFO: Task 1422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b3bf9247852ce20c492abf7e6d4a5b with result from task 1422 parsl.dataflow.dflow DEBUG: Task 1422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f645d75c-7098-4478-8869-f1d7eb3d00a7_isrForCrosstalkSources_855436_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/f645d75c-7098-4478-8869-f1d7eb3d00a7_isrForCrosstalkSources_855436_8.stderr parsl.dataflow.dflow INFO: Task 1452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81e52c04891f783049686c41d800e06d with result from task 1452 parsl.dataflow.dflow DEBUG: Task 1452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/589f6c3b-0f80-47aa-ab7f-7662f52567fb_isrForCrosstalkSources_855436_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/589f6c3b-0f80-47aa-ab7f-7662f52567fb_isrForCrosstalkSources_855436_40.stderr parsl.dataflow.dflow INFO: Task 1325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cf0d92ed826bb4d6ad13707e7f940f5 with result from task 1325 parsl.dataflow.dflow DEBUG: Task 1325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/66ac8fa6-220a-41ed-a47b-df9b0797929e_isrForCrosstalkSources_855414_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/66ac8fa6-220a-41ed-a47b-df9b0797929e_isrForCrosstalkSources_855414_21.stderr parsl.dataflow.dflow INFO: Task 1340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b3bf0f3db2339b2702b6f49b2dff1d with result from task 1340 parsl.dataflow.dflow DEBUG: Task 1340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2cba4a96-0593-45ee-8440-010b4ac57327_isrForCrosstalkSources_855400_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2cba4a96-0593-45ee-8440-010b4ac57327_isrForCrosstalkSources_855400_41.stderr parsl.dataflow.dflow INFO: Task 1420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 006904b1ff6c63589c2538e74360bab0 with result from task 1420 parsl.dataflow.dflow DEBUG: Task 1420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5bf1cd5d-a356-4f76-83fa-cf8590ac28d3_isrForCrosstalkSources_855456_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/5bf1cd5d-a356-4f76-83fa-cf8590ac28d3_isrForCrosstalkSources_855456_8.stderr parsl.dataflow.dflow INFO: Task 1473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95420ca6ad3b2191f3e20ff6bb89c256 with result from task 1473 parsl.dataflow.dflow DEBUG: Task 1473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/633735e3-81cb-48bd-ad65-212978855fcd_isrForCrosstalkSources_855442_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/633735e3-81cb-48bd-ad65-212978855fcd_isrForCrosstalkSources_855442_34.stderr parsl.dataflow.dflow INFO: Task 1403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2729de422831ff3a4709eebb178a4cb2 with result from task 1403 parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:55 proc_lsst.multi:146 [INFO] found job 16964063 in provider escience_a40 proc_lsst.multi INFO: found job 16964063 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow DEBUG: Task 1403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/00c0b93c-abeb-491a-b34c-034dc96bdb4d_isrForCrosstalkSources_855389_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/00c0b93c-abeb-491a-b34c-034dc96bdb4d_isrForCrosstalkSources_855389_56.stderr parsl.dataflow.dflow INFO: Task 1328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf16430c25dda243d1b306b7754705d with result from task 1328 parsl.dataflow.dflow DEBUG: Task 1328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/5ee26fa8-0da0-4f33-b24d-ddbdc04ea898_isrForCrosstalkSources_855392_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/5ee26fa8-0da0-4f33-b24d-ddbdc04ea898_isrForCrosstalkSources_855392_3.stderr parsl.dataflow.dflow INFO: Task 1482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0028954ea8d9035b0144b4dfd927b5 with result from task 1482 parsl.dataflow.dflow DEBUG: Task 1482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/099901cf-0314-4994-8d12-9ddd77245969_isrForCrosstalkSources_855392_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/099901cf-0314-4994-8d12-9ddd77245969_isrForCrosstalkSources_855392_25.stderr parsl.dataflow.dflow INFO: Task 1488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37a7898626f24c9477755bed599d1e6a with result from task 1488 parsl.dataflow.dflow DEBUG: Task 1488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e33deb42-1661-4653-a84d-1f458ce7f374_isrForCrosstalkSources_855402_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e33deb42-1661-4653-a84d-1f458ce7f374_isrForCrosstalkSources_855402_32.stderr parsl.dataflow.dflow INFO: Task 1404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34cde2d1bb3cf26075fd64c34ef0a146 with result from task 1404 parsl.dataflow.dflow DEBUG: Task 1404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/acc540fb-726f-4fc8-b52e-33a2edf5a5b8_isrForCrosstalkSources_855401_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/acc540fb-726f-4fc8-b52e-33a2edf5a5b8_isrForCrosstalkSources_855401_40.stderr parsl.dataflow.dflow INFO: Task 1344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cce492c83618c428d64b4fbe819d59c with result from task 1344 parsl.dataflow.dflow DEBUG: Task 1344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bb88b41f-fb98-4025-b945-e221ba11a15a_isrForCrosstalkSources_855420_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/bb88b41f-fb98-4025-b945-e221ba11a15a_isrForCrosstalkSources_855420_45.stderr parsl.dataflow.dflow INFO: Task 1393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86622a357660518c9ad24e78716dc87a with result from task 1393 parsl.dataflow.dflow DEBUG: Task 1393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a73b5a88-4d39-4e82-a486-eb5e4dc82fb5_isrForCrosstalkSources_855384_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a73b5a88-4d39-4e82-a486-eb5e4dc82fb5_isrForCrosstalkSources_855384_23.stderr parsl.dataflow.dflow INFO: Task 1355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 164b5c251e509f27699779b204538374 with result from task 1355 parsl.dataflow.dflow DEBUG: Task 1355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/41c385a2-5807-417c-972f-0d528d121718_isrForCrosstalkSources_855444_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/41c385a2-5807-417c-972f-0d528d121718_isrForCrosstalkSources_855444_10.stderr parsl.dataflow.dflow INFO: Task 1371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3df768a111db13a13177de1ca9706f67 with result from task 1371 parsl.dataflow.dflow DEBUG: Task 1371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b2330561-5f1a-4f00-ba40-effc69392b36_isrForCrosstalkSources_855408_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b2330561-5f1a-4f00-ba40-effc69392b36_isrForCrosstalkSources_855408_9.stderr parsl.dataflow.dflow INFO: Task 1402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c6ae5c2ff938e10b398bb802332c9c4 with result from task 1402 parsl.dataflow.dflow DEBUG: Task 1402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5d59860f-5ab8-48d6-ad44-1ec6dab65d51_isrForCrosstalkSources_855396_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5d59860f-5ab8-48d6-ad44-1ec6dab65d51_isrForCrosstalkSources_855396_27.stderr parsl.dataflow.dflow INFO: Task 1352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4388850d38be09dfb9a8911682d97438 with result from task 1352 parsl.dataflow.dflow DEBUG: Task 1352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/893f394b-8d19-4c2f-988a-982506614165_isrForCrosstalkSources_855440_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/893f394b-8d19-4c2f-988a-982506614165_isrForCrosstalkSources_855440_2.stderr parsl.dataflow.dflow INFO: Task 1318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b445f04bb24eecc65209224bdbcad7b with result from task 1318 parsl.dataflow.dflow DEBUG: Task 1318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/098b044f-9cff-48a1-ab6e-3f64e5a8dcd6_isrForCrosstalkSources_855437_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/098b044f-9cff-48a1-ab6e-3f64e5a8dcd6_isrForCrosstalkSources_855437_2.stderr parsl.dataflow.dflow INFO: Task 1368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45b0797b394ed68d0b6bd6c24de92f8a with result from task 1368 parsl.dataflow.dflow DEBUG: Task 1368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f50d54e4-a262-4d58-b46c-397244d0b85c_isrForCrosstalkSources_855449_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f50d54e4-a262-4d58-b46c-397244d0b85c_isrForCrosstalkSources_855449_6.stderr parsl.dataflow.dflow INFO: Task 1378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4c4925dcd06fbbe416aace81ce485d4 with result from task 1378 parsl.dataflow.dflow DEBUG: Task 1378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4c66839d-9c8f-481a-bfaf-f118dd7d629c_isrForCrosstalkSources_855391_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/4c66839d-9c8f-481a-bfaf-f118dd7d629c_isrForCrosstalkSources_855391_39.stderr parsl.dataflow.dflow INFO: Task 1221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry add435c544ce7a8cde3639169952955e with result from task 1221 parsl.dataflow.dflow DEBUG: Task 1221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c7a425de-d49b-42ef-a3a4-3b242dab0721_isrForCrosstalkSources_855444_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c7a425de-d49b-42ef-a3a4-3b242dab0721_isrForCrosstalkSources_855444_1.stderr parsl.dataflow.dflow INFO: Task 1248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c31ff5afd7e2b0dd85a312a4b168c633 with result from task 1248 parsl.dataflow.dflow DEBUG: Task 1248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/11fefe37-4beb-4319-8aa5-5982944206fb_isrForCrosstalkSources_855390_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/11fefe37-4beb-4319-8aa5-5982944206fb_isrForCrosstalkSources_855390_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:55 proc_lsst.multi:146 [INFO] found job 16964064 in provider escience_a40 proc_lsst.multi INFO: found job 16964064 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2280b578ac482229cc1544421e4c1a9f with result from task 1451 parsl.dataflow.dflow DEBUG: Task 1451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed45015e-8c66-45be-9117-f5197f167735_isrForCrosstalkSources_855419_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed45015e-8c66-45be-9117-f5197f167735_isrForCrosstalkSources_855419_17.stderr parsl.dataflow.dflow INFO: Task 1399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34e7cb32b0b4e24277731be74ad2c714 with result from task 1399 parsl.dataflow.dflow DEBUG: Task 1399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/759741aa-3fb6-4fe8-8efb-2b18188424b1_isrForCrosstalkSources_855380_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/759741aa-3fb6-4fe8-8efb-2b18188424b1_isrForCrosstalkSources_855380_34.stderr parsl.dataflow.dflow INFO: Task 1336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb46aa299c867909ee25e2808dc5e033 with result from task 1336 parsl.dataflow.dflow DEBUG: Task 1336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f7da6ec1-b172-45fc-ad93-17f0e034f252_isrForCrosstalkSources_855419_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f7da6ec1-b172-45fc-ad93-17f0e034f252_isrForCrosstalkSources_855419_5.stderr parsl.dataflow.dflow INFO: Task 1424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 051d0ccae0497d50b76ec03ba03f8a41 with result from task 1424 parsl.dataflow.dflow DEBUG: Task 1424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cc40cc80-3120-49e3-a589-66017eb1d55e_isrForCrosstalkSources_855458_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/cc40cc80-3120-49e3-a589-66017eb1d55e_isrForCrosstalkSources_855458_27.stderr parsl.dataflow.dflow INFO: Task 1337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8afb320335ed1b3ef6c94fec0799968e with result from task 1337 parsl.dataflow.dflow DEBUG: Task 1337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/62f0b021-4263-4e10-b35b-1e80741ab1e5_isrForCrosstalkSources_855445_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/62f0b021-4263-4e10-b35b-1e80741ab1e5_isrForCrosstalkSources_855445_13.stderr parsl.dataflow.dflow INFO: Task 1496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c13244230ee0f2acfa12de7c703357 with result from task 1496 parsl.dataflow.dflow DEBUG: Task 1496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0132acac-04cd-4411-8ae0-2a2daceb05f1_isrForCrosstalkSources_855444_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0132acac-04cd-4411-8ae0-2a2daceb05f1_isrForCrosstalkSources_855444_39.stderr parsl.dataflow.dflow INFO: Task 1373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8b8e74aab035069b37fb354af8afc0 with result from task 1373 parsl.dataflow.dflow DEBUG: Task 1373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fdbc26c7-6d6f-4cfb-a236-1f1fa25e3b52_isrForCrosstalkSources_855394_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fdbc26c7-6d6f-4cfb-a236-1f1fa25e3b52_isrForCrosstalkSources_855394_35.stderr parsl.dataflow.dflow INFO: Task 1307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19a9bea44c0fff2e7beade2788679385 with result from task 1307 parsl.dataflow.dflow DEBUG: Task 1307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6b242225-49dc-43d2-80b3-acb96d5602ec_isrForCrosstalkSources_855401_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6b242225-49dc-43d2-80b3-acb96d5602ec_isrForCrosstalkSources_855401_16.stderr parsl.dataflow.dflow INFO: Task 1480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c69ceb02bf70075bf58ec616413b6a with result from task 1480 parsl.dataflow.dflow DEBUG: Task 1480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/87b67279-14a6-429d-a0b0-2c9998e690f5_isrForCrosstalkSources_855387_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/87b67279-14a6-429d-a0b0-2c9998e690f5_isrForCrosstalkSources_855387_27.stderr parsl.dataflow.dflow INFO: Task 1288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfb1277a01588ebfbfc81101a932ee75 with result from task 1288 parsl.dataflow.dflow DEBUG: Task 1288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/9f9c534a-6878-4915-b900-c846a2a85848_isrForCrosstalkSources_855435_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/9f9c534a-6878-4915-b900-c846a2a85848_isrForCrosstalkSources_855435_29.stderr parsl.dataflow.dflow INFO: Task 1389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e0ddbe042155223e85119583163059 with result from task 1389 parsl.dataflow.dflow DEBUG: Task 1389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/94bfd977-d507-41f9-8bf0-21c463ea13e7_isrForCrosstalkSources_855436_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/94bfd977-d507-41f9-8bf0-21c463ea13e7_isrForCrosstalkSources_855436_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:56 proc_lsst.multi:146 [INFO] found job 16964065 in provider escience_a40 proc_lsst.multi INFO: found job 16964065 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1cb210ced759c60af30f3e367ae44c5 with result from task 1247 parsl.dataflow.dflow DEBUG: Task 1247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c940a931-48da-4e0e-a6a6-d5c7d4e5af15_isrForCrosstalkSources_855391_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/c940a931-48da-4e0e-a6a6-d5c7d4e5af15_isrForCrosstalkSources_855391_30.stderr parsl.dataflow.dflow INFO: Task 1418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67321dfb902375129886d36e75172954 with result from task 1418 parsl.dataflow.dflow DEBUG: Task 1418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/efc63e9b-12b0-46cb-921a-f4925213edec_isrForCrosstalkSources_855389_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/efc63e9b-12b0-46cb-921a-f4925213edec_isrForCrosstalkSources_855389_10.stderr parsl.dataflow.dflow INFO: Task 1416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6221bd606b80e2ed815027fa8bbb59f with result from task 1416 parsl.dataflow.dflow DEBUG: Task 1416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dc40f4f8-50a5-4837-90e3-726ede50627b_isrForCrosstalkSources_855396_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dc40f4f8-50a5-4837-90e3-726ede50627b_isrForCrosstalkSources_855396_30.stderr parsl.dataflow.dflow INFO: Task 1434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c99f58a83fbed7719dc9be56f68c7c72 with result from task 1434 parsl.dataflow.dflow DEBUG: Task 1434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d21e8ebc-3c9d-4f19-8e1c-6be54468abee_isrForCrosstalkSources_855414_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d21e8ebc-3c9d-4f19-8e1c-6be54468abee_isrForCrosstalkSources_855414_39.stderr parsl.dataflow.dflow INFO: Task 1360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e0b32088d493abf0c52fb2d8576654 with result from task 1360 parsl.dataflow.dflow DEBUG: Task 1360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f8e392ee-535d-4fe5-aa4f-7a19f73523da_isrForCrosstalkSources_855438_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f8e392ee-535d-4fe5-aa4f-7a19f73523da_isrForCrosstalkSources_855438_60.stderr parsl.dataflow.dflow INFO: Task 1457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2ba32d6b8505720511def9e79330b74 with result from task 1457 parsl.dataflow.dflow DEBUG: Task 1457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/eff26f19-9cc3-4184-bf73-8503e5981914_isrForCrosstalkSources_855439_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/eff26f19-9cc3-4184-bf73-8503e5981914_isrForCrosstalkSources_855439_6.stderr parsl.dataflow.dflow INFO: Task 1459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d2ce938022daf6c847446038837af84 with result from task 1459 parsl.dataflow.dflow DEBUG: Task 1459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e3e4fe03-cdc9-478f-b625-1eab1095e8c8_isrForCrosstalkSources_855424_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e3e4fe03-cdc9-478f-b625-1eab1095e8c8_isrForCrosstalkSources_855424_27.stderr parsl.dataflow.dflow INFO: Task 1489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e930f09ab00a3a3f81f23ab85175db0 with result from task 1489 parsl.dataflow.dflow DEBUG: Task 1489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/41ce8129-76ce-4f45-9fc2-ed5d089bac12_isrForCrosstalkSources_855399_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/41ce8129-76ce-4f45-9fc2-ed5d089bac12_isrForCrosstalkSources_855399_25.stderr parsl.dataflow.dflow INFO: Task 1245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 329a55b0c46595ca1076fd3070b72dd4 with result from task 1245 parsl.dataflow.dflow DEBUG: Task 1245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cba499f4-7ba6-43ef-8065-dcfda4e70006_isrForCrosstalkSources_855435_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cba499f4-7ba6-43ef-8065-dcfda4e70006_isrForCrosstalkSources_855435_6.stderr parsl.dataflow.dflow INFO: Task 1216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 998ad9d1af4d908a328487506048e907 with result from task 1216 parsl.dataflow.dflow DEBUG: Task 1216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8cf3fa29-ea68-4ae7-aaed-16e7fc07c740_isrForCrosstalkSources_855391_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8cf3fa29-ea68-4ae7-aaed-16e7fc07c740_isrForCrosstalkSources_855391_52.stderr parsl.dataflow.dflow INFO: Task 1612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38b9545ad188e1ec1b9f7a7fca46416a with result from task 1612 parsl.dataflow.dflow DEBUG: Task 1612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9d50c72-a5d8-4057-8967-e067c11fa666_isrForCrosstalkSources_855458_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c9d50c72-a5d8-4057-8967-e067c11fa666_isrForCrosstalkSources_855458_25.stderr parsl.dataflow.dflow INFO: Task 1382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d70d010886f9554287d54d6f9efcd47 with result from task 1382 parsl.dataflow.dflow DEBUG: Task 1382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/e11bcf25-4766-44c1-b620-a62860267d5c_isrForCrosstalkSources_855440_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/e11bcf25-4766-44c1-b620-a62860267d5c_isrForCrosstalkSources_855440_34.stderr parsl.dataflow.dflow INFO: Task 1522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e11222a91a5e591e5846cca6615a988 with result from task 1522 parsl.dataflow.dflow DEBUG: Task 1522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dddd2166-a442-42ff-98d4-9ecc39568f28_isrForCrosstalkSources_855432_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/dddd2166-a442-42ff-98d4-9ecc39568f28_isrForCrosstalkSources_855432_21.stderr parsl.dataflow.dflow INFO: Task 1408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af7b1a8c0238c2bd4e359b1b9bd0ab2 with result from task 1408 parsl.dataflow.dflow DEBUG: Task 1408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/606f616d-6a86-4a34-90c7-68440eee528b_isrForCrosstalkSources_855442_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/606f616d-6a86-4a34-90c7-68440eee528b_isrForCrosstalkSources_855442_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:56 proc_lsst.multi:146 [INFO] found job 16964066 in provider escience_a40 proc_lsst.multi INFO: found job 16964066 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5c30de376fd155560b87b8b62ae7e90 with result from task 1764 parsl.dataflow.dflow DEBUG: Task 1764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/25cad6c7-4f36-4ce3-8f88-e44379c04649_isrForCrosstalkSources_855450_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/25cad6c7-4f36-4ce3-8f88-e44379c04649_isrForCrosstalkSources_855450_31.stderr parsl.dataflow.dflow INFO: Task 1466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82c833b38705bbec0c0a18cb02b1fa85 with result from task 1466 parsl.dataflow.dflow DEBUG: Task 1466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f7443be1-caf8-4054-ac34-f652c4d822d6_isrForCrosstalkSources_855421_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f7443be1-caf8-4054-ac34-f652c4d822d6_isrForCrosstalkSources_855421_27.stderr parsl.dataflow.dflow INFO: Task 1346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f011f76cbeb7f0162e8109addd8977ba with result from task 1346 parsl.dataflow.dflow DEBUG: Task 1346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/990b30a9-6bc7-45db-a10a-e0fd10b173c6_isrForCrosstalkSources_855382_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/990b30a9-6bc7-45db-a10a-e0fd10b173c6_isrForCrosstalkSources_855382_48.stderr parsl.dataflow.dflow INFO: Task 1255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d6d79eac74fd74810e182ec97196a7b with result from task 1255 parsl.dataflow.dflow DEBUG: Task 1255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b1acab89-497c-419d-8195-cb47176a9303_isrForCrosstalkSources_855435_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b1acab89-497c-419d-8195-cb47176a9303_isrForCrosstalkSources_855435_57.stderr parsl.dataflow.dflow INFO: Task 1305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 683a8f38b3c518187818139647f3d3f3 with result from task 1305 parsl.dataflow.dflow DEBUG: Task 1305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/66b8cc81-7233-4e0d-bbe8-765c558ab114_isrForCrosstalkSources_855429_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/66b8cc81-7233-4e0d-bbe8-765c558ab114_isrForCrosstalkSources_855429_31.stderr parsl.dataflow.dflow INFO: Task 1405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691cb9d422c67202b8604acf28acf5d2 with result from task 1405 parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:57 proc_lsst.multi:146 [INFO] found job 16964067 in provider escience_a40 proc_lsst.multi INFO: found job 16964067 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow DEBUG: Task 1405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/28e1eccf-ec8c-406a-ad9f-ead44e5820cb_isrForCrosstalkSources_855443_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/28e1eccf-ec8c-406a-ad9f-ead44e5820cb_isrForCrosstalkSources_855443_46.stderr parsl.dataflow.dflow INFO: Task 1363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 609f2b1d4edf4f1cd7a987d132c8f544 with result from task 1363 parsl.dataflow.dflow DEBUG: Task 1363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9486a6e4-734b-4f06-b620-b29712722af2_isrForCrosstalkSources_855387_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9486a6e4-734b-4f06-b620-b29712722af2_isrForCrosstalkSources_855387_53.stderr parsl.dataflow.dflow INFO: Task 1377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7149e27344b35b367a8d7db77b5acba3 with result from task 1377 parsl.dataflow.dflow DEBUG: Task 1377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d2ec1941-9f0e-4d4e-a351-6a862f8638d2_isrForCrosstalkSources_855397_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d2ec1941-9f0e-4d4e-a351-6a862f8638d2_isrForCrosstalkSources_855397_44.stderr parsl.dataflow.dflow INFO: Task 1780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2ddebfde8811eb7bd76e5679037a5b with result from task 1780 parsl.dataflow.dflow DEBUG: Task 1780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6474e59c-aad6-4ceb-ae67-67b8058ac657_isrForCrosstalkSources_855440_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6474e59c-aad6-4ceb-ae67-67b8058ac657_isrForCrosstalkSources_855440_32.stderr parsl.dataflow.dflow INFO: Task 1300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e69664053b4c9e7cc56e7b0e823033 with result from task 1300 parsl.dataflow.dflow DEBUG: Task 1300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/da71700c-0fab-4941-9bb3-d4da42d5d72b_isrForCrosstalkSources_855420_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/da71700c-0fab-4941-9bb3-d4da42d5d72b_isrForCrosstalkSources_855420_59.stderr parsl.dataflow.dflow INFO: Task 1429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead8dad7f78166ba6ba706a7c8864f5d with result from task 1429 parsl.dataflow.dflow DEBUG: Task 1429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/d913ab73-bc61-4571-9c33-2068ced97910_isrForCrosstalkSources_855404_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/d913ab73-bc61-4571-9c33-2068ced97910_isrForCrosstalkSources_855404_8.stderr parsl.dataflow.dflow INFO: Task 1443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a82142d1bf325e8540b23c58a45db0d with result from task 1443 parsl.dataflow.dflow DEBUG: Task 1443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/8a5de12b-16e7-475b-8721-38ef653f6a41_isrForCrosstalkSources_855420_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/8a5de12b-16e7-475b-8721-38ef653f6a41_isrForCrosstalkSources_855420_13.stderr parsl.dataflow.dflow INFO: Task 1514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d8564066a9470f27ca5238fe046b71 with result from task 1514 parsl.dataflow.dflow DEBUG: Task 1514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/477489cf-d7f4-45ea-8d97-e9fcf75af794_isrForCrosstalkSources_855442_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/477489cf-d7f4-45ea-8d97-e9fcf75af794_isrForCrosstalkSources_855442_46.stderr parsl.dataflow.dflow INFO: Task 1504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb6873f2897830dcf9e2942448cbbca with result from task 1504 parsl.dataflow.dflow DEBUG: Task 1504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/474c0589-670d-458d-8ab9-6947d4e6fa1e_isrForCrosstalkSources_855407_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/474c0589-670d-458d-8ab9-6947d4e6fa1e_isrForCrosstalkSources_855407_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:57 proc_lsst.multi:146 [INFO] found job 16964068 in provider escience_a40 proc_lsst.multi INFO: found job 16964068 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b73f6bb7942783a1f395b779bf87a50 with result from task 1353 parsl.dataflow.dflow DEBUG: Task 1353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ae0cf2ba-cce6-414d-853c-ef06a59b1872_isrForCrosstalkSources_855411_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ae0cf2ba-cce6-414d-853c-ef06a59b1872_isrForCrosstalkSources_855411_13.stderr parsl.dataflow.dflow INFO: Task 1401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25972dfea27500b4b9ae91af22d06687 with result from task 1401 parsl.dataflow.dflow DEBUG: Task 1401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8735d1e3-413e-41ce-b9c8-80ed72829584_isrForCrosstalkSources_855445_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8735d1e3-413e-41ce-b9c8-80ed72829584_isrForCrosstalkSources_855445_11.stderr parsl.dataflow.dflow INFO: Task 1421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22fb50433cc1406e44f36140a5da31b3 with result from task 1421 parsl.dataflow.dflow DEBUG: Task 1421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0c8e370f-9ec6-4c4c-923e-aed0c012c23b_isrForCrosstalkSources_855451_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/0c8e370f-9ec6-4c4c-923e-aed0c012c23b_isrForCrosstalkSources_855451_40.stderr parsl.dataflow.dflow INFO: Task 1445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53abf23bcf121e3012c170fe0cd78de4 with result from task 1445 parsl.dataflow.dflow DEBUG: Task 1445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0f194849-81fb-4e67-95cc-e1d7ec72b214_isrForCrosstalkSources_855445_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0f194849-81fb-4e67-95cc-e1d7ec72b214_isrForCrosstalkSources_855445_51.stderr parsl.dataflow.dflow INFO: Task 1242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc8dcbb7ceff68da15fc74ba0f8168a9 with result from task 1242 parsl.dataflow.dflow DEBUG: Task 1242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5069b9c3-e25d-482a-b8f9-b6216483de2d_isrForCrosstalkSources_855400_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5069b9c3-e25d-482a-b8f9-b6216483de2d_isrForCrosstalkSources_855400_52.stderr parsl.dataflow.dflow INFO: Task 1458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c078c5b4ab229af46fa5b92523df073a with result from task 1458 parsl.dataflow.dflow DEBUG: Task 1458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/90e610cc-7633-4ae1-85d3-fa29e122ce4b_isrForCrosstalkSources_855388_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/90e610cc-7633-4ae1-85d3-fa29e122ce4b_isrForCrosstalkSources_855388_46.stderr parsl.dataflow.dflow INFO: Task 1341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a68f7a5dfeb2edf38a094dc5da502709 with result from task 1341 parsl.dataflow.dflow DEBUG: Task 1341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b5455a3f-abad-4a64-9336-5c37c2124f5f_isrForCrosstalkSources_855411_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b5455a3f-abad-4a64-9336-5c37c2124f5f_isrForCrosstalkSources_855411_48.stderr parsl.dataflow.dflow INFO: Task 1461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 949a71e8734258f9298fef2a1de311be with result from task 1461 parsl.dataflow.dflow DEBUG: Task 1461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3c36c0ab-723c-4b59-8cae-b8a530734ab6_isrForCrosstalkSources_855426_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3c36c0ab-723c-4b59-8cae-b8a530734ab6_isrForCrosstalkSources_855426_23.stderr parsl.dataflow.dflow INFO: Task 1439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01696d8c1a6febf9acaac1dcdd6fcaf9 with result from task 1439 parsl.dataflow.dflow DEBUG: Task 1439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e59df41b-5496-4527-8617-38ebc59b7bb2_isrForCrosstalkSources_855454_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e59df41b-5496-4527-8617-38ebc59b7bb2_isrForCrosstalkSources_855454_42.stderr parsl.dataflow.dflow INFO: Task 1335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf375c9ffde13854a8f895b7a3e254b9 with result from task 1335 parsl.dataflow.dflow DEBUG: Task 1335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/971d7850-438c-453f-915a-ac157922f84e_isrForCrosstalkSources_855399_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/971d7850-438c-453f-915a-ac157922f84e_isrForCrosstalkSources_855399_50.stderr parsl.dataflow.dflow INFO: Task 1406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b057ab0e388eac72094434440f8edbe5 with result from task 1406 parsl.dataflow.dflow DEBUG: Task 1406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/600f190f-7e70-4533-b14c-11f4c7347a86_isrForCrosstalkSources_855402_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/600f190f-7e70-4533-b14c-11f4c7347a86_isrForCrosstalkSources_855402_18.stderr parsl.dataflow.dflow INFO: Task 1289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 074e5e72c54854dd22c801d0354863fe with result from task 1289 parsl.dataflow.dflow DEBUG: Task 1289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/89499d58-dde5-48fb-83e6-bc7e2e3e592c_isrForCrosstalkSources_855448_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/89499d58-dde5-48fb-83e6-bc7e2e3e592c_isrForCrosstalkSources_855448_2.stderr parsl.dataflow.dflow INFO: Task 1761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2222e6b5b2e64daa14b00d68b61f3307 with result from task 1761 parsl.dataflow.dflow DEBUG: Task 1761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7c46d2eb-766b-4244-9222-c424f10f9b19_isrForCrosstalkSources_855413_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7c46d2eb-766b-4244-9222-c424f10f9b19_isrForCrosstalkSources_855413_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:58 proc_lsst.multi:146 [INFO] found job 16964069 in provider escience_a40 proc_lsst.multi INFO: found job 16964069 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4a4b911215198fabd558d49d3c54183 with result from task 1433 parsl.dataflow.dflow DEBUG: Task 1433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e65e8cde-ee91-4b49-9c36-73865d45ca68_isrForCrosstalkSources_855428_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e65e8cde-ee91-4b49-9c36-73865d45ca68_isrForCrosstalkSources_855428_53.stderr parsl.dataflow.dflow INFO: Task 1578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 470610a527d22bdf8ceaa580ab65a3ab with result from task 1578 parsl.dataflow.dflow DEBUG: Task 1578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1712e0d2-f64d-4f6e-ab17-3baa9ac02d1e_isrForCrosstalkSources_855428_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1712e0d2-f64d-4f6e-ab17-3baa9ac02d1e_isrForCrosstalkSources_855428_30.stderr parsl.dataflow.dflow INFO: Task 1775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e35c7cd7316c66160c7fa2875bb83ab with result from task 1775 parsl.dataflow.dflow DEBUG: Task 1775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/277b0d74-ad69-4db1-b16b-1eafe4e34014_isrForCrosstalkSources_855410_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/277b0d74-ad69-4db1-b16b-1eafe4e34014_isrForCrosstalkSources_855410_42.stderr parsl.dataflow.dflow INFO: Task 1398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0ee663ab7736e72e3ea12db54131daf with result from task 1398 parsl.dataflow.dflow DEBUG: Task 1398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/df18b367-4ead-498c-9dde-e47849cc576c_isrForCrosstalkSources_855381_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/df18b367-4ead-498c-9dde-e47849cc576c_isrForCrosstalkSources_855381_48.stderr parsl.dataflow.dflow INFO: Task 1298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408578cfc88a0dd039833b01b1a39bbb with result from task 1298 parsl.dataflow.dflow DEBUG: Task 1298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6f59c259-b387-405b-808d-de0d69819e63_isrForCrosstalkSources_855392_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6f59c259-b387-405b-808d-de0d69819e63_isrForCrosstalkSources_855392_57.stderr parsl.dataflow.dflow INFO: Task 1477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8454c45fd21443c69ec5467b733a391a with result from task 1477 parsl.dataflow.dflow DEBUG: Task 1477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7dd82a21-1e10-428e-a6f1-e9da3bc09160_isrForCrosstalkSources_855434_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7dd82a21-1e10-428e-a6f1-e9da3bc09160_isrForCrosstalkSources_855434_14.stderr parsl.dataflow.dflow INFO: Task 1680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2fc3509a1167e206a0c521157c46b4 with result from task 1680 parsl.dataflow.dflow DEBUG: Task 1680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f79a7d1a-8cde-4b3a-ae80-efd4ff935151_isrForCrosstalkSources_855453_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f79a7d1a-8cde-4b3a-ae80-efd4ff935151_isrForCrosstalkSources_855453_46.stderr parsl.dataflow.dflow INFO: Task 1410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea967f3c11115dc46df2afc9308e56c with result from task 1410 parsl.dataflow.dflow DEBUG: Task 1410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9a51e723-3b71-4c2c-9a43-abba468e42d1_isrForCrosstalkSources_855396_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9a51e723-3b71-4c2c-9a43-abba468e42d1_isrForCrosstalkSources_855396_23.stderr parsl.dataflow.dflow INFO: Task 1809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff0fdca7fcce234d17ea8c017ffc5502 with result from task 1809 parsl.dataflow.dflow DEBUG: Task 1809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7a58c13a-4866-4177-b7c1-1d7933cea294_isrForCrosstalkSources_855429_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/7a58c13a-4866-4177-b7c1-1d7933cea294_isrForCrosstalkSources_855429_12.stderr parsl.dataflow.dflow INFO: Task 1472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03fed52e14c32b0b61295b0021e14253 with result from task 1472 parsl.dataflow.dflow DEBUG: Task 1472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/15729995-4f17-44e7-b974-2ce57f727e90_isrForCrosstalkSources_855448_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/15729995-4f17-44e7-b974-2ce57f727e90_isrForCrosstalkSources_855448_58.stderr parsl.dataflow.dflow INFO: Task 1512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c6e456b057eeb6773932eb1cbb9e81a with result from task 1512 parsl.dataflow.dflow DEBUG: Task 1512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94cba6f2-d167-4756-9daf-468fd10b6c12_isrForCrosstalkSources_855448_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94cba6f2-d167-4756-9daf-468fd10b6c12_isrForCrosstalkSources_855448_30.stderr parsl.dataflow.dflow INFO: Task 1396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b6ac0b81de78eacfde41861f5c148e with result from task 1396 parsl.dataflow.dflow DEBUG: Task 1396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/251eff64-b67b-422d-82bb-dd36410d1285_isrForCrosstalkSources_855386_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/251eff64-b67b-422d-82bb-dd36410d1285_isrForCrosstalkSources_855386_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:58 proc_lsst.multi:146 [INFO] found job 16964070 in provider escience_a40 proc_lsst.multi INFO: found job 16964070 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87700f0f91f5334c5576ec6fcccf11e1 with result from task 1397 parsl.dataflow.dflow DEBUG: Task 1397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5884aea8-df75-4b38-9a13-09f81f1cca22_isrForCrosstalkSources_855400_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5884aea8-df75-4b38-9a13-09f81f1cca22_isrForCrosstalkSources_855400_7.stderr parsl.dataflow.dflow INFO: Task 1768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f590e2850b484fd949bdc3319b5d3a04 with result from task 1768 parsl.dataflow.dflow DEBUG: Task 1768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/86ff2a3e-0a67-47e1-8354-0832c6d76e55_isrForCrosstalkSources_855393_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/86ff2a3e-0a67-47e1-8354-0832c6d76e55_isrForCrosstalkSources_855393_60.stderr parsl.dataflow.dflow INFO: Task 1379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cda824fce314c8292bdff1c60c0e7f2 with result from task 1379 parsl.dataflow.dflow DEBUG: Task 1379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6b9bac88-2825-45ab-a621-fb2c0fe27619_isrForCrosstalkSources_855452_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6b9bac88-2825-45ab-a621-fb2c0fe27619_isrForCrosstalkSources_855452_58.stderr parsl.dataflow.dflow INFO: Task 1742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 959162171b919674722e0a4dbac01edf with result from task 1742 parsl.dataflow.dflow DEBUG: Task 1742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ba90c059-42b3-4510-a046-295483118435_isrForCrosstalkSources_855452_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ba90c059-42b3-4510-a046-295483118435_isrForCrosstalkSources_855452_40.stderr parsl.dataflow.dflow INFO: Task 1747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ce316cfc0587a19995e5052981dd783 with result from task 1747 parsl.dataflow.dflow DEBUG: Task 1747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f96fe12a-4054-48aa-9fdf-bed6922ae76d_isrForCrosstalkSources_855420_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f96fe12a-4054-48aa-9fdf-bed6922ae76d_isrForCrosstalkSources_855420_23.stderr parsl.dataflow.dflow INFO: Task 1531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 859bd1a169a8a05810413d19b004609c with result from task 1531 parsl.dataflow.dflow DEBUG: Task 1531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/8cfe2e41-f295-4604-ab09-8bb165147745_isrForCrosstalkSources_855452_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/8cfe2e41-f295-4604-ab09-8bb165147745_isrForCrosstalkSources_855452_18.stderr parsl.dataflow.dflow INFO: Task 1430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93515faeced4ea13ee507ec92120cddb with result from task 1430 parsl.dataflow.dflow DEBUG: Task 1430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/76bc1419-31dc-4366-8f01-0a699dc98987_isrForCrosstalkSources_855387_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/76bc1419-31dc-4366-8f01-0a699dc98987_isrForCrosstalkSources_855387_52.stderr parsl.dataflow.dflow INFO: Task 1400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07e1573495c486e63409972bb3803ead with result from task 1400 parsl.dataflow.dflow DEBUG: Task 1400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e5357042-1f36-4717-b602-b12d431aeb1f_isrForCrosstalkSources_855394_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e5357042-1f36-4717-b602-b12d431aeb1f_isrForCrosstalkSources_855394_55.stderr parsl.dataflow.dflow INFO: Task 1467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb7bdcf8c0cd3122147781c2d95f688d with result from task 1467 parsl.dataflow.dflow DEBUG: Task 1467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bafe2bf3-0373-4779-aadd-1e2fa1f4e9c1_isrForCrosstalkSources_855431_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/bafe2bf3-0373-4779-aadd-1e2fa1f4e9c1_isrForCrosstalkSources_855431_15.stderr parsl.dataflow.dflow INFO: Task 1690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a759bc2a9bd432ac9396b93375033c8 with result from task 1690 parsl.dataflow.dflow DEBUG: Task 1690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/61cb4592-809b-4006-8849-49ace21aab5c_isrForCrosstalkSources_855426_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/61cb4592-809b-4006-8849-49ace21aab5c_isrForCrosstalkSources_855426_33.stderr parsl.dataflow.dflow INFO: Task 1453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdb35f525966c4641c51f184b485f877 with result from task 1453 parsl.dataflow.dflow DEBUG: Task 1453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/964831aa-44bb-4e6d-b852-2a7b2f4631d4_isrForCrosstalkSources_855435_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/964831aa-44bb-4e6d-b852-2a7b2f4631d4_isrForCrosstalkSources_855435_16.stderr parsl.dataflow.dflow INFO: Task 1487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d324e914d9125c70695905d0a9a5f184 with result from task 1487 parsl.dataflow.dflow DEBUG: Task 1487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/23a2e29a-3745-483c-8aa7-000efd19e459_isrForCrosstalkSources_855384_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/23a2e29a-3745-483c-8aa7-000efd19e459_isrForCrosstalkSources_855384_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:52:59 proc_lsst.multi:146 [INFO] found job 16964071 in provider escience_a40 proc_lsst.multi INFO: found job 16964071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f4933778c92f99c678754aa28853202 with result from task 1755 parsl.dataflow.dflow DEBUG: Task 1755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e043fe1-b00f-4d84-b7fb-3f714b034101_isrForCrosstalkSources_855384_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e043fe1-b00f-4d84-b7fb-3f714b034101_isrForCrosstalkSources_855384_57.stderr parsl.dataflow.dflow INFO: Task 1750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7129bc81426cb203746446eb59c180f with result from task 1750 parsl.dataflow.dflow DEBUG: Task 1750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0ffe8e2-2e70-4ec6-96b1-7163c86fb6a7_isrForCrosstalkSources_855384_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e0ffe8e2-2e70-4ec6-96b1-7163c86fb6a7_isrForCrosstalkSources_855384_62.stderr parsl.dataflow.dflow INFO: Task 1529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91c7d40e40b698c3482528f8b19cfbf with result from task 1529 parsl.dataflow.dflow DEBUG: Task 1529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/020dcde1-e20f-4107-bd0b-e154235b1f45_isrForCrosstalkSources_855450_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/020dcde1-e20f-4107-bd0b-e154235b1f45_isrForCrosstalkSources_855450_51.stderr parsl.dataflow.dflow INFO: Task 1505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72b413566b74e0333d4a546ca6539606 with result from task 1505 parsl.dataflow.dflow DEBUG: Task 1505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6aa6f466-f412-4d7d-b982-bfe76e5dba98_isrForCrosstalkSources_855446_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6aa6f466-f412-4d7d-b982-bfe76e5dba98_isrForCrosstalkSources_855446_23.stderr parsl.dataflow.dflow INFO: Task 1413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9edd5ab7782316ea6d99e66d33359fef with result from task 1413 parsl.dataflow.dflow DEBUG: Task 1413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/400b5e4e-c33f-4cf5-bf95-06cb7b0fbca5_isrForCrosstalkSources_855386_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/400b5e4e-c33f-4cf5-bf95-06cb7b0fbca5_isrForCrosstalkSources_855386_10.stderr parsl.dataflow.dflow INFO: Task 1520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4630fc0d533f718e12a67cacfba54c3 with result from task 1520 parsl.dataflow.dflow DEBUG: Task 1520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/07e7c45b-c608-46b1-9cc0-95067304f1ba_isrForCrosstalkSources_855383_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/07e7c45b-c608-46b1-9cc0-95067304f1ba_isrForCrosstalkSources_855383_4.stderr parsl.dataflow.dflow INFO: Task 1791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50c338d2cb851f5fd09ff2bfcab860fa with result from task 1791 parsl.dataflow.dflow DEBUG: Task 1791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b6fe4a45-0d0b-47cb-a27b-efb6ae0eef7a_isrForCrosstalkSources_855428_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b6fe4a45-0d0b-47cb-a27b-efb6ae0eef7a_isrForCrosstalkSources_855428_26.stderr parsl.dataflow.dflow INFO: Task 1804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf2dd613a05470f9274a247e3c302bc0 with result from task 1804 parsl.dataflow.dflow DEBUG: Task 1804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/46df093c-4e7b-47fd-a3b5-ba9df0cc6e62_isrForCrosstalkSources_855413_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/46df093c-4e7b-47fd-a3b5-ba9df0cc6e62_isrForCrosstalkSources_855413_62.stderr parsl.dataflow.dflow INFO: Task 1736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6885c2e55168643e66a5e26de96354e7 with result from task 1736 parsl.dataflow.dflow DEBUG: Task 1736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0b2eab4e-904d-4d04-8633-f67a929bc488_isrForCrosstalkSources_855402_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/0b2eab4e-904d-4d04-8633-f67a929bc488_isrForCrosstalkSources_855402_29.stderr parsl.dataflow.dflow INFO: Task 1250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39485797f5cdaba5b9300ee989df993 with result from task 1250 parsl.dataflow.dflow DEBUG: Task 1250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8f2b549e-0f28-462e-aad7-5ef2b33e6e4d_isrForCrosstalkSources_855412_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8f2b549e-0f28-462e-aad7-5ef2b33e6e4d_isrForCrosstalkSources_855412_57.stderr parsl.dataflow.dflow INFO: Task 1479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 545d70e73161340aeeaea6fa7f61cd64 with result from task 1479 parsl.dataflow.dflow DEBUG: Task 1479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ec137fdd-fb3d-4fd2-b168-843aa25e272b_isrForCrosstalkSources_855433_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ec137fdd-fb3d-4fd2-b168-843aa25e272b_isrForCrosstalkSources_855433_31.stderr parsl.dataflow.dflow INFO: Task 1448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 677d2e2fa0f7036b15b4cd8b45c0910d with result from task 1448 parsl.dataflow.dflow DEBUG: Task 1448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f1f48ae9-6994-4fde-b1ec-553bc39ca626_isrForCrosstalkSources_855417_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f1f48ae9-6994-4fde-b1ec-553bc39ca626_isrForCrosstalkSources_855417_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:00 proc_lsst.multi:146 [INFO] found job 16964072 in provider escience_a40 proc_lsst.multi INFO: found job 16964072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93658fef48cb95239e21567718f92ab4 with result from task 1481 parsl.dataflow.dflow DEBUG: Task 1481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2ac88759-74a3-469d-bc5a-a688feba710c_isrForCrosstalkSources_855455_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2ac88759-74a3-469d-bc5a-a688feba710c_isrForCrosstalkSources_855455_21.stderr parsl.dataflow.dflow INFO: Task 1751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c449f835c63909c297a039524cadc7ea with result from task 1751 parsl.dataflow.dflow DEBUG: Task 1751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ce1ed493-0de8-4184-bbdb-4e9fc762a0ba_isrForCrosstalkSources_855403_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ce1ed493-0de8-4184-bbdb-4e9fc762a0ba_isrForCrosstalkSources_855403_7.stderr parsl.dataflow.dflow INFO: Task 1772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac389a8ff592628dd99239dd8b2917b8 with result from task 1772 parsl.dataflow.dflow DEBUG: Task 1772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/688b11f6-1bed-4fd6-981f-e4340b11aecf_isrForCrosstalkSources_855452_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/688b11f6-1bed-4fd6-981f-e4340b11aecf_isrForCrosstalkSources_855452_51.stderr parsl.dataflow.dflow INFO: Task 1677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca6175eec1121853acb00edae12c4e82 with result from task 1677 parsl.dataflow.dflow DEBUG: Task 1677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fcbd7c1e-66df-468b-b11a-19e14e50f693_isrForCrosstalkSources_855409_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fcbd7c1e-66df-468b-b11a-19e14e50f693_isrForCrosstalkSources_855409_40.stderr parsl.dataflow.dflow INFO: Task 1778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc15637379a2384ee085ed12cba32a4 with result from task 1778 parsl.dataflow.dflow DEBUG: Task 1778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/050287c7-5bee-4886-b278-c54fdc9d6981_isrForCrosstalkSources_855398_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/050287c7-5bee-4886-b278-c54fdc9d6981_isrForCrosstalkSources_855398_32.stderr parsl.dataflow.dflow INFO: Task 1500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b00feed41e53c5c709230724258bb336 with result from task 1500 parsl.dataflow.dflow DEBUG: Task 1500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4bb21ae8-d741-4a4d-9998-6eb3ebe0540f_isrForCrosstalkSources_855430_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4bb21ae8-d741-4a4d-9998-6eb3ebe0540f_isrForCrosstalkSources_855430_7.stderr parsl.dataflow.dflow INFO: Task 1763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 423ffd6fdd8e08262ba79558dc6a0ffa with result from task 1763 parsl.dataflow.dflow DEBUG: Task 1763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6a49aa4f-c0c2-47ff-9cc5-99d321d96d56_isrForCrosstalkSources_855380_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6a49aa4f-c0c2-47ff-9cc5-99d321d96d56_isrForCrosstalkSources_855380_56.stderr parsl.dataflow.dflow INFO: Task 1395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8cfebd20ed5d5abb1e11dc328519b9 with result from task 1395 parsl.dataflow.dflow DEBUG: Task 1395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/356131d0-ecdc-4d27-91c7-1bd46bf63aaf_isrForCrosstalkSources_855439_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/356131d0-ecdc-4d27-91c7-1bd46bf63aaf_isrForCrosstalkSources_855439_18.stderr parsl.dataflow.dflow INFO: Task 1415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b44687a8ab8fa5c70d112beb2e66841 with result from task 1415 parsl.dataflow.dflow DEBUG: Task 1415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f6ebd18a-38a4-4a87-af22-32ccd1294fef_isrForCrosstalkSources_855418_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f6ebd18a-38a4-4a87-af22-32ccd1294fef_isrForCrosstalkSources_855418_55.stderr parsl.dataflow.dflow INFO: Task 1785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6038dd3745a2c25ab0a132dee345d535 with result from task 1785 parsl.dataflow.dflow DEBUG: Task 1785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0cbbabae-67f7-4971-8194-89a54e8a625d_isrForCrosstalkSources_855454_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/0cbbabae-67f7-4971-8194-89a54e8a625d_isrForCrosstalkSources_855454_33.stderr parsl.dataflow.dflow INFO: Task 1370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7ee53bb87916e651484b3572d4c848e with result from task 1370 parsl.dataflow.dflow DEBUG: Task 1370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b00a077a-c1be-4c28-b312-8d52de0f41f5_isrForCrosstalkSources_855432_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b00a077a-c1be-4c28-b312-8d52de0f41f5_isrForCrosstalkSources_855432_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:00 proc_lsst.multi:146 [INFO] found job 16964073 in provider escience_a40 proc_lsst.multi INFO: found job 16964073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e747b594aa4d6390e23576cdafff3d9 with result from task 1417 parsl.dataflow.dflow DEBUG: Task 1417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cac2c76-d4e5-4730-8f4a-e352609beae2_isrForCrosstalkSources_855403_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cac2c76-d4e5-4730-8f4a-e352609beae2_isrForCrosstalkSources_855403_15.stderr parsl.dataflow.dflow INFO: Task 1770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f49bf88fa89be3e021cd8c67c969933 with result from task 1770 parsl.dataflow.dflow DEBUG: Task 1770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/71b10347-ee5b-4bab-b570-2376c0895018_isrForCrosstalkSources_855380_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/71b10347-ee5b-4bab-b570-2376c0895018_isrForCrosstalkSources_855380_17.stderr parsl.dataflow.dflow INFO: Task 1516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b683c69265a3f81dec55bfac22e5b149 with result from task 1516 parsl.dataflow.dflow DEBUG: Task 1516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ab2810ed-c4ed-4d49-8e97-018de2003ca5_isrForCrosstalkSources_855444_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/ab2810ed-c4ed-4d49-8e97-018de2003ca5_isrForCrosstalkSources_855444_15.stderr parsl.dataflow.dflow INFO: Task 1387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81f1548eecc0a023ade9b17e51678c97 with result from task 1387 parsl.dataflow.dflow DEBUG: Task 1387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b96cf070-4bfe-4c7b-b4c6-c17e6c1a3fbd_isrForCrosstalkSources_855419_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/b96cf070-4bfe-4c7b-b4c6-c17e6c1a3fbd_isrForCrosstalkSources_855419_16.stderr parsl.dataflow.dflow INFO: Task 1508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55b0e15fc1dadf01e1c399502d36d604 with result from task 1508 parsl.dataflow.dflow DEBUG: Task 1508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ba87233c-d026-4486-924c-a3b20163025f_isrForCrosstalkSources_855455_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ba87233c-d026-4486-924c-a3b20163025f_isrForCrosstalkSources_855455_2.stderr parsl.dataflow.dflow INFO: Task 1714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c82df0b9137dcf69c128aa8a7ebaec81 with result from task 1714 parsl.dataflow.dflow DEBUG: Task 1714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/95b999a8-330f-4fb9-8a69-f63cfd90ae21_isrForCrosstalkSources_855394_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/95b999a8-330f-4fb9-8a69-f63cfd90ae21_isrForCrosstalkSources_855394_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:01 proc_lsst.multi:146 [INFO] found job 16964074 in provider escience_a40 proc_lsst.multi INFO: found job 16964074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c33fa6335a4933f61381e6d416f74550 with result from task 1465 parsl.dataflow.dflow DEBUG: Task 1465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7cdd2fa0-f152-4d7f-8a7f-9e7ef820b994_isrForCrosstalkSources_855450_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/7cdd2fa0-f152-4d7f-8a7f-9e7ef820b994_isrForCrosstalkSources_855450_5.stderr parsl.dataflow.dflow INFO: Task 1718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d3569a262c5870385ee5a1cc9abda98 with result from task 1718 parsl.dataflow.dflow DEBUG: Task 1718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ab8d196c-7c79-435e-b71c-59e3e6da9129_isrForCrosstalkSources_855415_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ab8d196c-7c79-435e-b71c-59e3e6da9129_isrForCrosstalkSources_855415_33.stderr parsl.dataflow.dflow INFO: Task 1525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e0fb7e6deb4e7dbf47ef577973c175 with result from task 1525 parsl.dataflow.dflow DEBUG: Task 1525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/bdf22790-809d-4f82-b1ba-3553a647ab22_isrForCrosstalkSources_855398_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/bdf22790-809d-4f82-b1ba-3553a647ab22_isrForCrosstalkSources_855398_8.stderr parsl.dataflow.dflow INFO: Task 1385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5158be2c1cb0a9278d1f7262b3b9936f with result from task 1385 parsl.dataflow.dflow DEBUG: Task 1385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2a6dfab8-1e22-4930-82b5-7bf255a4da18_isrForCrosstalkSources_855456_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2a6dfab8-1e22-4930-82b5-7bf255a4da18_isrForCrosstalkSources_855456_31.stderr parsl.dataflow.dflow INFO: Task 1831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c653a7b98ef5c20454d4f42bcac31a with result from task 1831 parsl.dataflow.dflow DEBUG: Task 1831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f954c276-9690-40ce-b2ca-d6c67558a84b_isrForCrosstalkSources_855404_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f954c276-9690-40ce-b2ca-d6c67558a84b_isrForCrosstalkSources_855404_45.stderr parsl.dataflow.dflow INFO: Task 1388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8d952adcdc67af079ad97d8f28a79f7 with result from task 1388 parsl.dataflow.dflow DEBUG: Task 1388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6ec44280-9aaf-433d-bed1-de9ba89909c9_isrForCrosstalkSources_855425_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6ec44280-9aaf-433d-bed1-de9ba89909c9_isrForCrosstalkSources_855425_3.stderr parsl.dataflow.dflow INFO: Task 1436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7886a76b13524a09f56bb17af7e68320 with result from task 1436 parsl.dataflow.dflow DEBUG: Task 1436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2af8eb3f-0fe0-446d-9bcc-fea862240180_isrForCrosstalkSources_855443_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2af8eb3f-0fe0-446d-9bcc-fea862240180_isrForCrosstalkSources_855443_42.stderr parsl.dataflow.dflow INFO: Task 1485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf862e942a2c5f252866503ac193914 with result from task 1485 parsl.dataflow.dflow DEBUG: Task 1485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/524f5dcd-21ae-42b5-a14f-2cb7b9ab1068_isrForCrosstalkSources_855409_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/524f5dcd-21ae-42b5-a14f-2cb7b9ab1068_isrForCrosstalkSources_855409_9.stderr parsl.dataflow.dflow INFO: Task 1506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e2f3359a1906519d61227598dc1f38 with result from task 1506 parsl.dataflow.dflow DEBUG: Task 1506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/512e8d32-7314-466f-83ff-a69f3c96f315_isrForCrosstalkSources_855453_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/512e8d32-7314-466f-83ff-a69f3c96f315_isrForCrosstalkSources_855453_38.stderr parsl.dataflow.dflow INFO: Task 1367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2edb702030135a5c54fb9b4a4209cff1 with result from task 1367 parsl.dataflow.dflow DEBUG: Task 1367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a253ab8d-1274-4d48-9ac8-0ba0369238f0_isrForCrosstalkSources_855406_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a253ab8d-1274-4d48-9ac8-0ba0369238f0_isrForCrosstalkSources_855406_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:01 proc_lsst.multi:146 [INFO] found job 16964075 in provider escience_a40 proc_lsst.multi INFO: found job 16964075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e79800cfb614cf746ffec7a9a4b43b7f with result from task 1527 parsl.dataflow.dflow DEBUG: Task 1527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46346409-994f-4249-bfcf-7b49e55a950d_isrForCrosstalkSources_855419_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46346409-994f-4249-bfcf-7b49e55a950d_isrForCrosstalkSources_855419_24.stderr parsl.dataflow.dflow INFO: Task 1450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d10791a91b5379e7b2165e6d63a34e92 with result from task 1450 parsl.dataflow.dflow DEBUG: Task 1450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/24af258a-ea16-4be9-88b1-6414e946c60b_isrForCrosstalkSources_855443_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/24af258a-ea16-4be9-88b1-6414e946c60b_isrForCrosstalkSources_855443_35.stderr parsl.dataflow.dflow INFO: Task 1723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03a3820efac4bdbf7f10c4ed796d8a86 with result from task 1723 parsl.dataflow.dflow DEBUG: Task 1723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/74dc9071-d140-4bb7-9f0f-24c275564e80_isrForCrosstalkSources_855455_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/74dc9071-d140-4bb7-9f0f-24c275564e80_isrForCrosstalkSources_855455_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:02 proc_lsst.multi:146 [INFO] found job 16964076 in provider escience_a40 proc_lsst.multi INFO: found job 16964076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a48196984b2a9b5d941a6ca9be257d9 with result from task 1721 parsl.dataflow.dflow DEBUG: Task 1721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/73172a8e-f080-4f43-b3a7-781bc632b218_isrForCrosstalkSources_855396_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/73172a8e-f080-4f43-b3a7-781bc632b218_isrForCrosstalkSources_855396_17.stderr parsl.dataflow.dflow INFO: Task 1536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ba6224eee612a211a45db6c5ea238ac with result from task 1536 parsl.dataflow.dflow DEBUG: Task 1536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0ddfcac2-1355-4805-b008-c6855c526783_isrForCrosstalkSources_855429_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0ddfcac2-1355-4805-b008-c6855c526783_isrForCrosstalkSources_855429_57.stderr parsl.dataflow.dflow INFO: Task 1526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e71684c282cb5ba4eb41d29e3ced39d with result from task 1526 parsl.dataflow.dflow DEBUG: Task 1526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/75b7a3a0-3b2d-432c-8d2a-53dd7c6d0de9_isrForCrosstalkSources_855450_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/75b7a3a0-3b2d-432c-8d2a-53dd7c6d0de9_isrForCrosstalkSources_855450_28.stderr parsl.dataflow.dflow INFO: Task 1845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4168bdc411f9e60e8bbda91b6656cc with result from task 1845 parsl.dataflow.dflow DEBUG: Task 1845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1e41b010-8bf4-4b95-bc6b-7fa62a6252a9_isrForCrosstalkSources_855427_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1e41b010-8bf4-4b95-bc6b-7fa62a6252a9_isrForCrosstalkSources_855427_27.stderr parsl.dataflow.dflow INFO: Task 1799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20c3153101203f417cb857f6ad2b36c8 with result from task 1799 parsl.dataflow.dflow DEBUG: Task 1799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3f21396d-2a9d-43ba-bdb8-3b765dd29052_isrForCrosstalkSources_855400_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3f21396d-2a9d-43ba-bdb8-3b765dd29052_isrForCrosstalkSources_855400_30.stderr parsl.dataflow.dflow INFO: Task 1752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4372432ecd107a6b141d11fa68fcf87 with result from task 1752 parsl.dataflow.dflow DEBUG: Task 1752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/aebe3e93-61ce-4ff5-b4bc-57918e95ad4e_isrForCrosstalkSources_855410_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/aebe3e93-61ce-4ff5-b4bc-57918e95ad4e_isrForCrosstalkSources_855410_7.stderr parsl.dataflow.dflow INFO: Task 1786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 845ac5ace1dc994c0490cb47cd527f3d with result from task 1786 parsl.dataflow.dflow DEBUG: Task 1786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/df4a6195-dd91-4361-a484-43b6018812a8_isrForCrosstalkSources_855419_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/df4a6195-dd91-4361-a484-43b6018812a8_isrForCrosstalkSources_855419_48.stderr parsl.dataflow.dflow INFO: Task 1748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb10728f4f4aa5555b62c72dc09674bd with result from task 1748 parsl.dataflow.dflow DEBUG: Task 1748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c5385ff0-3604-49ed-a530-9dd2046b5d65_isrForCrosstalkSources_855419_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c5385ff0-3604-49ed-a530-9dd2046b5d65_isrForCrosstalkSources_855419_30.stderr parsl.dataflow.dflow INFO: Task 1407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0a8bb4534d7acd9e43bb1d3b013f994 with result from task 1407 parsl.dataflow.dflow DEBUG: Task 1407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a772475f-96db-490b-8c76-f81f1fd1dbb9_isrForCrosstalkSources_855407_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/a772475f-96db-490b-8c76-f81f1fd1dbb9_isrForCrosstalkSources_855407_15.stderr parsl.dataflow.dflow INFO: Task 1719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed22d0829471b831d43ca6e51885590e with result from task 1719 parsl.dataflow.dflow DEBUG: Task 1719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5650c506-2ae2-493b-bc2a-64ae9a06e258_isrForCrosstalkSources_855404_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5650c506-2ae2-493b-bc2a-64ae9a06e258_isrForCrosstalkSources_855404_51.stderr parsl.dataflow.dflow INFO: Task 1678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5bed85fd461510746ae64ef0250a9bb with result from task 1678 parsl.dataflow.dflow DEBUG: Task 1678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c5fd62da-d3b3-4c15-9237-b791e7dfca77_isrForCrosstalkSources_855397_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c5fd62da-d3b3-4c15-9237-b791e7dfca77_isrForCrosstalkSources_855397_40.stderr parsl.dataflow.dflow INFO: Task 1823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d71677e8c57d9fe2dd9d8129376c621b with result from task 1823 parsl.dataflow.dflow DEBUG: Task 1823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dee22110-bc50-4d50-84b3-d94197c64419_isrForCrosstalkSources_855396_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/dee22110-bc50-4d50-84b3-d94197c64419_isrForCrosstalkSources_855396_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:03 proc_lsst.multi:146 [INFO] found job 16964077 in provider escience_a40 proc_lsst.multi INFO: found job 16964077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1202d862ca748ba36633d2b2fb7d67 with result from task 1743 parsl.dataflow.dflow DEBUG: Task 1743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9cd382b3-a385-4c09-9548-8bbb193d4c77_isrForCrosstalkSources_855428_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9cd382b3-a385-4c09-9548-8bbb193d4c77_isrForCrosstalkSources_855428_39.stderr parsl.dataflow.dflow INFO: Task 1454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f04ffa8615e24cc65ca6b5313c6ccbf0 with result from task 1454 parsl.dataflow.dflow DEBUG: Task 1454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/4d1af1f6-4db0-452e-8619-91fdbdcc7be1_isrForCrosstalkSources_855455_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/4d1af1f6-4db0-452e-8619-91fdbdcc7be1_isrForCrosstalkSources_855455_30.stderr parsl.dataflow.dflow INFO: Task 1720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75aa3d2532eaff64385713b66bfbf71b with result from task 1720 parsl.dataflow.dflow DEBUG: Task 1720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/98fd582a-7fad-4d17-9310-ad30a207436f_isrForCrosstalkSources_855397_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/98fd582a-7fad-4d17-9310-ad30a207436f_isrForCrosstalkSources_855397_35.stderr parsl.dataflow.dflow INFO: Task 1726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f536c1afcbac63464bbd548ec5b5334 with result from task 1726 parsl.dataflow.dflow DEBUG: Task 1726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0892d2a7-4889-4d6d-b4ef-cf188bef40fe_isrForCrosstalkSources_855417_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0892d2a7-4889-4d6d-b4ef-cf188bef40fe_isrForCrosstalkSources_855417_9.stderr parsl.dataflow.dflow INFO: Task 1715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acbf3a62e4bcb1882c4b91fd1dc9819a with result from task 1715 parsl.dataflow.dflow DEBUG: Task 1715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2f5f68d7-35a6-42eb-8265-cd01024994b0_isrForCrosstalkSources_855379_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2f5f68d7-35a6-42eb-8265-cd01024994b0_isrForCrosstalkSources_855379_48.stderr parsl.dataflow.dflow INFO: Task 1603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 181eece884b121a23da13fc836e93894 with result from task 1603 parsl.dataflow.dflow DEBUG: Task 1603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ca438309-7033-40e3-b0de-27bc6e89a1c3_isrForCrosstalkSources_855450_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ca438309-7033-40e3-b0de-27bc6e89a1c3_isrForCrosstalkSources_855450_41.stderr parsl.dataflow.dflow INFO: Task 1492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b225482f93e3fddd65b23a9ae3f1a55e with result from task 1492 parsl.dataflow.dflow DEBUG: Task 1492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3f6eb9b4-473d-4c1e-bf5a-51367fbdc56c_isrForCrosstalkSources_855420_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3f6eb9b4-473d-4c1e-bf5a-51367fbdc56c_isrForCrosstalkSources_855420_38.stderr parsl.dataflow.dflow INFO: Task 1717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b5aaa2b1e8688904caa81d4e01f7ab9 with result from task 1717 parsl.dataflow.dflow DEBUG: Task 1717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/96317402-358a-46a3-99d9-847af6c87701_isrForCrosstalkSources_855410_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/96317402-358a-46a3-99d9-847af6c87701_isrForCrosstalkSources_855410_29.stderr parsl.dataflow.dflow INFO: Task 1753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c624e1a8ee03a0cc19f0384108068f2 with result from task 1753 parsl.dataflow.dflow DEBUG: Task 1753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de1f08af-5d7b-43d8-baf4-55a655068e37_isrForCrosstalkSources_855449_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de1f08af-5d7b-43d8-baf4-55a655068e37_isrForCrosstalkSources_855449_55.stderr parsl.dataflow.dflow INFO: Task 1509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc42253840ace406d912212ecc004c8f with result from task 1509 parsl.dataflow.dflow DEBUG: Task 1509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/40ad6e44-7f78-4755-8ebb-35785b94ec12_isrForCrosstalkSources_855414_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/40ad6e44-7f78-4755-8ebb-35785b94ec12_isrForCrosstalkSources_855414_24.stderr parsl.dataflow.dflow INFO: Task 1746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06a3f4be59fc0768b25ee5a294f3f1e with result from task 1746 parsl.dataflow.dflow DEBUG: Task 1746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/945198b5-5785-4fb1-b642-087959952f4c_isrForCrosstalkSources_855441_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/945198b5-5785-4fb1-b642-087959952f4c_isrForCrosstalkSources_855441_53.stderr parsl.dataflow.dflow INFO: Task 1725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267f9d23e8b7d9a7cd65f923a890a2ea with result from task 1725 parsl.dataflow.dflow DEBUG: Task 1725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/65480f8c-4ce3-4c87-b61e-22653815c812_isrForCrosstalkSources_855421_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/65480f8c-4ce3-4c87-b61e-22653815c812_isrForCrosstalkSources_855421_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:03 proc_lsst.multi:146 [INFO] found job 16964078 in provider escience_a40 proc_lsst.multi INFO: found job 16964078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1118e4c4f9f0a1a3f2c2c45f4ee8745b with result from task 1419 parsl.dataflow.dflow DEBUG: Task 1419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6630b181-bca6-4be3-9816-adc8cff6c554_isrForCrosstalkSources_855420_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6630b181-bca6-4be3-9816-adc8cff6c554_isrForCrosstalkSources_855420_30.stderr parsl.dataflow.dflow INFO: Task 1740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39545a4c058dcb17a9c9010b69b293b with result from task 1740 parsl.dataflow.dflow DEBUG: Task 1740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/11fea568-f789-4716-858c-c3f425c4742f_isrForCrosstalkSources_855395_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/11fea568-f789-4716-858c-c3f425c4742f_isrForCrosstalkSources_855395_21.stderr parsl.dataflow.dflow INFO: Task 1356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12eb1b0d2999cb89a16066447082dc97 with result from task 1356 parsl.dataflow.dflow DEBUG: Task 1356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/16ffbede-ca4c-47b1-94bb-26e90584e587_isrForCrosstalkSources_855420_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/16ffbede-ca4c-47b1-94bb-26e90584e587_isrForCrosstalkSources_855420_54.stderr parsl.dataflow.dflow INFO: Task 1252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26ee661bf66d80a540e8e4a33d84a9a5 with result from task 1252 parsl.dataflow.dflow DEBUG: Task 1252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d9829a2a-bd89-46e6-b11c-8a2d06cc689e_isrForCrosstalkSources_855430_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d9829a2a-bd89-46e6-b11c-8a2d06cc689e_isrForCrosstalkSources_855430_28.stderr parsl.dataflow.dflow INFO: Task 1716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 609573a1f9dc952bf1affb8bf63eb3a7 with result from task 1716 parsl.dataflow.dflow DEBUG: Task 1716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/408ad9b0-9d0e-49d5-b497-93e6226d5d79_isrForCrosstalkSources_855401_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/408ad9b0-9d0e-49d5-b497-93e6226d5d79_isrForCrosstalkSources_855401_55.stderr parsl.dataflow.dflow INFO: Task 1722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb32d665162190a92ebbf666de9b9562 with result from task 1722 parsl.dataflow.dflow DEBUG: Task 1722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e18181f8-a357-4dfd-b06f-490f7b18bc34_isrForCrosstalkSources_855458_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e18181f8-a357-4dfd-b06f-490f7b18bc34_isrForCrosstalkSources_855458_23.stderr parsl.dataflow.dflow INFO: Task 1519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec61dafc59af5a155fec0125d8b0381 with result from task 1519 parsl.dataflow.dflow DEBUG: Task 1519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa5fc20a-9ef8-4baa-a3ae-c183c9d249c3_isrForCrosstalkSources_855408_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa5fc20a-9ef8-4baa-a3ae-c183c9d249c3_isrForCrosstalkSources_855408_18.stderr parsl.dataflow.dflow INFO: Task 1803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2588f48f77f8a8cee110cb66fdcab0e with result from task 1803 parsl.dataflow.dflow DEBUG: Task 1803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c2d96fa2-2ea4-41af-864e-10b72be8a0a0_isrForCrosstalkSources_855431_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c2d96fa2-2ea4-41af-864e-10b72be8a0a0_isrForCrosstalkSources_855431_36.stderr parsl.dataflow.dflow INFO: Task 1390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 392e8afd15ea884951416898f52e3b52 with result from task 1390 parsl.dataflow.dflow DEBUG: Task 1390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cee2e386-00df-445d-b2e3-0e51b96e80c2_isrForCrosstalkSources_855454_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/cee2e386-00df-445d-b2e3-0e51b96e80c2_isrForCrosstalkSources_855454_5.stderr parsl.dataflow.dflow INFO: Task 1468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03778bd064035db39ae95e0149243052 with result from task 1468 parsl.dataflow.dflow DEBUG: Task 1468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f4d1c706-7978-4004-8168-075ad18f255c_isrForCrosstalkSources_855424_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f4d1c706-7978-4004-8168-075ad18f255c_isrForCrosstalkSources_855424_25.stderr parsl.dataflow.dflow INFO: Task 1470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d125c2a82d7eee5f9bb19693b9190cf9 with result from task 1470 parsl.dataflow.dflow DEBUG: Task 1470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/73ad37f6-d4d4-4b72-a5cb-e4b5a11bca0f_isrForCrosstalkSources_855438_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/73ad37f6-d4d4-4b72-a5cb-e4b5a11bca0f_isrForCrosstalkSources_855438_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:04 proc_lsst.multi:146 [INFO] found job 16964079 in provider escience_a40 proc_lsst.multi INFO: found job 16964079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:04 proc_lsst.multi:146 [INFO] found job 16964080 in provider escience_a40 proc_lsst.multi INFO: found job 16964080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 1503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a40368a11709f855ab195a2848402078 with result from task 1503 parsl.dataflow.dflow DEBUG: Task 1503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7337dd1e-dfac-40c3-a6ac-1d973e35205e_isrForCrosstalkSources_855445_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7337dd1e-dfac-40c3-a6ac-1d973e35205e_isrForCrosstalkSources_855445_29.stderr parsl.dataflow.dflow INFO: Task 1423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 130a6e835674bcea371b1f55e1b3e99b with result from task 1423 parsl.dataflow.dflow DEBUG: Task 1423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b84b73d7-1ca7-4e10-9507-fb4f9b0733f6_isrForCrosstalkSources_855442_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b84b73d7-1ca7-4e10-9507-fb4f9b0733f6_isrForCrosstalkSources_855442_53.stderr parsl.dataflow.dflow INFO: Task 1491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b69cb842fdee62a39a0d9f948834f3fe with result from task 1491 parsl.dataflow.dflow DEBUG: Task 1491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/854cbe61-03ac-46d1-8e67-f13f43acefb2_isrForCrosstalkSources_855388_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/854cbe61-03ac-46d1-8e67-f13f43acefb2_isrForCrosstalkSources_855388_49.stderr parsl.dataflow.dflow INFO: Task 1534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27377deff25883a9a68b0ac3619b756a with result from task 1534 parsl.dataflow.dflow DEBUG: Task 1534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f6a9e2e1-6e97-4abc-a67a-1e4999155189_isrForCrosstalkSources_855390_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f6a9e2e1-6e97-4abc-a67a-1e4999155189_isrForCrosstalkSources_855390_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:04 proc_lsst.multi:146 [INFO] found job 16964081 in provider checkpoint proc_lsst.multi INFO: found job 16964081 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dca48a8bc4660199e86a0045d997b41 with result from task 1817 parsl.dataflow.dflow DEBUG: Task 1817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/91f3a5dd-8eb8-4e80-b534-f9e57e9aba78_isrForCrosstalkSources_855445_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/91f3a5dd-8eb8-4e80-b534-f9e57e9aba78_isrForCrosstalkSources_855445_4.stderr parsl.dataflow.dflow INFO: Task 1710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22de2e5db8a319d74ed1a0d4fb473474 with result from task 1710 parsl.dataflow.dflow DEBUG: Task 1710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ad4b6995-d4b4-428d-9dd6-c55dc2df1765_isrForCrosstalkSources_855413_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ad4b6995-d4b4-428d-9dd6-c55dc2df1765_isrForCrosstalkSources_855413_3.stderr parsl.dataflow.dflow INFO: Task 1513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b032c8e34703cdef20dbd307cacffc with result from task 1513 parsl.dataflow.dflow DEBUG: Task 1513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/86cc4c6b-b206-4e22-8b2d-a6a542e635ba_isrForCrosstalkSources_855436_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/86cc4c6b-b206-4e22-8b2d-a6a542e635ba_isrForCrosstalkSources_855436_50.stderr parsl.dataflow.dflow INFO: Task 1521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87a35b3f072966e91ff903b8d668ebf7 with result from task 1521 parsl.dataflow.dflow DEBUG: Task 1521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4eadf268-452f-46e1-8803-449c0c48703c_isrForCrosstalkSources_855415_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4eadf268-452f-46e1-8803-449c0c48703c_isrForCrosstalkSources_855415_24.stderr parsl.dataflow.dflow INFO: Task 1724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43b2aba553ab3fcdc74e1507d0c9417 with result from task 1724 parsl.dataflow.dflow DEBUG: Task 1724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0fc3e80e-4919-4573-ab63-62da76ea80bf_isrForCrosstalkSources_855428_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0fc3e80e-4919-4573-ab63-62da76ea80bf_isrForCrosstalkSources_855428_3.stderr parsl.dataflow.dflow INFO: Task 1494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2fabd7438f0502ded01d01c1956ab5d with result from task 1494 parsl.dataflow.dflow DEBUG: Task 1494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7ebec094-6549-41fe-a868-06eee5b01662_isrForCrosstalkSources_855402_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7ebec094-6549-41fe-a868-06eee5b01662_isrForCrosstalkSources_855402_47.stderr parsl.dataflow.dflow INFO: Task 1524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8127247d0ab217f8e03f66c1d846467f with result from task 1524 parsl.dataflow.dflow DEBUG: Task 1524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e356d3c7-f0e8-40e9-b540-3928ae359fdd_isrForCrosstalkSources_855380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e356d3c7-f0e8-40e9-b540-3928ae359fdd_isrForCrosstalkSources_855380_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:05 proc_lsst.multi:146 [INFO] found job 16964082 in provider checkpoint proc_lsst.multi INFO: found job 16964082 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d15e45134c7e9a05a54e8b523bc6ce5 with result from task 1872 parsl.dataflow.dflow DEBUG: Task 1872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ce112fcf-3710-4146-8b71-b200cf2b259a_isrForCrosstalkSources_855420_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ce112fcf-3710-4146-8b71-b200cf2b259a_isrForCrosstalkSources_855420_2.stderr parsl.dataflow.dflow INFO: Task 1806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8748a1f89a948edff0e58f8d54170f7d with result from task 1806 parsl.dataflow.dflow DEBUG: Task 1806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8b2e9ce9-1b94-4853-a818-9ae1069a6ba9_isrForCrosstalkSources_855412_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8b2e9ce9-1b94-4853-a818-9ae1069a6ba9_isrForCrosstalkSources_855412_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:05 proc_lsst.multi:146 [INFO] found job 16964083 in provider checkpoint proc_lsst.multi INFO: found job 16964083 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57492ed1336c851d1acb0009724f54dc with result from task 1707 parsl.dataflow.dflow DEBUG: Task 1707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a82eee6b-359e-409c-8c0a-60fb6de5a6e8_isrForCrosstalkSources_855451_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a82eee6b-359e-409c-8c0a-60fb6de5a6e8_isrForCrosstalkSources_855451_14.stderr parsl.dataflow.dflow INFO: Task 1435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cf89bf395d67208490e4e37c36c758e with result from task 1435 parsl.dataflow.dflow DEBUG: Task 1435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4a3d4e85-6951-475b-9879-b5d839a36a05_isrForCrosstalkSources_855393_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4a3d4e85-6951-475b-9879-b5d839a36a05_isrForCrosstalkSources_855393_30.stderr parsl.dataflow.dflow INFO: Task 1532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3888059c14cb77ae0a257215ad7bd8c with result from task 1532 parsl.dataflow.dflow DEBUG: Task 1532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e4c828c1-0390-4889-946b-87679d3c630b_isrForCrosstalkSources_855435_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e4c828c1-0390-4889-946b-87679d3c630b_isrForCrosstalkSources_855435_50.stderr parsl.dataflow.dflow INFO: Task 1848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc575c1b962a12a976bf181dbf351dd4 with result from task 1848 parsl.dataflow.dflow DEBUG: Task 1848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3ebf37ba-a791-4692-a2df-72423164fad4_isrForCrosstalkSources_855438_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3ebf37ba-a791-4692-a2df-72423164fad4_isrForCrosstalkSources_855438_24.stderr parsl.dataflow.dflow INFO: Task 1499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66fd5200dbeeb73929339ce8ab2913fa with result from task 1499 parsl.dataflow.dflow DEBUG: Task 1499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/29d8e73e-85d6-4ba8-bcdd-8cbbb768c93d_isrForCrosstalkSources_855411_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/29d8e73e-85d6-4ba8-bcdd-8cbbb768c93d_isrForCrosstalkSources_855411_53.stderr parsl.dataflow.dflow INFO: Task 1749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1232ce651721d8a82cb79cbc852e9f2 with result from task 1749 parsl.dataflow.dflow DEBUG: Task 1749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff7ff9b9-9eb8-4344-a3e4-d3c66abc92ac_isrForCrosstalkSources_855405_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff7ff9b9-9eb8-4344-a3e4-d3c66abc92ac_isrForCrosstalkSources_855405_32.stderr parsl.dataflow.dflow INFO: Task 1738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d1e2736dc404e9c4f58ea011e9e18f with result from task 1738 parsl.dataflow.dflow DEBUG: Task 1738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9809c97e-1063-4145-aa53-516b025ea25f_isrForCrosstalkSources_855386_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9809c97e-1063-4145-aa53-516b025ea25f_isrForCrosstalkSources_855386_1.stderr parsl.dataflow.dflow INFO: Task 1800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1de711683fe906b403ce5a68d30b60c with result from task 1800 parsl.dataflow.dflow DEBUG: Task 1800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1280ade4-4364-4f01-a61b-c1b9204f362d_isrForCrosstalkSources_855401_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1280ade4-4364-4f01-a61b-c1b9204f362d_isrForCrosstalkSources_855401_14.stderr parsl.dataflow.dflow INFO: Task 1486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9d0bfb5fc625fdd60eb73ddfe5401c2 with result from task 1486 parsl.dataflow.dflow DEBUG: Task 1486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/049e97b8-ca23-4afb-a482-088fd19fce66_isrForCrosstalkSources_855417_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/049e97b8-ca23-4afb-a482-088fd19fce66_isrForCrosstalkSources_855417_61.stderr parsl.dataflow.dflow INFO: Task 1386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c0a7b067595d6b952de31bfd4d5bc49 with result from task 1386 parsl.dataflow.dflow DEBUG: Task 1386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/23c391d2-930e-413b-a7e8-72f22a0799db_isrForCrosstalkSources_855409_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/23c391d2-930e-413b-a7e8-72f22a0799db_isrForCrosstalkSources_855409_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:05 proc_lsst.multi:146 [INFO] found job 16964084 in provider checkpoint proc_lsst.multi INFO: found job 16964084 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b68197d1d3c99c7678b981fbd0fbc895 with result from task 1501 parsl.dataflow.dflow DEBUG: Task 1501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/475b9a03-a092-43b6-9511-6d480d2c08c7_isrForCrosstalkSources_855389_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/475b9a03-a092-43b6-9511-6d480d2c08c7_isrForCrosstalkSources_855389_12.stderr parsl.dataflow.dflow INFO: Task 1839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 791e952e2ccbcad0f50da06b723f2d0b with result from task 1839 parsl.dataflow.dflow DEBUG: Task 1839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3328db37-cb2f-4abd-b283-c5ccad33d94c_isrForCrosstalkSources_855414_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3328db37-cb2f-4abd-b283-c5ccad33d94c_isrForCrosstalkSources_855414_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:06 proc_lsst.multi:146 [INFO] found job 16964085 in provider checkpoint proc_lsst.multi INFO: found job 16964085 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93f28dfe6cf5be0f7113ab6adef2d675 with result from task 1783 parsl.dataflow.dflow DEBUG: Task 1783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8c0431f-be2a-4055-81fb-76a22d77ab01_isrForCrosstalkSources_855450_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8c0431f-be2a-4055-81fb-76a22d77ab01_isrForCrosstalkSources_855450_23.stderr parsl.dataflow.dflow INFO: Task 1523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94ba7d72c265a9fb326c4ea1694efcec with result from task 1523 parsl.dataflow.dflow DEBUG: Task 1523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e96b8913-57c4-47fe-82d9-ad3bc1dee730_isrForCrosstalkSources_855413_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e96b8913-57c4-47fe-82d9-ad3bc1dee730_isrForCrosstalkSources_855413_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:06 proc_lsst.multi:146 [INFO] found job 16964086 in provider checkpoint proc_lsst.multi INFO: found job 16964086 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80a5b79bdc31890373cd6ed2a6a6fee7 with result from task 1460 parsl.dataflow.dflow DEBUG: Task 1460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c8864470-e990-4d27-93a4-b1e86e2d3c1f_isrForCrosstalkSources_855380_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c8864470-e990-4d27-93a4-b1e86e2d3c1f_isrForCrosstalkSources_855380_62.stderr parsl.dataflow.dflow INFO: Task 1528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 152dbe4752d6ccef72fc33a6a4f5572f with result from task 1528 parsl.dataflow.dflow DEBUG: Task 1528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7e8bb3f6-647c-49fc-885d-9f5cbaad5770_isrForCrosstalkSources_855441_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7e8bb3f6-647c-49fc-885d-9f5cbaad5770_isrForCrosstalkSources_855441_50.stderr parsl.dataflow.dflow INFO: Task 1739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc26f193ebdeb3e99764b281f4535598 with result from task 1739 parsl.dataflow.dflow DEBUG: Task 1739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1499d3cc-61d7-48f2-8f7d-a5cca5bae524_isrForCrosstalkSources_855403_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1499d3cc-61d7-48f2-8f7d-a5cca5bae524_isrForCrosstalkSources_855403_16.stderr parsl.dataflow.dflow INFO: Task 1731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575cb31d842c68baeb066edb7e3dbc7c with result from task 1731 parsl.dataflow.dflow DEBUG: Task 1731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/144b1b44-cb43-461b-a17a-ff07f5ec7bb5_isrForCrosstalkSources_855455_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/144b1b44-cb43-461b-a17a-ff07f5ec7bb5_isrForCrosstalkSources_855455_14.stderr parsl.dataflow.dflow INFO: Task 1784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24123168b639e599e06b91bd271bcae with result from task 1784 parsl.dataflow.dflow DEBUG: Task 1784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bdec0b49-bc7e-4976-a487-72edd78c6203_isrForCrosstalkSources_855380_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bdec0b49-bc7e-4976-a487-72edd78c6203_isrForCrosstalkSources_855380_37.stderr parsl.dataflow.dflow INFO: Task 1518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acfc4b90785080daecd112de36ecf4c4 with result from task 1518 parsl.dataflow.dflow DEBUG: Task 1518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d020cc23-cf07-4cb9-8114-95f9cebec142_isrForCrosstalkSources_855429_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d020cc23-cf07-4cb9-8114-95f9cebec142_isrForCrosstalkSources_855429_61.stderr parsl.dataflow.dflow INFO: Task 1409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74f3245c7d8d56576bba9a1abc132fcb with result from task 1409 parsl.dataflow.dflow DEBUG: Task 1409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4d47d873-2b85-410b-a79d-3ceee7871bec_isrForCrosstalkSources_855409_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4d47d873-2b85-410b-a79d-3ceee7871bec_isrForCrosstalkSources_855409_62.stderr parsl.dataflow.dflow INFO: Task 1437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a57ba0a6d2a825438d9710afe2ff8e1f with result from task 1437 parsl.dataflow.dflow DEBUG: Task 1437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cb745ba9-2227-4705-8172-4d0411986ca1_isrForCrosstalkSources_855387_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cb745ba9-2227-4705-8172-4d0411986ca1_isrForCrosstalkSources_855387_57.stderr parsl.dataflow.dflow INFO: Task 1274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa474cc6d1bb0f334b6e4dd43fff571 with result from task 1274 parsl.dataflow.dflow DEBUG: Task 1274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/958208e4-18bf-48ce-9422-bcbd95dd5a1f_isrForCrosstalkSources_855416_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/958208e4-18bf-48ce-9422-bcbd95dd5a1f_isrForCrosstalkSources_855416_32.stderr parsl.dataflow.dflow INFO: Task 1510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db1323b4e8ad7aee58f0fd0ab1b294a8 with result from task 1510 parsl.dataflow.dflow DEBUG: Task 1510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dd97e0fd-92e6-46ef-947d-2776c36f6f27_isrForCrosstalkSources_855434_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dd97e0fd-92e6-46ef-947d-2776c36f6f27_isrForCrosstalkSources_855434_38.stderr parsl.dataflow.dflow INFO: Task 1273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9cd1f02f05c11c315ad9c7e625b25ca with result from task 1273 parsl.dataflow.dflow DEBUG: Task 1273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/695162b8-afdd-4095-a445-282465669bef_isrForCrosstalkSources_855427_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/695162b8-afdd-4095-a445-282465669bef_isrForCrosstalkSources_855427_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:07 proc_lsst.multi:146 [INFO] found job 16964087 in provider checkpoint proc_lsst.multi INFO: found job 16964087 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ade4032a1fc83a7006b524431a1467d1 with result from task 1754 parsl.dataflow.dflow DEBUG: Task 1754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1d1b4804-5c6a-4f0a-8b71-7e53ed87b4de_isrForCrosstalkSources_855412_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1d1b4804-5c6a-4f0a-8b71-7e53ed87b4de_isrForCrosstalkSources_855412_8.stderr parsl.dataflow.dflow INFO: Task 1771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4a0ec8b98755e1e101913b0d63cd97 with result from task 1771 parsl.dataflow.dflow DEBUG: Task 1771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f8fe032a-650e-40e2-bb9b-7f1907a5a649_isrForCrosstalkSources_855384_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f8fe032a-650e-40e2-bb9b-7f1907a5a649_isrForCrosstalkSources_855384_27.stderr parsl.dataflow.dflow INFO: Task 1728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8edbcfabcd120d2061d576372bb67e0 with result from task 1728 parsl.dataflow.dflow DEBUG: Task 1728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0d91cee1-17f8-4fdc-9b87-06414ace2d88_isrForCrosstalkSources_855404_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0d91cee1-17f8-4fdc-9b87-06414ace2d88_isrForCrosstalkSources_855404_22.stderr parsl.dataflow.dflow INFO: Task 1892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 895a5436c4b80cbf31ccb7eeb09b6f4a with result from task 1892 parsl.dataflow.dflow DEBUG: Task 1892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7dc4e89e-f1c0-484c-bc75-9e65d669ee15_isrForCrosstalkSources_855459_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7dc4e89e-f1c0-484c-bc75-9e65d669ee15_isrForCrosstalkSources_855459_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:07 proc_lsst.multi:146 [INFO] found job 16964088 in provider checkpoint proc_lsst.multi INFO: found job 16964088 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22e963d09f9c5ecacbfbbea8000cf75d with result from task 1810 parsl.dataflow.dflow DEBUG: Task 1810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ebe85b08-4d71-4f19-9d37-3eefb655b582_isrForCrosstalkSources_855437_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ebe85b08-4d71-4f19-9d37-3eefb655b582_isrForCrosstalkSources_855437_17.stderr parsl.dataflow.dflow INFO: Task 1427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 871f8e0127dd7dfbdd73093913cfc848 with result from task 1427 parsl.dataflow.dflow DEBUG: Task 1427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e8d18879-482a-4811-81f2-c3facf94e3f4_isrForCrosstalkSources_855380_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e8d18879-482a-4811-81f2-c3facf94e3f4_isrForCrosstalkSources_855380_29.stderr parsl.dataflow.dflow INFO: Task 1873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b604e4e7582619e2f1f7d43e49e9dce with result from task 1873 parsl.dataflow.dflow DEBUG: Task 1873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/cda88035-adac-491a-90e4-65d700d3e6fd_isrForCrosstalkSources_855419_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/cda88035-adac-491a-90e4-65d700d3e6fd_isrForCrosstalkSources_855419_34.stderr parsl.dataflow.dflow INFO: Task 1329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df43278fcb476eac4f2ba6ebaffbb103 with result from task 1329 parsl.dataflow.dflow DEBUG: Task 1329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ec5a2743-9a3b-4a3d-b22d-afe7352b502a_isrForCrosstalkSources_855439_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ec5a2743-9a3b-4a3d-b22d-afe7352b502a_isrForCrosstalkSources_855439_47.stderr parsl.dataflow.dflow INFO: Task 1774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd85dddd7ff0aa45d667f88d5064c6b4 with result from task 1774 parsl.dataflow.dflow DEBUG: Task 1774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f1999fdb-0a39-4fea-8dc0-4385003f2e1a_isrForCrosstalkSources_855420_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f1999fdb-0a39-4fea-8dc0-4385003f2e1a_isrForCrosstalkSources_855420_56.stderr parsl.dataflow.dflow INFO: Task 1727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8019578db3d1f88379761b12209b3b23 with result from task 1727 parsl.dataflow.dflow DEBUG: Task 1727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4fd1b57c-3bd4-4282-b49b-d2b225ec9846_isrForCrosstalkSources_855419_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/4fd1b57c-3bd4-4282-b49b-d2b225ec9846_isrForCrosstalkSources_855419_2.stderr parsl.dataflow.dflow INFO: Task 1732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e25eb3edd295de0bdf507df79ea4fd3 with result from task 1732 parsl.dataflow.dflow DEBUG: Task 1732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/55e245dd-3dab-4c1d-96f9-43e24b6ccb9e_isrForCrosstalkSources_855401_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/55e245dd-3dab-4c1d-96f9-43e24b6ccb9e_isrForCrosstalkSources_855401_45.stderr parsl.dataflow.dflow INFO: Task 1285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c821fdbffe62967bc9eba051fb9bac7c with result from task 1285 parsl.dataflow.dflow DEBUG: Task 1285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0efc5ed8-72a6-4e6d-a16c-f44afebf2d99_isrForCrosstalkSources_855457_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0efc5ed8-72a6-4e6d-a16c-f44afebf2d99_isrForCrosstalkSources_855457_45.stderr parsl.dataflow.dflow INFO: Task 1893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad5f855aeda9fd61c56b5fc304c9e566 with result from task 1893 parsl.dataflow.dflow DEBUG: Task 1893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1ea5598-d71a-4c02-9c41-6a4c1480d065_isrForCrosstalkSources_855412_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1ea5598-d71a-4c02-9c41-6a4c1480d065_isrForCrosstalkSources_855412_45.stderr parsl.dataflow.dflow INFO: Task 1788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db400145be06612166cf273a77749c3 with result from task 1788 parsl.dataflow.dflow DEBUG: Task 1788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ee4a4fcc-e5b2-40f9-8ddf-30b8046925e9_isrForCrosstalkSources_855413_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ee4a4fcc-e5b2-40f9-8ddf-30b8046925e9_isrForCrosstalkSources_855413_56.stderr parsl.dataflow.dflow INFO: Task 1787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a89c12396d21f4ada22d6d597b5a527a with result from task 1787 parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:08 proc_lsst.multi:146 [INFO] found job 16964089 in provider checkpoint proc_lsst.multi INFO: found job 16964089 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 1787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/74805213-a793-4f2e-a764-33ca8a8eb45d_isrForCrosstalkSources_855440_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/74805213-a793-4f2e-a764-33ca8a8eb45d_isrForCrosstalkSources_855440_17.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 690870ac5d3e534f6055e94f54569b7b with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/38f8e3d2-5c23-4b3a-8571-2f8128e32947_isrForCrosstalkSources_855395_43.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/38f8e3d2-5c23-4b3a-8571-2f8128e32947_isrForCrosstalkSources_855395_43.stderr parsl.dataflow.dflow INFO: Task 1681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62ab0fc04b046cd08763ed793d52af19 with result from task 1681 parsl.dataflow.dflow DEBUG: Task 1681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e7a0546c-9acf-4f98-8d9c-12a36781ae6d_isrForCrosstalkSources_855411_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e7a0546c-9acf-4f98-8d9c-12a36781ae6d_isrForCrosstalkSources_855411_3.stderr parsl.dataflow.dflow INFO: Task 1777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f820705ca714b77c204ccc74b457e63 with result from task 1777 parsl.dataflow.dflow DEBUG: Task 1777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/4632e42e-ef8e-43cf-95e1-1929ab140f1e_isrForCrosstalkSources_855444_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/4632e42e-ef8e-43cf-95e1-1929ab140f1e_isrForCrosstalkSources_855444_45.stderr parsl.dataflow.dflow INFO: Task 1797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fed02f95bd328a85487088d203d5f209 with result from task 1797 parsl.dataflow.dflow DEBUG: Task 1797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35551444-7132-4726-8388-8e6b637ef4f6_isrForCrosstalkSources_855409_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35551444-7132-4726-8388-8e6b637ef4f6_isrForCrosstalkSources_855409_51.stderr parsl.dataflow.dflow INFO: Task 1432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4578bdeafc277f495bbdf8490cd734a4 with result from task 1432 parsl.dataflow.dflow DEBUG: Task 1432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/75765d28-45bb-4f75-989e-b660e83ba923_isrForCrosstalkSources_855436_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/75765d28-45bb-4f75-989e-b660e83ba923_isrForCrosstalkSources_855436_49.stderr parsl.dataflow.dflow INFO: Task 1826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b334c5a9d5a7a856978c7ac14313b39d with result from task 1826 parsl.dataflow.dflow DEBUG: Task 1826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8e29da3c-dc1d-4dc1-81f9-f8be5273c0b3_isrForCrosstalkSources_855419_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8e29da3c-dc1d-4dc1-81f9-f8be5273c0b3_isrForCrosstalkSources_855419_26.stderr parsl.dataflow.dflow INFO: Task 1819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf22881e9fe67efd46d1570dbe35c134 with result from task 1819 parsl.dataflow.dflow DEBUG: Task 1819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/efb934a0-2c15-4d4e-ad16-c013b5aec410_isrForCrosstalkSources_855385_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/efb934a0-2c15-4d4e-ad16-c013b5aec410_isrForCrosstalkSources_855385_26.stderr parsl.dataflow.dflow INFO: Task 1484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9dbb8b209577d0bbe0e6faaecf3ac67 with result from task 1484 parsl.dataflow.dflow DEBUG: Task 1484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/07fabea5-0463-421b-b46c-8a34c5b99546_isrForCrosstalkSources_855440_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/07fabea5-0463-421b-b46c-8a34c5b99546_isrForCrosstalkSources_855440_11.stderr parsl.dataflow.dflow INFO: Task 1836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09358479c79134ae55c792579d152595 with result from task 1836 parsl.dataflow.dflow DEBUG: Task 1836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/25b6cc8b-4c29-4bc9-afc8-efd1b95650d5_isrForCrosstalkSources_855406_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/25b6cc8b-4c29-4bc9-afc8-efd1b95650d5_isrForCrosstalkSources_855406_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:09 proc_lsst.multi:146 [INFO] found job 16964090 in provider checkpoint proc_lsst.multi INFO: found job 16964090 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2adbf9403201100f37e7be0a0694d61 with result from task 1798 parsl.dataflow.dflow DEBUG: Task 1798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c996ab30-2e5b-4d19-ab8a-a308b07bc538_isrForCrosstalkSources_855411_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c996ab30-2e5b-4d19-ab8a-a308b07bc538_isrForCrosstalkSources_855411_42.stderr parsl.dataflow.dflow INFO: Task 1856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd7d5d2f71a09ee1e636752931000062 with result from task 1856 parsl.dataflow.dflow DEBUG: Task 1856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c76999f4-f934-47b9-908d-21cb091d71a4_isrForCrosstalkSources_855383_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c76999f4-f934-47b9-908d-21cb091d71a4_isrForCrosstalkSources_855383_8.stderr parsl.dataflow.dflow INFO: Task 1864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aabe8cc14b12445dc99746888940dfe0 with result from task 1864 parsl.dataflow.dflow DEBUG: Task 1864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/778195b8-f715-48b5-8719-32a2d1e5ea14_isrForCrosstalkSources_855381_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/778195b8-f715-48b5-8719-32a2d1e5ea14_isrForCrosstalkSources_855381_27.stderr parsl.dataflow.dflow INFO: Task 1808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3b79fc78fc80a41ad3c56ab4a0c3097 with result from task 1808 parsl.dataflow.dflow DEBUG: Task 1808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/71ca51bd-d941-4ac0-9aa2-a074f722d84f_isrForCrosstalkSources_855416_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/71ca51bd-d941-4ac0-9aa2-a074f722d84f_isrForCrosstalkSources_855416_25.stderr parsl.dataflow.dflow INFO: Task 1711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa53e2e584d21963e010301d01e03110 with result from task 1711 parsl.dataflow.dflow DEBUG: Task 1711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/85555475-f12c-40e5-a62a-5da147f424b4_isrForCrosstalkSources_855403_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/85555475-f12c-40e5-a62a-5da147f424b4_isrForCrosstalkSources_855403_2.stderr parsl.dataflow.dflow INFO: Task 1535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b9a15fd186537bc10e7e37b371a9fcd with result from task 1535 parsl.dataflow.dflow DEBUG: Task 1535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/140d19c5-8693-4ad5-9786-2fc260335100_isrForCrosstalkSources_855381_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/140d19c5-8693-4ad5-9786-2fc260335100_isrForCrosstalkSources_855381_35.stderr parsl.dataflow.dflow INFO: Task 1863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48be156b83bd45b91b54034a3b25d46f with result from task 1863 parsl.dataflow.dflow DEBUG: Task 1863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01e0e521-6a2e-4d15-baed-f8745149c72c_isrForCrosstalkSources_855433_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01e0e521-6a2e-4d15-baed-f8745149c72c_isrForCrosstalkSources_855433_19.stderr parsl.dataflow.dflow INFO: Task 1888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac507c8be67f3bc88e18c808e9be762 with result from task 1888 parsl.dataflow.dflow DEBUG: Task 1888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c142c38e-b5ed-4c47-adcc-717fb24ece10_isrForCrosstalkSources_855444_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c142c38e-b5ed-4c47-adcc-717fb24ece10_isrForCrosstalkSources_855444_5.stderr parsl.dataflow.dflow INFO: Task 1744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87b9a8687a45478150d289c69013de49 with result from task 1744 parsl.dataflow.dflow DEBUG: Task 1744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4ba63552-2ff3-4253-b5d4-0b1eaf8c1a4b_isrForCrosstalkSources_855408_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/4ba63552-2ff3-4253-b5d4-0b1eaf8c1a4b_isrForCrosstalkSources_855408_28.stderr parsl.dataflow.dflow INFO: Task 1709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1345af2422f943e4fe667bfbc5aaea73 with result from task 1709 parsl.dataflow.dflow DEBUG: Task 1709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a8aa44a1-8191-4665-b6b9-48f11cddf8a4_isrForCrosstalkSources_855432_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a8aa44a1-8191-4665-b6b9-48f11cddf8a4_isrForCrosstalkSources_855432_48.stderr parsl.dataflow.dflow INFO: Task 1824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b8c4c27fbde415f87abc52265d5fb3b with result from task 1824 parsl.dataflow.dflow DEBUG: Task 1824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7aef786a-7407-4f08-be65-e1dc733eb679_isrForCrosstalkSources_855380_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7aef786a-7407-4f08-be65-e1dc733eb679_isrForCrosstalkSources_855380_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:09 proc_lsst.multi:146 [INFO] found job 16964091 in provider checkpoint proc_lsst.multi INFO: found job 16964091 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85824f6aa7d1e7e9fa5359b1eaa272db with result from task 2011 parsl.dataflow.dflow DEBUG: Task 2011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/a1f6b7ac-4d22-46c5-b4a3-82267bd67d52_isrForCrosstalkSources_855392_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/a1f6b7ac-4d22-46c5-b4a3-82267bd67d52_isrForCrosstalkSources_855392_16.stderr parsl.dataflow.dflow INFO: Task 1758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8264ab09e7a91077ca2b9c2529f37423 with result from task 1758 parsl.dataflow.dflow DEBUG: Task 1758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fede1ab1-f75a-489c-8b82-00433584681e_isrForCrosstalkSources_855423_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fede1ab1-f75a-489c-8b82-00433584681e_isrForCrosstalkSources_855423_46.stderr parsl.dataflow.dflow INFO: Task 1796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f608de7ee66ec9f77d3808357144f727 with result from task 1796 parsl.dataflow.dflow DEBUG: Task 1796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b8ba462a-29b9-49a0-bc73-4ff7f90760f9_isrForCrosstalkSources_855439_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b8ba462a-29b9-49a0-bc73-4ff7f90760f9_isrForCrosstalkSources_855439_4.stderr parsl.dataflow.dflow INFO: Task 1880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d19061b948654a47897bfe64cb4ac85e with result from task 1880 parsl.dataflow.dflow DEBUG: Task 1880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3364ee19-d665-477d-9b8f-13f17ab1745a_isrForCrosstalkSources_855390_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3364ee19-d665-477d-9b8f-13f17ab1745a_isrForCrosstalkSources_855390_25.stderr parsl.dataflow.dflow INFO: Task 1868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c0d08809080a0105405d8233d3ce74 with result from task 1868 parsl.dataflow.dflow DEBUG: Task 1868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/547e5cd9-3e7f-458d-af71-cc8eb0438374_isrForCrosstalkSources_855399_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/547e5cd9-3e7f-458d-af71-cc8eb0438374_isrForCrosstalkSources_855399_39.stderr parsl.dataflow.dflow INFO: Task 1517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f04649c4079bb27079f1c013a4ccea8 with result from task 1517 parsl.dataflow.dflow DEBUG: Task 1517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1d3ec596-8447-40bc-b1da-540da8e4214b_isrForCrosstalkSources_855395_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1d3ec596-8447-40bc-b1da-540da8e4214b_isrForCrosstalkSources_855395_44.stderr parsl.dataflow.dflow INFO: Task 1853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e866d90337b7899ced2c035575d5ae3b with result from task 1853 parsl.dataflow.dflow DEBUG: Task 1853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5a159d82-15d3-4720-9f03-f01d7e457671_isrForCrosstalkSources_855430_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5a159d82-15d3-4720-9f03-f01d7e457671_isrForCrosstalkSources_855430_39.stderr parsl.dataflow.dflow INFO: Task 1762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cba04fd236d7d92adee1cb84caa23a09 with result from task 1762 parsl.dataflow.dflow DEBUG: Task 1762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/87728572-59e9-4264-8077-055a8ac01a1d_isrForCrosstalkSources_855450_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/87728572-59e9-4264-8077-055a8ac01a1d_isrForCrosstalkSources_855450_21.stderr parsl.dataflow.dflow INFO: Task 1851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db93201bdfcd3e648ce91433a850d89 with result from task 1851 parsl.dataflow.dflow DEBUG: Task 1851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3285667d-f96b-4289-aa70-8785b3699b2e_isrForCrosstalkSources_855394_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3285667d-f96b-4289-aa70-8785b3699b2e_isrForCrosstalkSources_855394_39.stderr parsl.dataflow.dflow INFO: Task 1507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c78ce8d1c3b94a48962b2f0f30f99307 with result from task 1507 parsl.dataflow.dflow DEBUG: Task 1507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/08302ac3-3304-4dee-bf6c-9c66e39f3748_isrForCrosstalkSources_855396_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/08302ac3-3304-4dee-bf6c-9c66e39f3748_isrForCrosstalkSources_855396_52.stderr parsl.dataflow.dflow INFO: Task 1773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3a7276d7d3c2b37769b5e920e15a8ec with result from task 1773 parsl.dataflow.dflow DEBUG: Task 1773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ec85978b-4bd0-4443-b869-870f14fc0b49_isrForCrosstalkSources_855381_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ec85978b-4bd0-4443-b869-870f14fc0b49_isrForCrosstalkSources_855381_24.stderr parsl.dataflow.dflow INFO: Task 1530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f269f8cd08cc6dbfbca5a44418367493 with result from task 1530 parsl.dataflow.dflow DEBUG: Task 1530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a7ea0167-22d9-494a-ab33-b907b0655d07_isrForCrosstalkSources_855422_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a7ea0167-22d9-494a-ab33-b907b0655d07_isrForCrosstalkSources_855422_17.stderr parsl.dataflow.dflow INFO: Task 1478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 234c3ec67ee49ec0d488b97fe73b50de with result from task 1478 parsl.dataflow.dflow DEBUG: Task 1478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/518b379a-199e-4da3-b02f-c9fc3df738c4_isrForCrosstalkSources_855432_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/518b379a-199e-4da3-b02f-c9fc3df738c4_isrForCrosstalkSources_855432_35.stderr parsl.dataflow.dflow INFO: Task 1765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 296626acd8c785cd786f1f7cf17dd98f with result from task 1765 parsl.dataflow.dflow DEBUG: Task 1765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/fc08558a-4b03-4d8f-bd46-c9e99944c0b4_isrForCrosstalkSources_855421_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/fc08558a-4b03-4d8f-bd46-c9e99944c0b4_isrForCrosstalkSources_855421_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:10 proc_lsst.multi:146 [INFO] found job 16964092 in provider checkpoint proc_lsst.multi INFO: found job 16964092 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e8a7afbe9f10876b41c65c6b1bc90c1 with result from task 1444 parsl.dataflow.dflow DEBUG: Task 1444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1214e390-964b-4c48-8221-44f8dbba4300_isrForCrosstalkSources_855427_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1214e390-964b-4c48-8221-44f8dbba4300_isrForCrosstalkSources_855427_29.stderr parsl.dataflow.dflow INFO: Task 1790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d909a259b0bbd2989a423e5047e3e672 with result from task 1790 parsl.dataflow.dflow DEBUG: Task 1790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/de1cea4a-7554-4141-b90f-6871f35ea3b8_isrForCrosstalkSources_855454_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/de1cea4a-7554-4141-b90f-6871f35ea3b8_isrForCrosstalkSources_855454_18.stderr parsl.dataflow.dflow INFO: Task 1818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61604737dca0ee2e7c4ec5d8c81160ad with result from task 1818 parsl.dataflow.dflow DEBUG: Task 1818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/589ece04-d72b-4ceb-b30f-df4dfd59c741_isrForCrosstalkSources_855393_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/589ece04-d72b-4ceb-b30f-df4dfd59c741_isrForCrosstalkSources_855393_21.stderr parsl.dataflow.dflow INFO: Task 1821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57d631cad749bf98d1e63febcf2e897b with result from task 1821 parsl.dataflow.dflow DEBUG: Task 1821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c88baf85-84a2-4b6a-a134-560fa0d35345_isrForCrosstalkSources_855440_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c88baf85-84a2-4b6a-a134-560fa0d35345_isrForCrosstalkSources_855440_39.stderr parsl.dataflow.dflow INFO: Task 1493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4913b0e228b33bc1a17039202fc244f8 with result from task 1493 parsl.dataflow.dflow DEBUG: Task 1493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8b73a08d-622d-48f0-b0f9-befa7ee6dec9_isrForCrosstalkSources_855441_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8b73a08d-622d-48f0-b0f9-befa7ee6dec9_isrForCrosstalkSources_855441_40.stderr parsl.dataflow.dflow INFO: Task 1865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2736ea17eee523e83edc92fb25d85330 with result from task 1865 parsl.dataflow.dflow DEBUG: Task 1865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/eeba60ba-9db8-4fd6-abf8-4337697430f4_isrForCrosstalkSources_855444_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/eeba60ba-9db8-4fd6-abf8-4337697430f4_isrForCrosstalkSources_855444_22.stderr parsl.dataflow.dflow INFO: Task 1814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 254bdbd1b658a440cfb6b1db0e44665b with result from task 1814 parsl.dataflow.dflow DEBUG: Task 1814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/68ceb3ca-e88e-4f24-aa0c-e8e5fc0f3551_isrForCrosstalkSources_855421_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/68ceb3ca-e88e-4f24-aa0c-e8e5fc0f3551_isrForCrosstalkSources_855421_51.stderr parsl.dataflow.dflow INFO: Task 1980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d7e58c2351166decd62fcad48f05c9c with result from task 1980 parsl.dataflow.dflow DEBUG: Task 1980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a8ae788d-5e07-46b6-88e9-b8ca659efcec_isrForCrosstalkSources_855442_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a8ae788d-5e07-46b6-88e9-b8ca659efcec_isrForCrosstalkSources_855442_25.stderr parsl.dataflow.dflow INFO: Task 1949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac02a026bf13fca8aa6482639db6951a with result from task 1949 parsl.dataflow.dflow DEBUG: Task 1949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6e5ce65d-65ba-4015-8e3f-9a4392fdb14d_isrForCrosstalkSources_855436_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6e5ce65d-65ba-4015-8e3f-9a4392fdb14d_isrForCrosstalkSources_855436_29.stderr parsl.dataflow.dflow INFO: Task 1901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b941fcc554f9b3d9080aed599cea4d8 with result from task 1901 parsl.dataflow.dflow DEBUG: Task 1901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/d71e285e-cd8a-4f2b-a447-7900e9343749_isrForCrosstalkSources_855394_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/d71e285e-cd8a-4f2b-a447-7900e9343749_isrForCrosstalkSources_855394_4.stderr parsl.dataflow.dflow INFO: Task 1483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9111dcef67c205a974cc538ae2400f with result from task 1483 parsl.dataflow.dflow DEBUG: Task 1483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/523bc2b1-f657-41f8-8f6a-90ddddc51644_isrForCrosstalkSources_855418_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/523bc2b1-f657-41f8-8f6a-90ddddc51644_isrForCrosstalkSources_855418_31.stderr parsl.dataflow.dflow INFO: Task 1841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffa960a65c325a637a62036131515bed with result from task 1841 parsl.dataflow.dflow DEBUG: Task 1841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f67c4abd-837f-449e-b47c-b088d7900a15_isrForCrosstalkSources_855396_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f67c4abd-837f-449e-b47c-b088d7900a15_isrForCrosstalkSources_855396_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:10 proc_lsst.multi:146 [INFO] found job 16964093 in provider checkpoint proc_lsst.multi INFO: found job 16964093 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c75523e7b0021163c7d0bdf2fbb644c4 with result from task 1757 parsl.dataflow.dflow DEBUG: Task 1757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cedd5363-ccf0-40c9-abf4-624ef9bba6bf_isrForCrosstalkSources_855415_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cedd5363-ccf0-40c9-abf4-624ef9bba6bf_isrForCrosstalkSources_855415_29.stderr parsl.dataflow.dflow INFO: Task 1827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9653abdb8cb67619a5b1a608ddacd78 with result from task 1827 parsl.dataflow.dflow DEBUG: Task 1827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/74dd5604-0701-465d-8783-f36f7541fbd9_isrForCrosstalkSources_855412_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/74dd5604-0701-465d-8783-f36f7541fbd9_isrForCrosstalkSources_855412_13.stderr parsl.dataflow.dflow INFO: Task 1945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2acb04aee918732028fcbfd7a07b62da with result from task 1945 parsl.dataflow.dflow DEBUG: Task 1945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36fd4de0-7ad6-400a-8f59-430a8ee2b1a2_isrForCrosstalkSources_855390_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36fd4de0-7ad6-400a-8f59-430a8ee2b1a2_isrForCrosstalkSources_855390_15.stderr parsl.dataflow.dflow INFO: Task 1476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b9b84c8bca63b82a594c7cca1a33698 with result from task 1476 parsl.dataflow.dflow DEBUG: Task 1476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3bc14a05-08c0-417d-bbf9-791a2c5019f5_isrForCrosstalkSources_855413_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3bc14a05-08c0-417d-bbf9-791a2c5019f5_isrForCrosstalkSources_855413_53.stderr parsl.dataflow.dflow INFO: Task 1820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4c2592dc1897ed3a51f17c246adb47b with result from task 1820 parsl.dataflow.dflow DEBUG: Task 1820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca57bf08-1443-456d-86b5-015ff17c8329_isrForCrosstalkSources_855437_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca57bf08-1443-456d-86b5-015ff17c8329_isrForCrosstalkSources_855437_21.stderr parsl.dataflow.dflow INFO: Task 1270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca6c2507c844e0bd147de17e059f30b with result from task 1270 parsl.dataflow.dflow DEBUG: Task 1270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b512c51b-8a1a-47f2-bd8d-180980fde0ee_isrForCrosstalkSources_855416_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/b512c51b-8a1a-47f2-bd8d-180980fde0ee_isrForCrosstalkSources_855416_18.stderr parsl.dataflow.dflow INFO: Task 1942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01bcf8f90e86cf92956f173f913d19fd with result from task 1942 parsl.dataflow.dflow DEBUG: Task 1942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b8ebbaeb-c48f-4d72-953c-9f522be6c62a_isrForCrosstalkSources_855432_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b8ebbaeb-c48f-4d72-953c-9f522be6c62a_isrForCrosstalkSources_855432_25.stderr parsl.dataflow.dflow INFO: Task 1781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ecdcef6972c0e14b8d48222dcc033f with result from task 1781 parsl.dataflow.dflow DEBUG: Task 1781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/857ebb41-15af-4d7c-9a47-6eae3a6941ca_isrForCrosstalkSources_855426_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/857ebb41-15af-4d7c-9a47-6eae3a6941ca_isrForCrosstalkSources_855426_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:11 proc_lsst.multi:146 [INFO] found job 16964094 in provider checkpoint proc_lsst.multi INFO: found job 16964094 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86871559c7ceae7510eec203a8ddb47f with result from task 1815 parsl.dataflow.dflow DEBUG: Task 1815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f1b8b52b-04e1-4221-a6ea-3034fafb1d53_isrForCrosstalkSources_855392_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f1b8b52b-04e1-4221-a6ea-3034fafb1d53_isrForCrosstalkSources_855392_5.stderr parsl.dataflow.dflow INFO: Task 1287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6f65bfa1ac189f07091096e1781535 with result from task 1287 parsl.dataflow.dflow DEBUG: Task 1287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4f1965fc-13be-4b7d-94a7-e331a31fc51c_isrForCrosstalkSources_855413_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4f1965fc-13be-4b7d-94a7-e331a31fc51c_isrForCrosstalkSources_855413_17.stderr parsl.dataflow.dflow INFO: Task 1282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ddc8a2631a8b691f4e30b8049d5a166 with result from task 1282 parsl.dataflow.dflow DEBUG: Task 1282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29333641-4b6d-4a47-882a-b308347aceba_isrForCrosstalkSources_855447_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/29333641-4b6d-4a47-882a-b308347aceba_isrForCrosstalkSources_855447_7.stderr parsl.dataflow.dflow INFO: Task 1708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef9219459a419cf4d041a0127923fbb with result from task 1708 parsl.dataflow.dflow DEBUG: Task 1708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d5366984-807a-46ad-8917-dca4162df657_isrForCrosstalkSources_855416_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d5366984-807a-46ad-8917-dca4162df657_isrForCrosstalkSources_855416_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:11 proc_lsst.multi:146 [INFO] found job 16964095 in provider checkpoint proc_lsst.multi INFO: found job 16964095 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e343244553517bfca22420e2c086b7db with result from task 1829 parsl.dataflow.dflow DEBUG: Task 1829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d9cd768d-5835-40a5-ae04-81af62f3a986_isrForCrosstalkSources_855436_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d9cd768d-5835-40a5-ae04-81af62f3a986_isrForCrosstalkSources_855436_51.stderr parsl.dataflow.dflow INFO: Task 1986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c069223f213b2228b332bb34cbf5f1e with result from task 1986 parsl.dataflow.dflow DEBUG: Task 1986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/63b8c2de-f8ab-4d97-b9fb-5737f5c54617_isrForCrosstalkSources_855383_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/63b8c2de-f8ab-4d97-b9fb-5737f5c54617_isrForCrosstalkSources_855383_51.stderr parsl.dataflow.dflow INFO: Task 2009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4be3fc914e136c39a76c71aa55deeca8 with result from task 2009 parsl.dataflow.dflow DEBUG: Task 2009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9a0614b-e370-4588-a200-ceee279c072e_isrForCrosstalkSources_855421_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9a0614b-e370-4588-a200-ceee279c072e_isrForCrosstalkSources_855421_4.stderr parsl.dataflow.dflow INFO: Task 1828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 488b7867c15a760e118d9de6b163add5 with result from task 1828 parsl.dataflow.dflow DEBUG: Task 1828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d9716ee5-e52a-4aac-99d6-1fbf48ac594b_isrForCrosstalkSources_855386_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d9716ee5-e52a-4aac-99d6-1fbf48ac594b_isrForCrosstalkSources_855386_35.stderr parsl.dataflow.dflow INFO: Task 1729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab09c5d266f056b6e92ebe6f8aaf3fe6 with result from task 1729 parsl.dataflow.dflow DEBUG: Task 1729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/57e2041e-1f88-47ad-9f4f-42d546ffeefd_isrForCrosstalkSources_855385_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/57e2041e-1f88-47ad-9f4f-42d546ffeefd_isrForCrosstalkSources_855385_38.stderr parsl.dataflow.dflow INFO: Task 1737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 203762c33a965a96e1398f87375df63f with result from task 1737 parsl.dataflow.dflow DEBUG: Task 1737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fe4f1090-1ad8-48da-8c78-f86d8d6ee59c_isrForCrosstalkSources_855459_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fe4f1090-1ad8-48da-8c78-f86d8d6ee59c_isrForCrosstalkSources_855459_10.stderr parsl.dataflow.dflow INFO: Task 1941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9336087f25e308e460e944ce94ed1bbf with result from task 1941 parsl.dataflow.dflow DEBUG: Task 1941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/72aa123a-2599-47f1-82ca-2a725d82f23f_isrForCrosstalkSources_855393_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/72aa123a-2599-47f1-82ca-2a725d82f23f_isrForCrosstalkSources_855393_23.stderr parsl.dataflow.dflow INFO: Task 1275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78e6a638db57d044188892d4c3171095 with result from task 1275 parsl.dataflow.dflow DEBUG: Task 1275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2f3fd13a-428a-4afa-8ef9-4c353ae7c8db_isrForCrosstalkSources_855411_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2f3fd13a-428a-4afa-8ef9-4c353ae7c8db_isrForCrosstalkSources_855411_17.stderr parsl.dataflow.dflow INFO: Task 1812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3f6f5080d59c2df076e5de3945d5de with result from task 1812 parsl.dataflow.dflow DEBUG: Task 1812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/bb0c4263-fd06-4d4c-b2c1-4988469a43fa_isrForCrosstalkSources_855455_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/bb0c4263-fd06-4d4c-b2c1-4988469a43fa_isrForCrosstalkSources_855455_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:12 proc_lsst.multi:146 [INFO] found job 16964096 in provider checkpoint proc_lsst.multi INFO: found job 16964096 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbbc7f409dbb559a2390ef54b406020f with result from task 1891 parsl.dataflow.dflow DEBUG: Task 1891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/fe542dbc-2f8f-482e-ac0f-8d51f159eee0_isrForCrosstalkSources_855410_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/fe542dbc-2f8f-482e-ac0f-8d51f159eee0_isrForCrosstalkSources_855410_22.stderr parsl.dataflow.dflow INFO: Task 1855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b474af37a9b2728b292782d9371dd45b with result from task 1855 parsl.dataflow.dflow DEBUG: Task 1855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6fe1e1f3-87f5-4c01-8b48-535de91ccf6f_isrForCrosstalkSources_855421_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6fe1e1f3-87f5-4c01-8b48-535de91ccf6f_isrForCrosstalkSources_855421_3.stderr parsl.dataflow.dflow INFO: Task 1833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cc6f4262004823a045dce2a40681dde with result from task 1833 parsl.dataflow.dflow DEBUG: Task 1833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/71b1b6bf-ef91-4728-b133-8e84bd2e75b9_isrForCrosstalkSources_855394_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/71b1b6bf-ef91-4728-b133-8e84bd2e75b9_isrForCrosstalkSources_855394_8.stderr parsl.dataflow.dflow INFO: Task 1794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfb06a47a8ed17c9eb96f2ad91b785ae with result from task 1794 parsl.dataflow.dflow DEBUG: Task 1794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cc335d55-7238-441b-a06a-6fd266e70d06_isrForCrosstalkSources_855449_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cc335d55-7238-441b-a06a-6fd266e70d06_isrForCrosstalkSources_855449_40.stderr parsl.dataflow.dflow INFO: Task 1776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eef6bed81031b173d739a5d91bcd2e38 with result from task 1776 parsl.dataflow.dflow DEBUG: Task 1776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b714a4d3-c007-4170-a2d3-654111aa21fa_isrForCrosstalkSources_855454_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b714a4d3-c007-4170-a2d3-654111aa21fa_isrForCrosstalkSources_855454_12.stderr parsl.dataflow.dflow INFO: Task 2003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48908e2de20b2a144e02c224f00c15bb with result from task 2003 parsl.dataflow.dflow DEBUG: Task 2003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f9db51d2-3dcd-42f0-912a-27c2a5a39b9a_isrForCrosstalkSources_855395_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f9db51d2-3dcd-42f0-912a-27c2a5a39b9a_isrForCrosstalkSources_855395_56.stderr parsl.dataflow.dflow INFO: Task 1882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd156d898a43751ff672401c43ba55a9 with result from task 1882 parsl.dataflow.dflow DEBUG: Task 1882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/48ca5edd-34a6-4bd1-8aa4-a50d897851be_isrForCrosstalkSources_855445_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/48ca5edd-34a6-4bd1-8aa4-a50d897851be_isrForCrosstalkSources_855445_43.stderr parsl.dataflow.dflow INFO: Task 1795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4615334a8349b4c3decd9cd91e4929c with result from task 1795 parsl.dataflow.dflow DEBUG: Task 1795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/710ce5a3-f4fa-47d4-8c07-0d595fd56f68_isrForCrosstalkSources_855392_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/710ce5a3-f4fa-47d4-8c07-0d595fd56f68_isrForCrosstalkSources_855392_36.stderr parsl.dataflow.dflow INFO: Task 1469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74492a77f9e89c34af27601a99326faa with result from task 1469 parsl.dataflow.dflow DEBUG: Task 1469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/69488734-090d-43d8-84d9-62d0c9c286e3_isrForCrosstalkSources_855405_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/69488734-090d-43d8-84d9-62d0c9c286e3_isrForCrosstalkSources_855405_59.stderr parsl.dataflow.dflow INFO: Task 1837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3106a0edd901b17eee304cf89155cd53 with result from task 1837 parsl.dataflow.dflow DEBUG: Task 1837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/12621c1b-b93a-49b0-a9fe-9d88547e61b7_isrForCrosstalkSources_855387_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/12621c1b-b93a-49b0-a9fe-9d88547e61b7_isrForCrosstalkSources_855387_23.stderr parsl.dataflow.dflow INFO: Task 1843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e8254cf29f64d95cdbb499246b45fa3 with result from task 1843 parsl.dataflow.dflow DEBUG: Task 1843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/3babbf68-ed10-4651-92e4-3b7966947cb9_isrForCrosstalkSources_855392_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/3babbf68-ed10-4651-92e4-3b7966947cb9_isrForCrosstalkSources_855392_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:12 proc_lsst.multi:146 [INFO] found job 16964097 in provider checkpoint proc_lsst.multi INFO: found job 16964097 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed36760a194ac88104db54c3923cdb5e with result from task 1979 parsl.dataflow.dflow DEBUG: Task 1979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7e707fae-158d-442e-ad93-c2022a669134_isrForCrosstalkSources_855421_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7e707fae-158d-442e-ad93-c2022a669134_isrForCrosstalkSources_855421_24.stderr parsl.dataflow.dflow INFO: Task 1876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e210ed259cc81cb97836f705db9bd4af with result from task 1876 parsl.dataflow.dflow DEBUG: Task 1876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2498b053-89c2-46ab-be72-1454bbfd7ab4_isrForCrosstalkSources_855450_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2498b053-89c2-46ab-be72-1454bbfd7ab4_isrForCrosstalkSources_855450_34.stderr parsl.dataflow.dflow INFO: Task 1963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead55a59a8e0bd530478fb1f001123ca with result from task 1963 parsl.dataflow.dflow DEBUG: Task 1963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/76c3ab63-9f5b-420f-8820-8ce827dba1d9_isrForCrosstalkSources_855410_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/76c3ab63-9f5b-420f-8820-8ce827dba1d9_isrForCrosstalkSources_855410_49.stderr parsl.dataflow.dflow INFO: Task 1991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf8af5e8768822741673516d7981da5 with result from task 1991 parsl.dataflow.dflow DEBUG: Task 1991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4a11e37-7432-452b-8ace-c642b0aa8978_isrForCrosstalkSources_855416_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4a11e37-7432-452b-8ace-c642b0aa8978_isrForCrosstalkSources_855416_28.stderr parsl.dataflow.dflow INFO: Task 1276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540e4a9f16d3913d291f08138106f809 with result from task 1276 parsl.dataflow.dflow DEBUG: Task 1276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0825954f-f40a-4eaa-acc9-4ad9fe09ab7c_isrForCrosstalkSources_855439_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0825954f-f40a-4eaa-acc9-4ad9fe09ab7c_isrForCrosstalkSources_855439_30.stderr parsl.dataflow.dflow INFO: Task 1940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b88716743f249d2f3bc318db3c3cf12f with result from task 1940 parsl.dataflow.dflow DEBUG: Task 1940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a476bb34-a9bc-4eea-bf60-a5cd7f433a60_isrForCrosstalkSources_855434_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a476bb34-a9bc-4eea-bf60-a5cd7f433a60_isrForCrosstalkSources_855434_39.stderr parsl.dataflow.dflow INFO: Task 2005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 712fbd8d37c4bc05598c522e4a7f1c03 with result from task 2005 parsl.dataflow.dflow DEBUG: Task 2005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f238862c-26ae-4a34-93e5-9c4d5aca61a5_isrForCrosstalkSources_855412_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f238862c-26ae-4a34-93e5-9c4d5aca61a5_isrForCrosstalkSources_855412_51.stderr parsl.dataflow.dflow INFO: Task 1284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30a5b5f37c416eb5d1c0d99e1320df0a with result from task 1284 parsl.dataflow.dflow DEBUG: Task 1284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5002c368-60c1-4754-90a1-e00547876651_isrForCrosstalkSources_855434_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5002c368-60c1-4754-90a1-e00547876651_isrForCrosstalkSources_855434_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:13 proc_lsst.multi:146 [INFO] found job 16964099 in provider checkpoint proc_lsst.multi INFO: found job 16964099 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf111da78bda043bd3fb5e0eba46f93a with result from task 1951 parsl.dataflow.dflow DEBUG: Task 1951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/250fd2bc-b6d9-4c9b-8de5-9c3a04b74d34_isrForCrosstalkSources_855459_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/250fd2bc-b6d9-4c9b-8de5-9c3a04b74d34_isrForCrosstalkSources_855459_20.stderr parsl.dataflow.dflow INFO: Task 1269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3478b1b4c27441e1b226c1442c58578 with result from task 1269 parsl.dataflow.dflow DEBUG: Task 1269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9e864875-c95a-4508-9213-cc41c538f83f_isrForCrosstalkSources_855439_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9e864875-c95a-4508-9213-cc41c538f83f_isrForCrosstalkSources_855439_52.stderr parsl.dataflow.dflow INFO: Task 1759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a8ebb890b1b04667df094da07476eb7 with result from task 1759 parsl.dataflow.dflow DEBUG: Task 1759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/96de1389-fb7e-4225-8fec-64a8a8d29bf6_isrForCrosstalkSources_855399_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/96de1389-fb7e-4225-8fec-64a8a8d29bf6_isrForCrosstalkSources_855399_31.stderr parsl.dataflow.dflow INFO: Task 2040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 650019115c10617615b765d1641f3605 with result from task 2040 parsl.dataflow.dflow DEBUG: Task 2040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/e4d947eb-7fe2-4fff-aec8-4115f0fc3405_isrForCrosstalkSources_855381_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/e4d947eb-7fe2-4fff-aec8-4115f0fc3405_isrForCrosstalkSources_855381_19.stderr parsl.dataflow.dflow INFO: Task 1954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00d8062b54bce6f1450a7b76bcbf3b85 with result from task 1954 parsl.dataflow.dflow DEBUG: Task 1954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/449b6622-843f-4e6d-b708-7a3004773eb4_isrForCrosstalkSources_855450_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/449b6622-843f-4e6d-b708-7a3004773eb4_isrForCrosstalkSources_855450_26.stderr parsl.dataflow.dflow INFO: Task 1919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 100a5d7da982b2e8ee146ea753f133f0 with result from task 1919 parsl.dataflow.dflow DEBUG: Task 1919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d8de3be5-9645-4fba-b414-ae559ac5b33d_isrForCrosstalkSources_855389_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d8de3be5-9645-4fba-b414-ae559ac5b33d_isrForCrosstalkSources_855389_34.stderr parsl.dataflow.dflow INFO: Task 1858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50d03f7a856b74ce9b199f59d5a7e67f with result from task 1858 parsl.dataflow.dflow DEBUG: Task 1858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/58b95eab-d647-4de1-9845-d519a4564081_isrForCrosstalkSources_855459_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/58b95eab-d647-4de1-9845-d519a4564081_isrForCrosstalkSources_855459_24.stderr parsl.dataflow.dflow INFO: Task 1805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5502a4083360695a7ba714712bf0627a with result from task 1805 parsl.dataflow.dflow DEBUG: Task 1805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f7b9e6a6-147c-448f-8cb3-805538ee09ab_isrForCrosstalkSources_855443_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/f7b9e6a6-147c-448f-8cb3-805538ee09ab_isrForCrosstalkSources_855443_28.stderr parsl.dataflow.dflow INFO: Task 1857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab17b59798e1c265ce3eab2c89efeb01 with result from task 1857 parsl.dataflow.dflow DEBUG: Task 1857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7e39670a-1d29-4b6f-b8ec-60a616d8d406_isrForCrosstalkSources_855388_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7e39670a-1d29-4b6f-b8ec-60a616d8d406_isrForCrosstalkSources_855388_42.stderr parsl.dataflow.dflow INFO: Task 2002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa891eb5f1aa886529daf8a7ce995b49 with result from task 2002 parsl.dataflow.dflow DEBUG: Task 2002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/3f5f8e1d-9433-48f8-a4f2-3b205378eb80_isrForCrosstalkSources_855437_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/3f5f8e1d-9433-48f8-a4f2-3b205378eb80_isrForCrosstalkSources_855437_52.stderr parsl.dataflow.dflow INFO: Task 1802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1f8144c9fa594bf97f63c9ce3c2817 with result from task 1802 parsl.dataflow.dflow DEBUG: Task 1802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f3b4ba15-ea67-4e95-9d51-0bb1374cb963_isrForCrosstalkSources_855384_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f3b4ba15-ea67-4e95-9d51-0bb1374cb963_isrForCrosstalkSources_855384_12.stderr parsl.dataflow.dflow INFO: Task 1782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f3fb433b72239827d93497d83f0e2fa with result from task 1782 parsl.dataflow.dflow DEBUG: Task 1782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/bf6a51de-0071-4421-8f86-0af91faf8c99_isrForCrosstalkSources_855390_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/bf6a51de-0071-4421-8f86-0af91faf8c99_isrForCrosstalkSources_855390_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 1834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc6e86a25e167e488eb2c94f92a4914c with result from task 1834 parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:14 proc_lsst.multi:146 [INFO] found job 16964100 in provider checkpoint proc_lsst.multi INFO: found job 16964100 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 1834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8401fc31-aef5-49ff-8d91-9421bc2edf05_isrForCrosstalkSources_855400_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8401fc31-aef5-49ff-8d91-9421bc2edf05_isrForCrosstalkSources_855400_39.stderr parsl.dataflow.dflow INFO: Task 1936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d657832b22b690e24c38b9f384a1c90 with result from task 1936 parsl.dataflow.dflow DEBUG: Task 1936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/21e265d9-4b97-4bdf-ab2b-10288144d2be_isrForCrosstalkSources_855419_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/21e265d9-4b97-4bdf-ab2b-10288144d2be_isrForCrosstalkSources_855419_21.stderr parsl.dataflow.dflow INFO: Task 1846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 658dfa4c0d2eb230c299ae20e6d75987 with result from task 1846 parsl.dataflow.dflow DEBUG: Task 1846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0970ceb5-50d0-4b08-9c50-5f49a7a2571a_isrForCrosstalkSources_855458_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0970ceb5-50d0-4b08-9c50-5f49a7a2571a_isrForCrosstalkSources_855458_45.stderr parsl.dataflow.dflow INFO: Task 1959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7043481fa64f33c4a651ab59796d9520 with result from task 1959 parsl.dataflow.dflow DEBUG: Task 1959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/63f1d73f-fdbb-4e5b-9069-be014fb73f59_isrForCrosstalkSources_855451_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/63f1d73f-fdbb-4e5b-9069-be014fb73f59_isrForCrosstalkSources_855451_61.stderr parsl.dataflow.dflow INFO: Task 1811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7789a6d8738d3060ed8a0a3706af480 with result from task 1811 parsl.dataflow.dflow DEBUG: Task 1811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/39906da9-4ea3-4f30-b6cc-772b7b5eb4f9_isrForCrosstalkSources_855422_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/39906da9-4ea3-4f30-b6cc-772b7b5eb4f9_isrForCrosstalkSources_855422_37.stderr parsl.dataflow.dflow INFO: Task 1874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aff32aba34f6f4f6572c9ff74617d375 with result from task 1874 parsl.dataflow.dflow DEBUG: Task 1874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e02b1aff-cbea-4a0b-a07e-cfe262dfdf78_isrForCrosstalkSources_855444_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e02b1aff-cbea-4a0b-a07e-cfe262dfdf78_isrForCrosstalkSources_855444_28.stderr parsl.dataflow.dflow INFO: Task 1779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c15ed4a31829ad9ec60d795d9d9cb3b with result from task 1779 parsl.dataflow.dflow DEBUG: Task 1779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/77b28a06-daba-4f41-bba6-95cdbe9a339f_isrForCrosstalkSources_855417_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/77b28a06-daba-4f41-bba6-95cdbe9a339f_isrForCrosstalkSources_855417_52.stderr parsl.dataflow.dflow INFO: Task 1807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95d2cf1609e2d9026b6039e34052dd59 with result from task 1807 parsl.dataflow.dflow DEBUG: Task 1807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/453a0617-e261-4f0f-982d-c13a22f235dd_isrForCrosstalkSources_855447_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/453a0617-e261-4f0f-982d-c13a22f235dd_isrForCrosstalkSources_855447_58.stderr parsl.dataflow.dflow INFO: Task 1280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da96e29ef240041d0bd664eee71bd8c5 with result from task 1280 parsl.dataflow.dflow DEBUG: Task 1280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9d449ebf-d4f4-4ff1-987d-8ca57b0ba138_isrForCrosstalkSources_855441_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9d449ebf-d4f4-4ff1-987d-8ca57b0ba138_isrForCrosstalkSources_855441_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:14 proc_lsst.multi:146 [INFO] found job 16964101 in provider checkpoint proc_lsst.multi INFO: found job 16964101 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efd3e555382c947ce8db7c4387ac3cfe with result from task 1879 parsl.dataflow.dflow DEBUG: Task 1879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/74ff7f73-0ccf-4d4e-9db7-fdb83997c5bf_isrForCrosstalkSources_855383_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/74ff7f73-0ccf-4d4e-9db7-fdb83997c5bf_isrForCrosstalkSources_855383_15.stderr parsl.dataflow.dflow INFO: Task 1953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79f4ebe69ecb017b7e36287fd411aade with result from task 1953 parsl.dataflow.dflow DEBUG: Task 1953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c20ecdbd-11be-4605-94f1-96c6a05601d1_isrForCrosstalkSources_855456_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c20ecdbd-11be-4605-94f1-96c6a05601d1_isrForCrosstalkSources_855456_46.stderr parsl.dataflow.dflow INFO: Task 1957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7264e67aff37027d0c00311e4b9cfe7 with result from task 1957 parsl.dataflow.dflow DEBUG: Task 1957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6895d392-8be2-463f-8105-cf4ef1228024_isrForCrosstalkSources_855456_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6895d392-8be2-463f-8105-cf4ef1228024_isrForCrosstalkSources_855456_56.stderr parsl.dataflow.dflow INFO: Task 1734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3472a2261cd732b6c7c43e32d45d9c72 with result from task 1734 parsl.dataflow.dflow DEBUG: Task 1734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/350813de-7427-4538-bf05-bf8a3be802d7_isrForCrosstalkSources_855383_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/350813de-7427-4538-bf05-bf8a3be802d7_isrForCrosstalkSources_855383_50.stderr parsl.dataflow.dflow INFO: Task 1861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de0e6f1c11565aa5e4e0e12a63ccd6fa with result from task 1861 parsl.dataflow.dflow DEBUG: Task 1861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/626982b9-0775-4657-b62e-94f7f2e97ceb_isrForCrosstalkSources_855431_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/626982b9-0775-4657-b62e-94f7f2e97ceb_isrForCrosstalkSources_855431_11.stderr parsl.dataflow.dflow INFO: Task 1894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdf88075d7f6e271f20c62b3e5592b39 with result from task 1894 parsl.dataflow.dflow DEBUG: Task 1894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8a785cba-f527-44ca-a17c-fd23cb9a0a3c_isrForCrosstalkSources_855412_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/8a785cba-f527-44ca-a17c-fd23cb9a0a3c_isrForCrosstalkSources_855412_56.stderr parsl.dataflow.dflow INFO: Task 1969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f068bc185ef4f8528bf6a91fe379443a with result from task 1969 parsl.dataflow.dflow DEBUG: Task 1969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b7c909f3-c8b3-46f0-a63e-222561f37adf_isrForCrosstalkSources_855385_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/b7c909f3-c8b3-46f0-a63e-222561f37adf_isrForCrosstalkSources_855385_8.stderr parsl.dataflow.dflow INFO: Task 1903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05a76c201573590c8048d07899a4339c with result from task 1903 parsl.dataflow.dflow DEBUG: Task 1903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b9d49ebf-a5b1-4f71-8959-cc8c891a352c_isrForCrosstalkSources_855389_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b9d49ebf-a5b1-4f71-8959-cc8c891a352c_isrForCrosstalkSources_855389_11.stderr parsl.dataflow.dflow INFO: Task 1902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b6bd94b06888a05d7f03a0f293ff60d with result from task 1902 parsl.dataflow.dflow DEBUG: Task 1902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e423fb9-4d30-4f31-bfd6-3b25c89cb74f_isrForCrosstalkSources_855412_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e423fb9-4d30-4f31-bfd6-3b25c89cb74f_isrForCrosstalkSources_855412_24.stderr parsl.dataflow.dflow INFO: Task 1766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 646a340b73a56bb428e64546786f6d6e with result from task 1766 parsl.dataflow.dflow DEBUG: Task 1766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04ec688b-6542-40a4-99f6-c00a3a89a859_isrForCrosstalkSources_855412_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04ec688b-6542-40a4-99f6-c00a3a89a859_isrForCrosstalkSources_855412_47.stderr parsl.dataflow.dflow INFO: Task 1756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3649496546f6e8dc0f052a8fb844c9 with result from task 1756 parsl.dataflow.dflow DEBUG: Task 1756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/799044fd-d423-415c-ae73-38243ebe223d_isrForCrosstalkSources_855435_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/799044fd-d423-415c-ae73-38243ebe223d_isrForCrosstalkSources_855435_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:15 proc_lsst.multi:146 [INFO] found job 16964102 in provider checkpoint proc_lsst.multi INFO: found job 16964102 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 783b2ddfd188c7bf5a6a3f19eab33812 with result from task 1713 parsl.dataflow.dflow DEBUG: Task 1713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c3b41fd2-eea0-4e22-85d8-212f0df63acc_isrForCrosstalkSources_855404_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/c3b41fd2-eea0-4e22-85d8-212f0df63acc_isrForCrosstalkSources_855404_16.stderr parsl.dataflow.dflow INFO: Task 1813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e93a9bc0120ca1c06e0093ea90e0b229 with result from task 1813 parsl.dataflow.dflow DEBUG: Task 1813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4b17960-2029-48cc-b4df-6c3b4ae4e7f9_isrForCrosstalkSources_855393_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c4b17960-2029-48cc-b4df-6c3b4ae4e7f9_isrForCrosstalkSources_855393_62.stderr parsl.dataflow.dflow INFO: Task 1946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59df9c4e2fc7a7da2fd0a93eda8f8490 with result from task 1946 parsl.dataflow.dflow DEBUG: Task 1946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/58a6e9a4-3f11-4603-becd-6f49817981de_isrForCrosstalkSources_855448_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/58a6e9a4-3f11-4603-becd-6f49817981de_isrForCrosstalkSources_855448_22.stderr parsl.dataflow.dflow INFO: Task 1733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6240a3c5346e349f61e5d2a5376b454 with result from task 1733 parsl.dataflow.dflow DEBUG: Task 1733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/60d758d9-8221-412d-abf9-1244eedea323_isrForCrosstalkSources_855406_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/60d758d9-8221-412d-abf9-1244eedea323_isrForCrosstalkSources_855406_43.stderr parsl.dataflow.dflow INFO: Task 1735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e912636d8628ed2d357b02c34a552de with result from task 1735 parsl.dataflow.dflow DEBUG: Task 1735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/44fd24ef-cb85-4272-933d-4b70427b1906_isrForCrosstalkSources_855426_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/44fd24ef-cb85-4272-933d-4b70427b1906_isrForCrosstalkSources_855426_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:15 proc_lsst.multi:146 [INFO] found job 16964103 in provider checkpoint proc_lsst.multi INFO: found job 16964103 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd7701fe557eb147870908cf409d43de with result from task 1911 parsl.dataflow.dflow DEBUG: Task 1911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/21195a7e-bb75-4026-9fcd-ef144a982fce_isrForCrosstalkSources_855410_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/21195a7e-bb75-4026-9fcd-ef144a982fce_isrForCrosstalkSources_855410_1.stderr parsl.dataflow.dflow INFO: Task 1875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5770a661a3ae48ea9bdb7bfb2c6839e5 with result from task 1875 parsl.dataflow.dflow DEBUG: Task 1875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9cb028d3-3a2a-4a24-8ca8-1a025ee85ec9_isrForCrosstalkSources_855439_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/9cb028d3-3a2a-4a24-8ca8-1a025ee85ec9_isrForCrosstalkSources_855439_2.stderr parsl.dataflow.dflow INFO: Task 1789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baa068fb62192fe9f7c3040c0a0b3d61 with result from task 1789 parsl.dataflow.dflow DEBUG: Task 1789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f816d250-83ef-4f25-b3c8-21c8be0ab492_isrForCrosstalkSources_855397_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f816d250-83ef-4f25-b3c8-21c8be0ab492_isrForCrosstalkSources_855397_49.stderr parsl.dataflow.dflow INFO: Task 1860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c003a7e6451fc556df8abd27a2682a41 with result from task 1860 parsl.dataflow.dflow DEBUG: Task 1860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6e91be4c-fe7a-4b06-86a6-2d1984f35767_isrForCrosstalkSources_855407_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/6e91be4c-fe7a-4b06-86a6-2d1984f35767_isrForCrosstalkSources_855407_7.stderr parsl.dataflow.dflow INFO: Task 1935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0ca8fa9f42b214c4296009d8a591beb with result from task 1935 parsl.dataflow.dflow DEBUG: Task 1935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8ab65490-f988-48a3-a695-d217c272ec7a_isrForCrosstalkSources_855454_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8ab65490-f988-48a3-a695-d217c272ec7a_isrForCrosstalkSources_855454_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:15 proc_lsst.multi:146 [INFO] found job 16964104 in provider checkpoint proc_lsst.multi INFO: found job 16964104 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e52ea9baa6803319af51e072743f7e with result from task 1928 parsl.dataflow.dflow DEBUG: Task 1928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1113e670-6436-4030-b097-0754a87fb00c_isrForCrosstalkSources_855424_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1113e670-6436-4030-b097-0754a87fb00c_isrForCrosstalkSources_855424_56.stderr parsl.dataflow.dflow INFO: Task 1832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7441ad79bed26e87c7ac5cc1870401b5 with result from task 1832 parsl.dataflow.dflow DEBUG: Task 1832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b779c11d-7edf-48da-8d28-ae0ad501e80f_isrForCrosstalkSources_855396_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b779c11d-7edf-48da-8d28-ae0ad501e80f_isrForCrosstalkSources_855396_16.stderr parsl.dataflow.dflow INFO: Task 1838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a52b314391ad73325490b6ae6d5c939d with result from task 1838 parsl.dataflow.dflow DEBUG: Task 1838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/582419fa-4ebd-4df0-b17b-ffe466d13fa6_isrForCrosstalkSources_855431_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/582419fa-4ebd-4df0-b17b-ffe466d13fa6_isrForCrosstalkSources_855431_46.stderr parsl.dataflow.dflow INFO: Task 1930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c9953660bd0c8a527f0c3f3bf040f35 with result from task 1930 parsl.dataflow.dflow DEBUG: Task 1930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bec1640a-e947-43b0-89c9-374608d593f0_isrForCrosstalkSources_855396_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/bec1640a-e947-43b0-89c9-374608d593f0_isrForCrosstalkSources_855396_7.stderr parsl.dataflow.dflow INFO: Task 1944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36e72837f835b36965f42d533b3c2be2 with result from task 1944 parsl.dataflow.dflow DEBUG: Task 1944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f3ba20f4-7186-4e3e-95b0-87cabf2fbfc7_isrForCrosstalkSources_855393_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f3ba20f4-7186-4e3e-95b0-87cabf2fbfc7_isrForCrosstalkSources_855393_17.stderr parsl.dataflow.dflow INFO: Task 1921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb0527a5969d4e1e1b2043e390dedbe0 with result from task 1921 parsl.dataflow.dflow DEBUG: Task 1921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/940dc7ff-7a7d-457e-8ab6-08cbf3d3853c_isrForCrosstalkSources_855429_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/940dc7ff-7a7d-457e-8ab6-08cbf3d3853c_isrForCrosstalkSources_855429_39.stderr parsl.dataflow.dflow INFO: Task 1741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdd2873aa2ea4fc91b433d95868a1277 with result from task 1741 parsl.dataflow.dflow DEBUG: Task 1741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/98eded79-5774-4173-a265-498850a32124_isrForCrosstalkSources_855425_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/98eded79-5774-4173-a265-498850a32124_isrForCrosstalkSources_855425_48.stderr parsl.dataflow.dflow INFO: Task 1932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0255c703a430fbd75fda354b6a1b1764 with result from task 1932 parsl.dataflow.dflow DEBUG: Task 1932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b00798bf-8bcf-41c2-880c-308d9a85a370_isrForCrosstalkSources_855439_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b00798bf-8bcf-41c2-880c-308d9a85a370_isrForCrosstalkSources_855439_1.stderr parsl.dataflow.dflow INFO: Task 1955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ea0b1451abd4d2671148c95640b63a with result from task 1955 parsl.dataflow.dflow DEBUG: Task 1955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbf36d2d-f079-4b87-b3d2-75298b7455c1_isrForCrosstalkSources_855431_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbf36d2d-f079-4b87-b3d2-75298b7455c1_isrForCrosstalkSources_855431_17.stderr parsl.dataflow.dflow INFO: Task 1730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd01af9b45823307dd3570af851fc8c6 with result from task 1730 parsl.dataflow.dflow DEBUG: Task 1730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ad2ff7ba-ccf0-4e1f-bd3a-9d6f4271bb61_isrForCrosstalkSources_855430_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ad2ff7ba-ccf0-4e1f-bd3a-9d6f4271bb61_isrForCrosstalkSources_855430_55.stderr parsl.dataflow.dflow INFO: Task 1947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a995fe995d38b7f30ab71d3762a0ad32 with result from task 1947 parsl.dataflow.dflow DEBUG: Task 1947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/961bc602-d802-4d2b-a13b-586dd12e4893_isrForCrosstalkSources_855397_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/961bc602-d802-4d2b-a13b-586dd12e4893_isrForCrosstalkSources_855397_56.stderr parsl.dataflow.dflow INFO: Task 1996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98dbc79d803cc363ea36d402b8fb07c7 with result from task 1996 parsl.dataflow.dflow DEBUG: Task 1996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a584c91c-ba0f-4996-9884-13965bc4e644_isrForCrosstalkSources_855384_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a584c91c-ba0f-4996-9884-13965bc4e644_isrForCrosstalkSources_855384_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:16 proc_lsst.multi:146 [INFO] found job 16964105 in provider checkpoint proc_lsst.multi INFO: found job 16964105 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ef7d321d2f8718f4063695911d382d with result from task 1286 parsl.dataflow.dflow DEBUG: Task 1286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/aa5a47e5-3de3-407f-9ab8-72cf09b19cf1_isrForCrosstalkSources_855398_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/aa5a47e5-3de3-407f-9ab8-72cf09b19cf1_isrForCrosstalkSources_855398_2.stderr parsl.dataflow.dflow INFO: Task 1885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2617f1cf10a377319f61f747d30e1a9d with result from task 1885 parsl.dataflow.dflow DEBUG: Task 1885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c90706ad-c141-48f0-92d2-076b767568b4_isrForCrosstalkSources_855385_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c90706ad-c141-48f0-92d2-076b767568b4_isrForCrosstalkSources_855385_15.stderr parsl.dataflow.dflow INFO: Task 1816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9667916d1ada5a3ace08480414e3bc1 with result from task 1816 parsl.dataflow.dflow DEBUG: Task 1816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c4b6436f-2e35-425f-9901-528a36631b6f_isrForCrosstalkSources_855422_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/c4b6436f-2e35-425f-9901-528a36631b6f_isrForCrosstalkSources_855422_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:16 proc_lsst.multi:146 [INFO] found job 16964106 in provider checkpoint proc_lsst.multi INFO: found job 16964106 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 137dc3ee7c68f56fd0bc676030536752 with result from task 1793 parsl.dataflow.dflow DEBUG: Task 1793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2e1a02dc-c049-4a62-9a0d-009cca33daee_isrForCrosstalkSources_855459_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2e1a02dc-c049-4a62-9a0d-009cca33daee_isrForCrosstalkSources_855459_9.stderr parsl.dataflow.dflow INFO: Task 1988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0b68b3aca6bab55fca45205f65e3e22 with result from task 1988 parsl.dataflow.dflow DEBUG: Task 1988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/587399b0-894e-4b0f-94f6-0c39693e5a60_isrForCrosstalkSources_855394_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/587399b0-894e-4b0f-94f6-0c39693e5a60_isrForCrosstalkSources_855394_9.stderr parsl.dataflow.dflow INFO: Task 1887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82041ef037e6a0eb13b511ade6f016c3 with result from task 1887 parsl.dataflow.dflow DEBUG: Task 1887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ad5e897e-4c2a-4097-9f78-2cc093682b93_isrForCrosstalkSources_855386_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/ad5e897e-4c2a-4097-9f78-2cc093682b93_isrForCrosstalkSources_855386_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 1830 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry a6aa59df0c968e1d7feb9b5f852653c2 with result from task 1830 parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 1830 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc2fbac9-4959-4638-bef6-bec5edf9fd06_isrForCrosstalkSources_855403_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc2fbac9-4959-4638-bef6-bec5edf9fd06_isrForCrosstalkSources_855403_21.stderr parsl.dataflow.dflow INFO: Task 1281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf62f5bfd6a77653dff9a29c6623f89 with result from task 1281 parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 1281 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/36240c6f-d100-46aa-af24-cfff02026b8e_isrForCrosstalkSources_855402_58.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/36240c6f-d100-46aa-af24-cfff02026b8e_isrForCrosstalkSources_855402_58.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:16 proc_lsst.multi:146 [INFO] found job 16964107 in provider checkpoint proc_lsst.multi INFO: found job 16964107 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5980be11b386b812ba8b17c6ddc18f with result from task 2036 parsl.dataflow.dflow DEBUG: Task 2036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6ddef829-2254-4ef4-b6b6-6373d1e43c72_isrForCrosstalkSources_855443_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6ddef829-2254-4ef4-b6b6-6373d1e43c72_isrForCrosstalkSources_855443_18.stderr parsl.dataflow.dflow INFO: Task 1886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d625b30cc85fda5da85aebb5a2d8989 with result from task 1886 parsl.dataflow.dflow DEBUG: Task 1886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/315d54ab-d072-4363-b854-90af72f41549_isrForCrosstalkSources_855427_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/315d54ab-d072-4363-b854-90af72f41549_isrForCrosstalkSources_855427_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:17 proc_lsst.multi:146 [INFO] found job 16964108 in provider checkpoint proc_lsst.multi INFO: found job 16964108 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf7c3afd1ad2eeca56bd90232c6d2bd4 with result from task 1866 parsl.dataflow.dflow DEBUG: Task 1866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e67f1533-7fa1-425c-91ae-4d2cc8cf1ccb_isrForCrosstalkSources_855447_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e67f1533-7fa1-425c-91ae-4d2cc8cf1ccb_isrForCrosstalkSources_855447_6.stderr parsl.dataflow.dflow INFO: Task 1985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cca5ff3b391d4a5ccb2124fea9e115c with result from task 1985 parsl.dataflow.dflow DEBUG: Task 1985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/704b92d2-59ef-4efd-b43a-afd3c6cebadb_isrForCrosstalkSources_855438_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/704b92d2-59ef-4efd-b43a-afd3c6cebadb_isrForCrosstalkSources_855438_51.stderr parsl.dataflow.dflow INFO: Task 1958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12336cbde60981a01294dc2f0411c6f8 with result from task 1958 parsl.dataflow.dflow DEBUG: Task 1958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/55fad503-a752-4174-abf2-f5ce5b34a68d_isrForCrosstalkSources_855386_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/55fad503-a752-4174-abf2-f5ce5b34a68d_isrForCrosstalkSources_855386_47.stderr parsl.dataflow.dflow INFO: Task 1869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af99031d99876db2ba62289bc730a9f3 with result from task 1869 parsl.dataflow.dflow DEBUG: Task 1869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e5c93510-4d93-4fd7-9bc2-1bd1a392b102_isrForCrosstalkSources_855453_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e5c93510-4d93-4fd7-9bc2-1bd1a392b102_isrForCrosstalkSources_855453_37.stderr parsl.dataflow.dflow INFO: Task 1862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0a92ac22a2e57770a9f059de7597b7 with result from task 1862 parsl.dataflow.dflow DEBUG: Task 1862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/42dbce6d-ed36-46ef-95a6-b580f9be7844_isrForCrosstalkSources_855398_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/42dbce6d-ed36-46ef-95a6-b580f9be7844_isrForCrosstalkSources_855398_56.stderr parsl.dataflow.dflow INFO: Task 1976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8d20c7f0d04e9ab3869474678c16c88 with result from task 1976 parsl.dataflow.dflow DEBUG: Task 1976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7a9e82a0-1b98-4f28-a556-ec6bbb9d3952_isrForCrosstalkSources_855402_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/7a9e82a0-1b98-4f28-a556-ec6bbb9d3952_isrForCrosstalkSources_855402_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:17 proc_lsst.multi:146 [INFO] found job 16964109 in provider checkpoint proc_lsst.multi INFO: found job 16964109 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d415d454e99d9297bd509746ef54340f with result from task 1847 parsl.dataflow.dflow DEBUG: Task 1847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/1cf4752c-989c-49f3-8cc6-5c4039fc5fd5_isrForCrosstalkSources_855406_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/1cf4752c-989c-49f3-8cc6-5c4039fc5fd5_isrForCrosstalkSources_855406_31.stderr parsl.dataflow.dflow INFO: Task 1971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccbc29f71476092d2f81a36b88da7007 with result from task 1971 parsl.dataflow.dflow DEBUG: Task 1971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a25d29e6-66e9-4c4f-9cde-049f27e6cff6_isrForCrosstalkSources_855401_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a25d29e6-66e9-4c4f-9cde-049f27e6cff6_isrForCrosstalkSources_855401_42.stderr parsl.dataflow.dflow INFO: Task 1920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdd2247f958bc23aa2ed8052fa6556e3 with result from task 1920 parsl.dataflow.dflow DEBUG: Task 1920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/db09b754-5d2b-42aa-b271-e795ead71bbe_isrForCrosstalkSources_855455_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/db09b754-5d2b-42aa-b271-e795ead71bbe_isrForCrosstalkSources_855455_29.stderr parsl.dataflow.dflow INFO: Task 1923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4a42ae73d03b07138a72ff77cac629a with result from task 1923 parsl.dataflow.dflow DEBUG: Task 1923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/632a077a-4a2e-45b5-8380-f4d81fc98397_isrForCrosstalkSources_855402_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/632a077a-4a2e-45b5-8380-f4d81fc98397_isrForCrosstalkSources_855402_6.stderr parsl.dataflow.dflow INFO: Task 1279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 734fb6b3dab56eef7070fa2fe2d16919 with result from task 1279 parsl.dataflow.dflow DEBUG: Task 1279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1ad3f91-1ee0-41a4-b3d3-6ca93fa41397_isrForCrosstalkSources_855395_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1ad3f91-1ee0-41a4-b3d3-6ca93fa41397_isrForCrosstalkSources_855395_62.stderr parsl.dataflow.dflow INFO: Task 1801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d66a5aa0fe86966a732eed01c1d2fb82 with result from task 1801 parsl.dataflow.dflow DEBUG: Task 1801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/206149cf-ce73-448b-889e-c4cbcd1009e7_isrForCrosstalkSources_855459_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/206149cf-ce73-448b-889e-c4cbcd1009e7_isrForCrosstalkSources_855459_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:17 proc_lsst.multi:146 [INFO] found job 16964110 in provider checkpoint proc_lsst.multi INFO: found job 16964110 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae32aab056d801804389aff132542bdf with result from task 1283 parsl.dataflow.dflow DEBUG: Task 1283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ad6c8989-4394-4259-9567-b77d43318e0e_isrForCrosstalkSources_855437_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ad6c8989-4394-4259-9567-b77d43318e0e_isrForCrosstalkSources_855437_54.stderr parsl.dataflow.dflow INFO: Task 1917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34e9be1fa6c8d760b1c07ca02ff847f0 with result from task 1917 parsl.dataflow.dflow DEBUG: Task 1917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/2e267d5f-8875-4758-8816-c510fd7d766f_isrForCrosstalkSources_855431_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/2e267d5f-8875-4758-8816-c510fd7d766f_isrForCrosstalkSources_855431_58.stderr parsl.dataflow.dflow INFO: Task 1792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b351b5809831c01028a36f6b75993ed7 with result from task 1792 parsl.dataflow.dflow DEBUG: Task 1792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/7920339d-1b00-4337-b1e0-f6ce81b0942b_isrForCrosstalkSources_855394_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/7920339d-1b00-4337-b1e0-f6ce81b0942b_isrForCrosstalkSources_855394_57.stderr parsl.dataflow.dflow INFO: Task 1895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb66f8579234545bc245b1ebfd4e6605 with result from task 1895 parsl.dataflow.dflow DEBUG: Task 1895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/171516e8-8a45-445c-9c47-9cb7fa908dd6_isrForCrosstalkSources_855407_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/171516e8-8a45-445c-9c47-9cb7fa908dd6_isrForCrosstalkSources_855407_42.stderr parsl.dataflow.dflow INFO: Task 2038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4610a26e54e6992fa1334a0985e53b with result from task 2038 parsl.dataflow.dflow DEBUG: Task 2038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b870f740-302b-4c57-a489-eb174b55514a_isrForCrosstalkSources_855408_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b870f740-302b-4c57-a489-eb174b55514a_isrForCrosstalkSources_855408_34.stderr parsl.dataflow.dflow INFO: Task 2018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e15649753590282583ae83384f2caeb with result from task 2018 parsl.dataflow.dflow DEBUG: Task 2018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/dd81a42b-9b0f-4f84-b467-a01ee476377b_isrForCrosstalkSources_855448_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/dd81a42b-9b0f-4f84-b467-a01ee476377b_isrForCrosstalkSources_855448_36.stderr parsl.dataflow.dflow INFO: Task 2103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 669ed5968d19f4e155dcd640e392ee08 with result from task 2103 parsl.dataflow.dflow DEBUG: Task 2103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/41b341fc-dd52-4d81-a5ee-6af936aac9e1_isrForCrosstalkSources_855416_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/41b341fc-dd52-4d81-a5ee-6af936aac9e1_isrForCrosstalkSources_855416_33.stderr parsl.dataflow.dflow INFO: Task 1948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0579c09819e2bb2583492a8a9f3712b with result from task 1948 parsl.dataflow.dflow DEBUG: Task 1948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/78389a41-b710-4f5c-8bbc-b7d3d412b714_isrForCrosstalkSources_855399_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/78389a41-b710-4f5c-8bbc-b7d3d412b714_isrForCrosstalkSources_855399_14.stderr parsl.dataflow.dflow INFO: Task 1933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0caf66eb79b2929cfa0c9f147d3c9c4f with result from task 1933 parsl.dataflow.dflow DEBUG: Task 1933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0e07786e-be2f-4290-a2af-91064a7a9d2a_isrForCrosstalkSources_855384_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0e07786e-be2f-4290-a2af-91064a7a9d2a_isrForCrosstalkSources_855384_36.stderr parsl.dataflow.dflow INFO: Task 1924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2863a77775343fa2ab431eb61dfe373 with result from task 1924 parsl.dataflow.dflow DEBUG: Task 1924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/499d1780-4255-42fc-a166-2ee06a63996f_isrForCrosstalkSources_855440_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/499d1780-4255-42fc-a166-2ee06a63996f_isrForCrosstalkSources_855440_44.stderr parsl.dataflow.dflow INFO: Task 1502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69d0648962967274672c4d17b37f257a with result from task 1502 parsl.dataflow.dflow DEBUG: Task 1502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/13b4a2d7-d988-4b00-9a70-1a608501ce84_isrForCrosstalkSources_855424_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/13b4a2d7-d988-4b00-9a70-1a608501ce84_isrForCrosstalkSources_855424_3.stderr parsl.dataflow.dflow INFO: Task 2090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0763553b745a4a98e33a2126aca147bf with result from task 2090 parsl.dataflow.dflow DEBUG: Task 2090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f2e1c7f5-2f69-4a1c-b036-58060b61e4a7_isrForCrosstalkSources_855419_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f2e1c7f5-2f69-4a1c-b036-58060b61e4a7_isrForCrosstalkSources_855419_9.stderr parsl.dataflow.dflow INFO: Task 2048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4640c8d289cf4d303e38aa0ad7ffe1 with result from task 2048 parsl.dataflow.dflow DEBUG: Task 2048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9cc6f4e4-59fd-406b-af78-1e0f678db0dd_isrForCrosstalkSources_855408_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/9cc6f4e4-59fd-406b-af78-1e0f678db0dd_isrForCrosstalkSources_855408_11.stderr parsl.dataflow.dflow INFO: Task 1900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7687b50ba338a9caf0533950c386919 with result from task 1900 parsl.dataflow.dflow DEBUG: Task 1900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/59af85cd-c067-46c2-ae9f-ff46e120faaf_isrForCrosstalkSources_855457_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/59af85cd-c067-46c2-ae9f-ff46e120faaf_isrForCrosstalkSources_855457_50.stderr parsl.dataflow.dflow INFO: Task 1767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c71b09b5de2c40c451d21394fd59866a with result from task 1767 parsl.dataflow.dflow DEBUG: Task 1767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2310f16b-cef8-49a2-a2ea-ca7f2655e064_isrForCrosstalkSources_855382_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2310f16b-cef8-49a2-a2ea-ca7f2655e064_isrForCrosstalkSources_855382_35.stderr parsl.dataflow.dflow INFO: Task 1277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b808380ea4f1c89f8ab8c9a74ce32956 with result from task 1277 parsl.dataflow.dflow DEBUG: Task 1277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f05fd4f0-6b1f-4540-adb9-303eb0bcf69c_isrForCrosstalkSources_855396_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f05fd4f0-6b1f-4540-adb9-303eb0bcf69c_isrForCrosstalkSources_855396_49.stderr parsl.dataflow.dflow INFO: Task 2021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b8d584de3b9044c95693f5fe8d77276 with result from task 2021 parsl.dataflow.dflow DEBUG: Task 2021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3fca9484-443c-488d-a9bc-b69bc4c26f68_isrForCrosstalkSources_855408_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/3fca9484-443c-488d-a9bc-b69bc4c26f68_isrForCrosstalkSources_855408_24.stderr parsl.dataflow.dflow INFO: Task 1545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f75298542a956e416631a6e5eefb7256 with result from task 1545 parsl.dataflow.dflow DEBUG: Task 1545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0e5efe77-a5d5-4de1-982e-ae0c232038dd_isrForCrosstalkSources_855387_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0e5efe77-a5d5-4de1-982e-ae0c232038dd_isrForCrosstalkSources_855387_59.stderr parsl.dataflow.dflow INFO: Task 2016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afcade48dc852bb89902614510e22290 with result from task 2016 parsl.dataflow.dflow DEBUG: Task 2016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3865894-a9a7-4f20-9d80-07ea690abaa6_isrForCrosstalkSources_855379_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/a3865894-a9a7-4f20-9d80-07ea690abaa6_isrForCrosstalkSources_855379_3.stderr parsl.dataflow.dflow INFO: Task 1922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 431a1c04f332c44832afe15b9db556da with result from task 1922 parsl.dataflow.dflow DEBUG: Task 1922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2b9ce8c8-2d0f-4615-a7e7-f5c1b089809f_isrForCrosstalkSources_855409_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2b9ce8c8-2d0f-4615-a7e7-f5c1b089809f_isrForCrosstalkSources_855409_44.stderr parsl.dataflow.dflow INFO: Task 1978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55cee7b48badbb113c808e8e401323ff with result from task 1978 parsl.dataflow.dflow DEBUG: Task 1978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/18770fe1-8839-4a78-b345-0eabaab45efd_isrForCrosstalkSources_855445_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/18770fe1-8839-4a78-b345-0eabaab45efd_isrForCrosstalkSources_855445_47.stderr parsl.dataflow.dflow INFO: Task 2063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8146886aa77092799d35e69f1eaa0f5f with result from task 2063 parsl.dataflow.dflow DEBUG: Task 2063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2b3e1556-1818-422a-87a8-1579a5172393_isrForCrosstalkSources_855449_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/2b3e1556-1818-422a-87a8-1579a5172393_isrForCrosstalkSources_855449_22.stderr parsl.dataflow.dflow INFO: Task 2047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e24de92c4ad6d9d1c5e6d19af10d88d6 with result from task 2047 parsl.dataflow.dflow DEBUG: Task 2047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/86344761-d67c-4c1d-b347-c1e371fc5352_isrForCrosstalkSources_855459_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/86344761-d67c-4c1d-b347-c1e371fc5352_isrForCrosstalkSources_855459_45.stderr parsl.dataflow.dflow INFO: Task 1844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aa90c9c67ccddec815c392a26b08816 with result from task 1844 parsl.dataflow.dflow DEBUG: Task 1844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8688f86c-e315-44dc-b3c9-d2d8f3c98851_isrForCrosstalkSources_855459_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8688f86c-e315-44dc-b3c9-d2d8f3c98851_isrForCrosstalkSources_855459_28.stderr parsl.dataflow.dflow INFO: Task 2075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac23bdeeb02edfb934463735282edcc5 with result from task 2075 parsl.dataflow.dflow DEBUG: Task 2075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b20cc6b7-984e-44f7-8e83-9b21cdaa8d62_isrForCrosstalkSources_855394_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/b20cc6b7-984e-44f7-8e83-9b21cdaa8d62_isrForCrosstalkSources_855394_25.stderr parsl.dataflow.dflow INFO: Task 1822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35e47b27678291839f35869bab9c62af with result from task 1822 parsl.dataflow.dflow DEBUG: Task 1822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8fa3c929-dfad-4b93-820a-8a9cffd05210_isrForCrosstalkSources_855425_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8fa3c929-dfad-4b93-820a-8a9cffd05210_isrForCrosstalkSources_855425_50.stderr parsl.dataflow.dflow INFO: Task 2030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec418bec519a6a34071ba1d2f26dd6dd with result from task 2030 parsl.dataflow.dflow DEBUG: Task 2030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/9c66eb67-7d1e-4b4d-abe7-796a3119d987_isrForCrosstalkSources_855416_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/9c66eb67-7d1e-4b4d-abe7-796a3119d987_isrForCrosstalkSources_855416_14.stderr parsl.dataflow.dflow INFO: Task 2052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64fd292af5305885ca3554650a9549ea with result from task 2052 parsl.dataflow.dflow DEBUG: Task 2052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4ecc0614-68fa-4922-b095-63ad18be6b9a_isrForCrosstalkSources_855412_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/4ecc0614-68fa-4922-b095-63ad18be6b9a_isrForCrosstalkSources_855412_46.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ed0fdebe233f58ac373d8dc1ff0a33 with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ae041e43-af2a-4f42-a904-d5512ffc1c10_isrForCrosstalkSources_855388_9.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ae041e43-af2a-4f42-a904-d5512ffc1c10_isrForCrosstalkSources_855388_9.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a11218381ee787a92fd4e038db4e87 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c7e73acf-a325-4ea6-b95e-899129ad20fd_isrForCrosstalkSources_855400_24.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c7e73acf-a325-4ea6-b95e-899129ad20fd_isrForCrosstalkSources_855400_24.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a245962cff82485bb73626f62bbbead0 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3ff40679-4ec4-46a9-aa11-f12c7f116106_isrForCrosstalkSources_855448_53.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3ff40679-4ec4-46a9-aa11-f12c7f116106_isrForCrosstalkSources_855448_53.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5292c9db49eb0473e33377351e741ca7 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4bcb196-143d-487d-b4f2-7528c6ab524f_isrForCrosstalkSources_855434_51.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a4bcb196-143d-487d-b4f2-7528c6ab524f_isrForCrosstalkSources_855434_51.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e535ca99cd08fc7444c30a8a4e28769 with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e69ac845-2e19-4ec0-a56f-9e233a79202e_isrForCrosstalkSources_855382_6.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e69ac845-2e19-4ec0-a56f-9e233a79202e_isrForCrosstalkSources_855382_6.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de69c4075a61ec935e656cda56b43ee9 with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6f993781-2545-4125-9b63-e2d111ce0d47_isrForCrosstalkSources_855446_15.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6f993781-2545-4125-9b63-e2d111ce0d47_isrForCrosstalkSources_855446_15.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fac9973091e3a47ba74611d606b627e8 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/15e8b1f3-1652-4878-9f6e-3707a892d2ad_isrForCrosstalkSources_855438_46.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/15e8b1f3-1652-4878-9f6e-3707a892d2ad_isrForCrosstalkSources_855438_46.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 976a0e4346d4dea6d15f6b1581c86d06 with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/0c0ac941-263e-4408-94a8-cee234a07647_isrForCrosstalkSources_855442_1.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/0c0ac941-263e-4408-94a8-cee234a07647_isrForCrosstalkSources_855442_1.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 761c5d10f22ad8878b5796d0df2e79e3 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c532a6e8-f216-4913-8873-9332ef81cda8_isrForCrosstalkSources_855389_52.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c532a6e8-f216-4913-8873-9332ef81cda8_isrForCrosstalkSources_855389_52.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23408013c1def771d6fc102b41c2fe28 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dd141d94-1bb1-4c9a-8945-9a3ff788b183_isrForCrosstalkSources_855450_3.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dd141d94-1bb1-4c9a-8945-9a3ff788b183_isrForCrosstalkSources_855450_3.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d05cac9ee482bbb7faf68c797b86986 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ec1d25b1-af6b-4f8f-9aac-68b930f219d5_isrForCrosstalkSources_855410_30.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ec1d25b1-af6b-4f8f-9aac-68b930f219d5_isrForCrosstalkSources_855410_30.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbc625620899d92fd03ad5327ec5923 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d21b97e6-c5d7-4aca-94ea-b54aa9debfcb_isrForCrosstalkSources_855439_21.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d21b97e6-c5d7-4aca-94ea-b54aa9debfcb_isrForCrosstalkSources_855439_21.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f719a1de98535bb9c3e26cccb44701a6 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/40751d0e-c512-42cb-b54f-f25b11fd7fde_isrForCrosstalkSources_855402_1.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/40751d0e-c512-42cb-b54f-f25b11fd7fde_isrForCrosstalkSources_855402_1.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d213174072af27e9763ab5cdd9278b56 with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6270cd0a-309c-4c45-94b6-94e80ed23445_isrForCrosstalkSources_855446_19.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/6270cd0a-309c-4c45-94b6-94e80ed23445_isrForCrosstalkSources_855446_19.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bdeb857e67af0934778b0af73ecd95b with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/3c2c280f-b215-4004-96d4-10f3081c1630_isrForCrosstalkSources_855455_35.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/3c2c280f-b215-4004-96d4-10f3081c1630_isrForCrosstalkSources_855455_35.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e1917d2eb4fc993f7321fc2642fe195 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9a7ebf9a-fff8-4d0f-8245-b56304172dcd_isrForCrosstalkSources_855383_42.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9a7ebf9a-fff8-4d0f-8245-b56304172dcd_isrForCrosstalkSources_855383_42.stderr parsl.dataflow.dflow INFO: Task 1939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e45b9a04a88f19f1ae4f188acba2290 with result from task 1939 parsl.dataflow.dflow DEBUG: Task 1939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/29765ba4-3dbb-4a26-b60a-55381865609a_isrForCrosstalkSources_855402_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/29765ba4-3dbb-4a26-b60a-55381865609a_isrForCrosstalkSources_855402_37.stderr parsl.dataflow.dflow INFO: Task 1871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c688089bdcdae1c016683d7fcb47131c with result from task 1871 parsl.dataflow.dflow DEBUG: Task 1871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3bae6cc4-20d4-4bb4-98a1-4ce92136d249_isrForCrosstalkSources_855432_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3bae6cc4-20d4-4bb4-98a1-4ce92136d249_isrForCrosstalkSources_855432_41.stderr parsl.dataflow.dflow INFO: Task 2051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b866fccce1bc8e927094baeaee8c2295 with result from task 2051 parsl.dataflow.dflow DEBUG: Task 2051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1176dc03-5ab3-4bc3-a18c-e22fc0a95661_isrForCrosstalkSources_855450_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1176dc03-5ab3-4bc3-a18c-e22fc0a95661_isrForCrosstalkSources_855450_22.stderr parsl.dataflow.dflow INFO: Task 1842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8af34aa1f2367e3287af3dd113eb026 with result from task 1842 parsl.dataflow.dflow DEBUG: Task 1842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/19106b9d-249d-4a6d-b127-9a82f066dbe0_isrForCrosstalkSources_855416_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/19106b9d-249d-4a6d-b127-9a82f066dbe0_isrForCrosstalkSources_855416_58.stderr parsl.dataflow.dflow INFO: Task 2020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 975441cf9fc110ad26afc2c8d07286d8 with result from task 2020 parsl.dataflow.dflow DEBUG: Task 2020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d38d1b23-3fb4-460d-8797-6ec7e2170343_isrForCrosstalkSources_855442_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d38d1b23-3fb4-460d-8797-6ec7e2170343_isrForCrosstalkSources_855442_58.stderr parsl.dataflow.dflow INFO: Task 1937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da1c11049e66327377a8f28460c4c609 with result from task 1937 parsl.dataflow.dflow DEBUG: Task 1937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/8ce61220-b620-4364-9847-da9968f9167f_isrForCrosstalkSources_855409_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/8ce61220-b620-4364-9847-da9968f9167f_isrForCrosstalkSources_855409_42.stderr parsl.dataflow.dflow INFO: Task 1977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f6a61fef0fa522c13c38653ab174909 with result from task 1977 parsl.dataflow.dflow DEBUG: Task 1977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/01e9aebf-9640-4ce8-9cec-854de39acb74_isrForCrosstalkSources_855457_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/01e9aebf-9640-4ce8-9cec-854de39acb74_isrForCrosstalkSources_855457_18.stderr parsl.dataflow.dflow INFO: Task 2023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fbd21acadfae3f2b3f82a5990dd123c with result from task 2023 parsl.dataflow.dflow DEBUG: Task 2023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e3b53adc-d253-4ac9-98f1-0af128e8dd66_isrForCrosstalkSources_855416_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e3b53adc-d253-4ac9-98f1-0af128e8dd66_isrForCrosstalkSources_855416_21.stderr parsl.dataflow.dflow INFO: Task 2019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb9d93ad4be2799fe1086073bf2b91ca with result from task 2019 parsl.dataflow.dflow DEBUG: Task 2019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f7b665dc-08ca-4aac-98e9-65f1546dde36_isrForCrosstalkSources_855438_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/f7b665dc-08ca-4aac-98e9-65f1546dde36_isrForCrosstalkSources_855438_54.stderr parsl.dataflow.dflow INFO: Task 2095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3fa4bd094ed2dd1ce14d1ca140ef974 with result from task 2095 parsl.dataflow.dflow DEBUG: Task 2095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/58e73ece-3098-4d39-9c64-35883c2b4bfe_isrForCrosstalkSources_855382_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/58e73ece-3098-4d39-9c64-35883c2b4bfe_isrForCrosstalkSources_855382_14.stderr parsl.dataflow.dflow INFO: Task 2032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99dcd577d353af5935bd53b7b621d447 with result from task 2032 parsl.dataflow.dflow DEBUG: Task 2032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78ef59ed-dbad-42a2-82ea-b29dba0a1efa_isrForCrosstalkSources_855387_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78ef59ed-dbad-42a2-82ea-b29dba0a1efa_isrForCrosstalkSources_855387_3.stderr parsl.dataflow.dflow INFO: Task 2060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95a51e9b04f234fa4a5433027a9dc15e with result from task 2060 parsl.dataflow.dflow DEBUG: Task 2060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/8a5c0463-567c-49e5-8338-15fc6b93e5a0_isrForCrosstalkSources_855403_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/8a5c0463-567c-49e5-8338-15fc6b93e5a0_isrForCrosstalkSources_855403_5.stderr parsl.dataflow.dflow INFO: Task 2076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58ee79c6c44eb8b905fa0ffcfc0af836 with result from task 2076 parsl.dataflow.dflow DEBUG: Task 2076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/343e3f8a-7485-4648-9b07-be90d4d00308_isrForCrosstalkSources_855385_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/343e3f8a-7485-4648-9b07-be90d4d00308_isrForCrosstalkSources_855385_4.stderr parsl.dataflow.dflow INFO: Task 2072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f0eeac64016c7701d4f84f074bfaafb with result from task 2072 parsl.dataflow.dflow DEBUG: Task 2072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/87027ef8-e8f9-4984-ace6-1aa69f70e8f7_isrForCrosstalkSources_855432_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/87027ef8-e8f9-4984-ace6-1aa69f70e8f7_isrForCrosstalkSources_855432_1.stderr parsl.dataflow.dflow INFO: Task 2026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbdaa91fc807c6e97d04e6f43a8878da with result from task 2026 parsl.dataflow.dflow DEBUG: Task 2026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/b1412fa1-6820-468b-a712-76e8370a7f81_isrForCrosstalkSources_855429_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/b1412fa1-6820-468b-a712-76e8370a7f81_isrForCrosstalkSources_855429_51.stderr parsl.dataflow.dflow INFO: Task 2008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03ea40607eafcbd959b94a6072978f72 with result from task 2008 parsl.dataflow.dflow DEBUG: Task 2008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ea1a670-6a2e-45bc-bcfe-bf3f016705a7_isrForCrosstalkSources_855429_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ea1a670-6a2e-45bc-bcfe-bf3f016705a7_isrForCrosstalkSources_855429_18.stderr parsl.dataflow.dflow INFO: Task 2064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bb962bcdc8a29d21fd1ded339916167 with result from task 2064 parsl.dataflow.dflow DEBUG: Task 2064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/014a0e5c-9567-45cb-a4dd-c075bcf7069c_isrForCrosstalkSources_855389_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/014a0e5c-9567-45cb-a4dd-c075bcf7069c_isrForCrosstalkSources_855389_6.stderr parsl.dataflow.dflow INFO: Task 1712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b91368d5954019b7d1da365b15199eb3 with result from task 1712 parsl.dataflow.dflow DEBUG: Task 1712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5cfcf57e-d7e9-4239-a953-4c553d051a29_isrForCrosstalkSources_855413_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5cfcf57e-d7e9-4239-a953-4c553d051a29_isrForCrosstalkSources_855413_52.stderr parsl.dataflow.dflow INFO: Task 1878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adea9fb7481d583cf48ed51aaf5684e9 with result from task 1878 parsl.dataflow.dflow DEBUG: Task 1878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c1e3ddb-47b7-4ca9-8cf6-d084d4db4af1_isrForCrosstalkSources_855435_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c1e3ddb-47b7-4ca9-8cf6-d084d4db4af1_isrForCrosstalkSources_855435_59.stderr parsl.dataflow.dflow INFO: Task 1907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b00780516c5543a99d78ef033cb24eb with result from task 1907 parsl.dataflow.dflow DEBUG: Task 1907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f8419ac3-49a7-4689-8330-a33e380dfc6c_isrForCrosstalkSources_855405_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/f8419ac3-49a7-4689-8330-a33e380dfc6c_isrForCrosstalkSources_855405_57.stderr parsl.dataflow.dflow INFO: Task 1897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c250cf5da4db7ece3fb57f6de7420ae7 with result from task 1897 parsl.dataflow.dflow DEBUG: Task 1897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c45fe67d-f1be-4a6a-a036-530fc7c8dd3d_isrForCrosstalkSources_855393_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c45fe67d-f1be-4a6a-a036-530fc7c8dd3d_isrForCrosstalkSources_855393_44.stderr parsl.dataflow.dflow INFO: Task 2173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea7c70d20d276a6330ef0ed6acc8cb38 with result from task 2173 parsl.dataflow.dflow DEBUG: Task 2173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3576c736-0e40-4ec2-bf88-aa7d01e341ec_isrForCrosstalkSources_855413_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/3576c736-0e40-4ec2-bf88-aa7d01e341ec_isrForCrosstalkSources_855413_22.stderr parsl.dataflow.dflow INFO: Task 1994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c70a28d0b89ea66d9e06cc9c94d4470a with result from task 1994 parsl.dataflow.dflow DEBUG: Task 1994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e78afb0-3815-4d4f-aa98-7299d01d489b_isrForCrosstalkSources_855408_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e78afb0-3815-4d4f-aa98-7299d01d489b_isrForCrosstalkSources_855408_1.stderr parsl.dataflow.dflow INFO: Task 2065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f273482e0fb38cb358ef772c1c06be36 with result from task 2065 parsl.dataflow.dflow DEBUG: Task 2065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d2bd6483-2394-4e86-a0ae-8c09055840a5_isrForCrosstalkSources_855445_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/d2bd6483-2394-4e86-a0ae-8c09055840a5_isrForCrosstalkSources_855445_7.stderr parsl.dataflow.dflow INFO: Task 1679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d360bb1eb1732108eebbae17a2fbbd1c with result from task 1679 parsl.dataflow.dflow DEBUG: Task 1679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/cffaf12a-d6c3-4ca5-818f-f9bcb2154602_isrForCrosstalkSources_855437_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/cffaf12a-d6c3-4ca5-818f-f9bcb2154602_isrForCrosstalkSources_855437_62.stderr parsl.dataflow.dflow INFO: Task 1974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccacad4239eec19dd5ab0e6a81d052d5 with result from task 1974 parsl.dataflow.dflow DEBUG: Task 1974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b1986399-46f8-4043-9a91-b0c8b01f3062_isrForCrosstalkSources_855420_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b1986399-46f8-4043-9a91-b0c8b01f3062_isrForCrosstalkSources_855420_49.stderr parsl.dataflow.dflow INFO: Task 1232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55e06c104b675eb80c69425066d4e4f2 with result from task 1232 parsl.dataflow.dflow DEBUG: Task 1232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a8c1046-6c15-4319-9a47-b8a9deec866c_isrForCrosstalkSources_855393_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1a8c1046-6c15-4319-9a47-b8a9deec866c_isrForCrosstalkSources_855393_22.stderr parsl.dataflow.dflow INFO: Task 1234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16ce97ef6ad4c16978016c6149e29a8 with result from task 1234 parsl.dataflow.dflow DEBUG: Task 1234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/bd4634f3-e64f-4b5e-b664-ac90290ac434_isrForCrosstalkSources_855419_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/bd4634f3-e64f-4b5e-b664-ac90290ac434_isrForCrosstalkSources_855419_51.stderr parsl.dataflow.dflow INFO: Task 1235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d8b8331b231b7282cbfbc2679ffcfdb with result from task 1235 parsl.dataflow.dflow DEBUG: Task 1235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/684a58ff-156b-4ed3-9c35-285da086a8ad_isrForCrosstalkSources_855436_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/684a58ff-156b-4ed3-9c35-285da086a8ad_isrForCrosstalkSources_855436_15.stderr parsl.dataflow.dflow INFO: Task 1228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abada59f25577c20e3bd85de6f03a396 with result from task 1228 parsl.dataflow.dflow DEBUG: Task 1228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/af4953b5-fc74-4f0e-b26f-8d735356a68b_isrForCrosstalkSources_855391_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/af4953b5-fc74-4f0e-b26f-8d735356a68b_isrForCrosstalkSources_855391_45.stderr parsl.dataflow.dflow INFO: Task 1229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b64ca2f54749989b6173dce0df34119 with result from task 1229 parsl.dataflow.dflow DEBUG: Task 1229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e94b448e-74f3-49b9-bebb-590c056f5126_isrForCrosstalkSources_855398_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/e94b448e-74f3-49b9-bebb-590c056f5126_isrForCrosstalkSources_855398_52.stderr parsl.dataflow.dflow INFO: Task 2015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428deed87a56db920b5573310a66085a with result from task 2015 parsl.dataflow.dflow DEBUG: Task 2015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1efc6253-115f-4e72-800c-983c0327c020_isrForCrosstalkSources_855447_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1efc6253-115f-4e72-800c-983c0327c020_isrForCrosstalkSources_855447_44.stderr parsl.dataflow.dflow INFO: Task 1230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 975fae408489f778ba9ad93aca7cde37 with result from task 1230 parsl.dataflow.dflow DEBUG: Task 1230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/37b1ab62-3e07-448f-b869-822d1b1d7aa7_isrForCrosstalkSources_855456_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/37b1ab62-3e07-448f-b869-822d1b1d7aa7_isrForCrosstalkSources_855456_42.stderr parsl.dataflow.dflow INFO: Task 1231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fc060141bfe54b6b73ceebce1c0fca5 with result from task 1231 parsl.dataflow.dflow DEBUG: Task 1231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0e016ba9-3800-4ec4-83f5-88969b772915_isrForCrosstalkSources_855403_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0e016ba9-3800-4ec4-83f5-88969b772915_isrForCrosstalkSources_855403_60.stderr parsl.dataflow.dflow INFO: Task 1912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc56b9a416f3480f22c171e727cc510 with result from task 1912 parsl.dataflow.dflow DEBUG: Task 1912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e2e6a2b6-59f7-4de6-ad5a-91ef1d0341a0_isrForCrosstalkSources_855392_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e2e6a2b6-59f7-4de6-ad5a-91ef1d0341a0_isrForCrosstalkSources_855392_30.stderr parsl.dataflow.dflow INFO: Task 1233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2082e3e90d3ec1407f91f4dd70b06a7 with result from task 1233 parsl.dataflow.dflow DEBUG: Task 1233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7b3ad21f-31a5-47b2-928d-39f05841e196_isrForCrosstalkSources_855439_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/7b3ad21f-31a5-47b2-928d-39f05841e196_isrForCrosstalkSources_855439_53.stderr parsl.dataflow.dflow INFO: Task 1889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9db9a5a2e622cebbae5b03a4040d640 with result from task 1889 parsl.dataflow.dflow DEBUG: Task 1889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a8a2a8e4-a193-4e76-b541-7d4b0df0bc9f_isrForCrosstalkSources_855388_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a8a2a8e4-a193-4e76-b541-7d4b0df0bc9f_isrForCrosstalkSources_855388_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964162 PD 16964161 PD 16964160 PD 16964159 PD 16964158 PD 16964157 PD 16964156 PD 16964155 PD 16964154 PD 16964153 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:21 proc_lsst.multi:146 [INFO] found job 16964111 in provider checkpoint proc_lsst.multi INFO: found job 16964111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d77c3c7a08afd7aefa0147c57925812 with result from task 2192 parsl.dataflow.dflow DEBUG: Task 2192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/79cfa3ab-b197-4a11-a294-3aac68736802_isrForCrosstalkSources_855393_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/79cfa3ab-b197-4a11-a294-3aac68736802_isrForCrosstalkSources_855393_61.stderr parsl.dataflow.dflow INFO: Task 2191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ce762825b0d1a08c4507efafbdc8fd with result from task 2191 parsl.dataflow.dflow DEBUG: Task 2191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/091cb321-d3ff-437b-9ff7-b375342a1cc6_isrForCrosstalkSources_855411_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/091cb321-d3ff-437b-9ff7-b375342a1cc6_isrForCrosstalkSources_855411_45.stderr parsl.dataflow.dflow INFO: Task 2115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6889898b1ad71137cf247d0808155940 with result from task 2115 parsl.dataflow.dflow DEBUG: Task 2115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/892f3446-5c5e-4325-9301-208614979cff_isrForCrosstalkSources_855425_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/892f3446-5c5e-4325-9301-208614979cff_isrForCrosstalkSources_855425_1.stderr parsl.dataflow.dflow INFO: Task 1867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 733065f6fb7a1feae76773f2edf5a0d2 with result from task 1867 parsl.dataflow.dflow DEBUG: Task 1867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0fb5b9c-f733-4cf4-b90b-b1415524481c_isrForCrosstalkSources_855380_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0fb5b9c-f733-4cf4-b90b-b1415524481c_isrForCrosstalkSources_855380_11.stderr parsl.dataflow.dflow INFO: Task 1840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbc286e08be3358a9d613db03c769393 with result from task 1840 parsl.dataflow.dflow DEBUG: Task 1840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/b04b2089-f61b-4002-b8ce-c1dc71ed2f06_isrForCrosstalkSources_855390_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/b04b2089-f61b-4002-b8ce-c1dc71ed2f06_isrForCrosstalkSources_855390_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964162 PD 16964161 PD 16964160 PD 16964159 PD 16964158 PD 16964157 PD 16964156 PD 16964155 PD 16964154 PD 16964153 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:21 proc_lsst.multi:146 [INFO] found job 16964112 in provider checkpoint proc_lsst.multi INFO: found job 16964112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ece644447128fa9a0cef4ab7b17e8d0 with result from task 1918 parsl.dataflow.dflow DEBUG: Task 1918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/86524e44-ef25-46cd-b5dc-d4b510c17ab8_isrForCrosstalkSources_855424_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/86524e44-ef25-46cd-b5dc-d4b510c17ab8_isrForCrosstalkSources_855424_7.stderr parsl.dataflow.dflow INFO: Task 2085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92e66890898663ec85521ad91a42d8d7 with result from task 2085 parsl.dataflow.dflow DEBUG: Task 2085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c447a757-845c-4105-9b78-e0ca27532d34_isrForCrosstalkSources_855458_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c447a757-845c-4105-9b78-e0ca27532d34_isrForCrosstalkSources_855458_9.stderr parsl.dataflow.dflow INFO: Task 2045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d673ff54c1a43f0fd083f0ee32f67b with result from task 2045 parsl.dataflow.dflow DEBUG: Task 2045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/143594e5-c3f8-41fc-9796-034f04d7245f_isrForCrosstalkSources_855436_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/143594e5-c3f8-41fc-9796-034f04d7245f_isrForCrosstalkSources_855436_61.stderr parsl.dataflow.dflow INFO: Task 2204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f045f8ac7a41ddb8bb79d81e876ede50 with result from task 2204 parsl.dataflow.dflow DEBUG: Task 2204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3b453b68-9046-41a3-b49c-eabc3e4babcc_isrForCrosstalkSources_855407_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/3b453b68-9046-41a3-b49c-eabc3e4babcc_isrForCrosstalkSources_855407_31.stderr parsl.dataflow.dflow INFO: Task 2050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5820bdd953c02bf4bafbd08eeda0d789 with result from task 2050 parsl.dataflow.dflow DEBUG: Task 2050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f1eea7b1-2ff9-4239-9a7b-ed3d11932273_isrForCrosstalkSources_855444_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f1eea7b1-2ff9-4239-9a7b-ed3d11932273_isrForCrosstalkSources_855444_47.stderr parsl.dataflow.dflow INFO: Task 1835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fff43b2b0b6b2cc958802f21b02dcc5 with result from task 1835 parsl.dataflow.dflow DEBUG: Task 1835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/4b45e9c9-1d7a-486e-93e1-eadcb8d86b9d_isrForCrosstalkSources_855438_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/4b45e9c9-1d7a-486e-93e1-eadcb8d86b9d_isrForCrosstalkSources_855438_52.stderr parsl.dataflow.dflow INFO: Task 2218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56b8e0496e4dcd6fd7091a1a4813e485 with result from task 2218 parsl.dataflow.dflow DEBUG: Task 2218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/10e42b24-e996-4756-9d65-f35469fafe8a_isrForCrosstalkSources_855387_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/10e42b24-e996-4756-9d65-f35469fafe8a_isrForCrosstalkSources_855387_8.stderr parsl.dataflow.dflow INFO: Task 2013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610e7f214bd9dc4a9595004f64785e27 with result from task 2013 parsl.dataflow.dflow DEBUG: Task 2013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/637cde83-aac4-45ee-bc6a-f57139fa0421_isrForCrosstalkSources_855380_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/637cde83-aac4-45ee-bc6a-f57139fa0421_isrForCrosstalkSources_855380_46.stderr parsl.dataflow.dflow INFO: Task 1908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c268f6aaf1262e5683e15c639f6055 with result from task 1908 parsl.dataflow.dflow DEBUG: Task 1908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e329a1ef-6cbd-4d53-8f87-987f06ebc622_isrForCrosstalkSources_855451_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e329a1ef-6cbd-4d53-8f87-987f06ebc622_isrForCrosstalkSources_855451_8.stderr parsl.dataflow.dflow INFO: Task 1825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3836d52f10e87e86a6ab7ba211c9c69a with result from task 1825 parsl.dataflow.dflow DEBUG: Task 1825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/16253fce-b4b4-42f0-9bd3-993b904c1754_isrForCrosstalkSources_855399_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/16253fce-b4b4-42f0-9bd3-993b904c1754_isrForCrosstalkSources_855399_58.stderr parsl.dataflow.dflow INFO: Task 1745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 903e8cecc619049f44e4791611fe45e0 with result from task 1745 parsl.dataflow.dflow DEBUG: Task 1745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b6a36e57-4821-4c4c-a310-c4293bf3ca6b_isrForCrosstalkSources_855402_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b6a36e57-4821-4c4c-a310-c4293bf3ca6b_isrForCrosstalkSources_855402_61.stderr parsl.dataflow.dflow INFO: Task 2037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72a07e3e34c02c7a8f2715c028407b18 with result from task 2037 parsl.dataflow.dflow DEBUG: Task 2037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/87e64576-c024-491a-89ef-4737e6b2eb3f_isrForCrosstalkSources_855434_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/87e64576-c024-491a-89ef-4737e6b2eb3f_isrForCrosstalkSources_855434_30.stderr parsl.dataflow.dflow INFO: Task 2121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36ec56242dce2b442348a6e69a2096a4 with result from task 2121 parsl.dataflow.dflow DEBUG: Task 2121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/51163d5b-4f7a-48d7-abf7-e5d1acf06f9e_isrForCrosstalkSources_855434_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/51163d5b-4f7a-48d7-abf7-e5d1acf06f9e_isrForCrosstalkSources_855434_8.stderr parsl.dataflow.dflow INFO: Task 2022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d109bb031051d07ea74b0dcd6d6bb62 with result from task 2022 parsl.dataflow.dflow DEBUG: Task 2022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4efe991c-5ed3-45a8-bdcc-b11cea328692_isrForCrosstalkSources_855383_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4efe991c-5ed3-45a8-bdcc-b11cea328692_isrForCrosstalkSources_855383_22.stderr parsl.dataflow.dflow INFO: Task 1990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4c0d614a6781806fb43d915b9e2ad7d with result from task 1990 parsl.dataflow.dflow DEBUG: Task 1990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/75e2ffb3-6754-436f-9f4b-db03ab2e175b_isrForCrosstalkSources_855440_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/75e2ffb3-6754-436f-9f4b-db03ab2e175b_isrForCrosstalkSources_855440_30.stderr parsl.dataflow.dflow INFO: Task 1989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d4a367932c4bc03e315a0cef96de3d0 with result from task 1989 parsl.dataflow.dflow DEBUG: Task 1989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c2be462b-ebff-4e7e-a94e-c053a2a43ecd_isrForCrosstalkSources_855410_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c2be462b-ebff-4e7e-a94e-c053a2a43ecd_isrForCrosstalkSources_855410_50.stderr parsl.dataflow.dflow INFO: Task 1905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 870b1729864e53a99a7bd58a5ad5047a with result from task 1905 parsl.dataflow.dflow DEBUG: Task 1905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8b4990ff-537f-4da1-99fe-a293dc3243bf_isrForCrosstalkSources_855392_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8b4990ff-537f-4da1-99fe-a293dc3243bf_isrForCrosstalkSources_855392_51.stderr parsl.dataflow.dflow INFO: Task 1870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aed7fab241ef8e6717de7283bcd0b50 with result from task 1870 parsl.dataflow.dflow DEBUG: Task 1870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/451a3e40-0c64-410a-9639-7923d8aaa797_isrForCrosstalkSources_855418_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/451a3e40-0c64-410a-9639-7923d8aaa797_isrForCrosstalkSources_855418_36.stderr parsl.dataflow.dflow INFO: Task 2227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23805fe21f975f8221486f5f24d61292 with result from task 2227 parsl.dataflow.dflow DEBUG: Task 2227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7b032554-e377-4419-bbed-ca13b7c4989c_isrForCrosstalkSources_855445_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/7b032554-e377-4419-bbed-ca13b7c4989c_isrForCrosstalkSources_855445_39.stderr parsl.dataflow.dflow INFO: Task 1904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ff41c01e58cbeae8d39f0070871ad81 with result from task 1904 parsl.dataflow.dflow DEBUG: Task 1904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b3f07334-6780-461b-b266-3492f1abffdc_isrForCrosstalkSources_855421_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/b3f07334-6780-461b-b266-3492f1abffdc_isrForCrosstalkSources_855421_46.stderr parsl.dataflow.dflow INFO: Task 1760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4b0f0efb6917a3187fd54e49f197042 with result from task 1760 parsl.dataflow.dflow DEBUG: Task 1760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ae52cbac-66e1-49d9-923b-6bb40a62de5a_isrForCrosstalkSources_855449_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ae52cbac-66e1-49d9-923b-6bb40a62de5a_isrForCrosstalkSources_855449_53.stderr parsl.dataflow.dflow INFO: Task 1890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef624d439363af105bc68f4cdd4fd4f0 with result from task 1890 parsl.dataflow.dflow DEBUG: Task 1890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/09c368f0-2470-4d94-9428-e8e044ff43e6_isrForCrosstalkSources_855407_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/09c368f0-2470-4d94-9428-e8e044ff43e6_isrForCrosstalkSources_855407_59.stderr parsl.dataflow.dflow INFO: Task 1964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7fb64041e9c80e957d5af748ab318f with result from task 1964 parsl.dataflow.dflow DEBUG: Task 1964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/32d12d65-9cfd-4173-8962-1ab1bac4a6cd_isrForCrosstalkSources_855444_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/32d12d65-9cfd-4173-8962-1ab1bac4a6cd_isrForCrosstalkSources_855444_25.stderr parsl.dataflow.dflow INFO: Task 1973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57d44c140458e6e57941f2504b166804 with result from task 1973 parsl.dataflow.dflow DEBUG: Task 1973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/db595576-d727-46f6-95b0-ccf44e84fabf_isrForCrosstalkSources_855416_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/db595576-d727-46f6-95b0-ccf44e84fabf_isrForCrosstalkSources_855416_24.stderr parsl.dataflow.dflow INFO: Task 2079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16984164ee7ba7d7b2a2c167b555d362 with result from task 2079 parsl.dataflow.dflow DEBUG: Task 2079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/9e224c68-dbe2-479b-b916-33054328f857_isrForCrosstalkSources_855390_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/9e224c68-dbe2-479b-b916-33054328f857_isrForCrosstalkSources_855390_14.stderr parsl.dataflow.dflow INFO: Task 2035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9635fe8a5eb7400ff682f664e1992405 with result from task 2035 parsl.dataflow.dflow DEBUG: Task 2035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df5c5641-200d-46aa-917a-f4fd47c77ea7_isrForCrosstalkSources_855422_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df5c5641-200d-46aa-917a-f4fd47c77ea7_isrForCrosstalkSources_855422_54.stderr parsl.dataflow.dflow INFO: Task 2213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34e60009b39e1c2685c725e032b54245 with result from task 2213 parsl.dataflow.dflow DEBUG: Task 2213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5eff4b7f-bc69-4a32-9cef-71fd542d678e_isrForCrosstalkSources_855449_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5eff4b7f-bc69-4a32-9cef-71fd542d678e_isrForCrosstalkSources_855449_28.stderr parsl.dataflow.dflow INFO: Task 1950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f789e522b82763cd457d1757ee72f02f with result from task 1950 parsl.dataflow.dflow DEBUG: Task 1950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e6dcff97-45ff-400b-88ed-f93754e23183_isrForCrosstalkSources_855447_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/e6dcff97-45ff-400b-88ed-f93754e23183_isrForCrosstalkSources_855447_54.stderr parsl.dataflow.dflow INFO: Task 2133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06c62fc21e78ce19e85ba90d6cc4b919 with result from task 2133 parsl.dataflow.dflow DEBUG: Task 2133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/bd204bf0-6e2d-448c-be95-8a922e401250_isrForCrosstalkSources_855425_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/bd204bf0-6e2d-448c-be95-8a922e401250_isrForCrosstalkSources_855425_62.stderr parsl.dataflow.dflow INFO: Task 2012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63cb3ce130aed94d4d4c92c7fb4aef2a with result from task 2012 parsl.dataflow.dflow DEBUG: Task 2012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a879fe82-2669-42c9-bf10-e470c1909d33_isrForCrosstalkSources_855455_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a879fe82-2669-42c9-bf10-e470c1909d33_isrForCrosstalkSources_855455_28.stderr parsl.dataflow.dflow INFO: Task 1931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 754dc4d0e0fdd48ffc74748c2875e704 with result from task 1931 parsl.dataflow.dflow DEBUG: Task 1931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a61d5f90-15b9-4325-a461-bd45cf7e984d_isrForCrosstalkSources_855408_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a61d5f90-15b9-4325-a461-bd45cf7e984d_isrForCrosstalkSources_855408_32.stderr parsl.dataflow.dflow INFO: Task 2233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 513e372b409bcb2f5fa9208f6e2c4695 with result from task 2233 parsl.dataflow.dflow DEBUG: Task 2233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6643d875-685d-455b-8a8e-7468dca90ea7_isrForCrosstalkSources_855404_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6643d875-685d-455b-8a8e-7468dca90ea7_isrForCrosstalkSources_855404_15.stderr parsl.dataflow.dflow INFO: Task 2034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d853fa5b06b0062212c3e95acf653b3f with result from task 2034 parsl.dataflow.dflow DEBUG: Task 2034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9f9e479-ebf5-424c-9998-943d98a921a0_isrForCrosstalkSources_855421_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a9f9e479-ebf5-424c-9998-943d98a921a0_isrForCrosstalkSources_855421_13.stderr parsl.dataflow.dflow INFO: Task 1968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcb78f65808c370597ed5213cb8aea3c with result from task 1968 parsl.dataflow.dflow DEBUG: Task 1968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/23e08001-f68b-487b-beb2-c32e9af3eea8_isrForCrosstalkSources_855424_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/23e08001-f68b-487b-beb2-c32e9af3eea8_isrForCrosstalkSources_855424_33.stderr parsl.dataflow.dflow INFO: Task 1992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eedd02a0fe00a530abcef8f64abf3b00 with result from task 1992 parsl.dataflow.dflow DEBUG: Task 1992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6f7778e2-edbc-437e-8f5f-73ec8ede16f6_isrForCrosstalkSources_855431_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6f7778e2-edbc-437e-8f5f-73ec8ede16f6_isrForCrosstalkSources_855431_24.stderr parsl.dataflow.dflow INFO: Task 2061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a850c83d5ac4601f2da4e133bd359f83 with result from task 2061 parsl.dataflow.dflow DEBUG: Task 2061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/075ecc01-19a6-423b-b39c-da83ae682311_isrForCrosstalkSources_855455_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/075ecc01-19a6-423b-b39c-da83ae682311_isrForCrosstalkSources_855455_41.stderr parsl.dataflow.dflow INFO: Task 1961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7b29b3b6d8c3d0fb42e6f5c9f32597a with result from task 1961 parsl.dataflow.dflow DEBUG: Task 1961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dde268da-04f5-43ce-b303-ff26d9fb13ba_isrForCrosstalkSources_855456_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dde268da-04f5-43ce-b303-ff26d9fb13ba_isrForCrosstalkSources_855456_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964125 R 16964126 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:23 proc_lsst.multi:146 [INFO] found job 16964113 in provider checkpoint proc_lsst.multi INFO: found job 16964113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ef3188a80d6aea267d7964a50f6c601 with result from task 1999 parsl.dataflow.dflow DEBUG: Task 1999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7d6b8a24-a7cb-4f19-aa6f-835efe763a5f_isrForCrosstalkSources_855449_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7d6b8a24-a7cb-4f19-aa6f-835efe763a5f_isrForCrosstalkSources_855449_34.stderr parsl.dataflow.dflow INFO: Task 2108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43eb4e1ddfff7628296a3bb2831a0eb8 with result from task 2108 parsl.dataflow.dflow DEBUG: Task 2108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5843f953-649c-4709-aee3-8b3a8c4ffdc2_isrForCrosstalkSources_855436_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5843f953-649c-4709-aee3-8b3a8c4ffdc2_isrForCrosstalkSources_855436_38.stderr parsl.dataflow.dflow INFO: Task 1769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2b7fe819b5f906a88247b99f6be2526 with result from task 1769 parsl.dataflow.dflow DEBUG: Task 1769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d56a0fea-a2ba-4850-a1d4-1f974fcb6ad6_isrForCrosstalkSources_855444_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d56a0fea-a2ba-4850-a1d4-1f974fcb6ad6_isrForCrosstalkSources_855444_62.stderr parsl.dataflow.dflow INFO: Task 1975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 708647a803c9fa56086238256ed21dba with result from task 1975 parsl.dataflow.dflow DEBUG: Task 1975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d90f960-bd89-4928-9d57-d9edaa64e95f_isrForCrosstalkSources_855459_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d90f960-bd89-4928-9d57-d9edaa64e95f_isrForCrosstalkSources_855459_36.stderr parsl.dataflow.dflow INFO: Task 2055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5836349961fcddb350ace374bd83915 with result from task 2055 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b77ef49a-d761-439f-9644-6e797e5197ba_isrForCrosstalkSources_855440_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b77ef49a-d761-439f-9644-6e797e5197ba_isrForCrosstalkSources_855440_49.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:24 proc_lsst.multi:146 [INFO] found job 16964114 in provider checkpoint proc_lsst.multi INFO: found job 16964114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d3b06c7565a77f82de68e6378dced69 with result from task 2216 parsl.dataflow.dflow DEBUG: Task 2216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eca3d4c1-1dc4-44c4-b08b-e958a70392dd_isrForCrosstalkSources_855381_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/eca3d4c1-1dc4-44c4-b08b-e958a70392dd_isrForCrosstalkSources_855381_26.stderr parsl.dataflow.dflow INFO: Task 1983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03730bef1c96def8b3befe1fbd30dc9 with result from task 1983 parsl.dataflow.dflow DEBUG: Task 1983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e672a244-2f45-460e-800a-1890050a0227_isrForCrosstalkSources_855451_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e672a244-2f45-460e-800a-1890050a0227_isrForCrosstalkSources_855451_50.stderr parsl.dataflow.dflow INFO: Task 1987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c419c3f4c9b4cf1f881dc970029efc3 with result from task 1987 parsl.dataflow.dflow DEBUG: Task 1987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fcd71c4a-3355-46da-849e-07fd561ffbdc_isrForCrosstalkSources_855412_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fcd71c4a-3355-46da-849e-07fd561ffbdc_isrForCrosstalkSources_855412_3.stderr parsl.dataflow.dflow INFO: Task 1906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79bfc861033631e5be4dd5a7268718a2 with result from task 1906 parsl.dataflow.dflow DEBUG: Task 1906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/33eb5894-e6a4-45ba-927a-e2aa9a25830d_isrForCrosstalkSources_855386_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/33eb5894-e6a4-45ba-927a-e2aa9a25830d_isrForCrosstalkSources_855386_60.stderr parsl.dataflow.dflow INFO: Task 1967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35c11c795f9d6a10448af567edab7128 with result from task 1967 parsl.dataflow.dflow DEBUG: Task 1967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/286d3e28-9b38-4a27-a9cb-9a9cc4b2e136_isrForCrosstalkSources_855388_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/286d3e28-9b38-4a27-a9cb-9a9cc4b2e136_isrForCrosstalkSources_855388_40.stderr parsl.dataflow.dflow INFO: Task 2043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c4e60f667ebc02ff2887e91a4f040f5 with result from task 2043 parsl.dataflow.dflow DEBUG: Task 2043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/fa7902bf-f75d-4df7-a71f-f5693b424445_isrForCrosstalkSources_855436_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/fa7902bf-f75d-4df7-a71f-f5693b424445_isrForCrosstalkSources_855436_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:24 proc_lsst.multi:146 [INFO] found job 16964115 in provider checkpoint proc_lsst.multi INFO: found job 16964115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 429de8704ffe33634e270d28d5a8334d with result from task 1925 parsl.dataflow.dflow DEBUG: Task 1925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9eb1faf3-0daf-4faa-886c-59d78f33b402_isrForCrosstalkSources_855382_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9eb1faf3-0daf-4faa-886c-59d78f33b402_isrForCrosstalkSources_855382_62.stderr parsl.dataflow.dflow INFO: Task 2089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 443e23d6f97ee6b4ad9ab522b5f204f4 with result from task 2089 parsl.dataflow.dflow DEBUG: Task 2089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d79eb5fb-bf83-47af-bc42-87357c0af806_isrForCrosstalkSources_855415_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/d79eb5fb-bf83-47af-bc42-87357c0af806_isrForCrosstalkSources_855415_46.stderr parsl.dataflow.dflow INFO: Task 1914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc79ad30e237e723ae591f255f8b3dc with result from task 1914 parsl.dataflow.dflow DEBUG: Task 1914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b2e283e1-eddf-4f82-8fac-44d98ee934e4_isrForCrosstalkSources_855402_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b2e283e1-eddf-4f82-8fac-44d98ee934e4_isrForCrosstalkSources_855402_31.stderr parsl.dataflow.dflow INFO: Task 2053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5468a3889783197f9d20146d18d1e49 with result from task 2053 parsl.dataflow.dflow DEBUG: Task 2053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b9fbccd2-c4d5-4be8-a5df-ace971615556_isrForCrosstalkSources_855438_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b9fbccd2-c4d5-4be8-a5df-ace971615556_isrForCrosstalkSources_855438_20.stderr parsl.dataflow.dflow INFO: Task 2058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 435c153b64416b938ec396a07fb84e48 with result from task 2058 parsl.dataflow.dflow DEBUG: Task 2058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/def14c90-f076-4901-870b-1198836beca6_isrForCrosstalkSources_855457_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/def14c90-f076-4901-870b-1198836beca6_isrForCrosstalkSources_855457_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:25 proc_lsst.multi:146 [INFO] found job 16964116 in provider checkpoint proc_lsst.multi INFO: found job 16964116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1513acd87f2d8f4a899284627cd407c with result from task 1896 parsl.dataflow.dflow DEBUG: Task 1896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/98a9e8b0-fe15-4110-9ccb-54dc63d22816_isrForCrosstalkSources_855443_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/98a9e8b0-fe15-4110-9ccb-54dc63d22816_isrForCrosstalkSources_855443_58.stderr parsl.dataflow.dflow INFO: Task 2073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2d73d26a7c6dd9249377a54679f9111 with result from task 2073 parsl.dataflow.dflow DEBUG: Task 2073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/af61e1ab-60d6-4f88-9da1-9aa39dc938d3_isrForCrosstalkSources_855438_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/af61e1ab-60d6-4f88-9da1-9aa39dc938d3_isrForCrosstalkSources_855438_13.stderr parsl.dataflow.dflow INFO: Task 2044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d98014ed561155266f91b95e56983866 with result from task 2044 parsl.dataflow.dflow DEBUG: Task 2044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/bc638559-8c7e-45d0-ba24-da0c63588592_isrForCrosstalkSources_855393_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/bc638559-8c7e-45d0-ba24-da0c63588592_isrForCrosstalkSources_855393_45.stderr parsl.dataflow.dflow INFO: Task 1981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdeba43162ca2ff2ca2e08a79b15069d with result from task 1981 parsl.dataflow.dflow DEBUG: Task 1981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/94569c96-1225-4f42-aa66-5dd75579b88f_isrForCrosstalkSources_855385_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/94569c96-1225-4f42-aa66-5dd75579b88f_isrForCrosstalkSources_855385_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:25 proc_lsst.multi:146 [INFO] found job 16964117 in provider checkpoint proc_lsst.multi INFO: found job 16964117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a3af129bb9ff90219060891a21c7bb0 with result from task 1859 parsl.dataflow.dflow DEBUG: Task 1859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ef162b1-547d-4e84-8265-9c7d2245bb79_isrForCrosstalkSources_855379_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ef162b1-547d-4e84-8265-9c7d2245bb79_isrForCrosstalkSources_855379_36.stderr parsl.dataflow.dflow INFO: Task 2067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91af3ef6db4b540af4b16b800f386e2 with result from task 2067 parsl.dataflow.dflow DEBUG: Task 2067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/14626533-51a2-4964-8335-85b142cde9ef_isrForCrosstalkSources_855458_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/14626533-51a2-4964-8335-85b142cde9ef_isrForCrosstalkSources_855458_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:25 proc_lsst.multi:146 [INFO] found job 16964118 in provider checkpoint proc_lsst.multi INFO: found job 16964118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6be75777e8aad9e2e43867377cb1b79b with result from task 2190 parsl.dataflow.dflow DEBUG: Task 2190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1caaf1cc-679e-436a-95c1-322b4fb09d48_isrForCrosstalkSources_855454_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1caaf1cc-679e-436a-95c1-322b4fb09d48_isrForCrosstalkSources_855454_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:26 proc_lsst.multi:146 [INFO] found job 16964119 in provider checkpoint proc_lsst.multi INFO: found job 16964119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e6810112e511ffccecb2ee8b11614d7 with result from task 1960 parsl.dataflow.dflow DEBUG: Task 1960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9b1d0284-62f1-4108-87cc-3325d15f29f2_isrForCrosstalkSources_855427_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9b1d0284-62f1-4108-87cc-3325d15f29f2_isrForCrosstalkSources_855427_23.stderr parsl.dataflow.dflow INFO: Task 2001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1941404f65be821f3038aabab301782e with result from task 2001 parsl.dataflow.dflow DEBUG: Task 2001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/2986b154-ee66-4c68-94db-8ffe39f0cfff_isrForCrosstalkSources_855458_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/2986b154-ee66-4c68-94db-8ffe39f0cfff_isrForCrosstalkSources_855458_13.stderr parsl.dataflow.dflow INFO: Task 1898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7057ca4392567a00bcc549310125feca with result from task 1898 parsl.dataflow.dflow DEBUG: Task 1898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a692f81-41e6-4970-bc54-884499c36532_isrForCrosstalkSources_855379_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a692f81-41e6-4970-bc54-884499c36532_isrForCrosstalkSources_855379_16.stderr parsl.dataflow.dflow INFO: Task 1984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2447f7525e8ae69ecf162183fd6b9895 with result from task 1984 parsl.dataflow.dflow DEBUG: Task 1984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/534f5b42-d310-4d58-afd2-9c9957cb47fb_isrForCrosstalkSources_855454_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/534f5b42-d310-4d58-afd2-9c9957cb47fb_isrForCrosstalkSources_855454_47.stderr parsl.dataflow.dflow INFO: Task 2010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d93ab9107e27425bcae0c28282820290 with result from task 2010 parsl.dataflow.dflow DEBUG: Task 2010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3dfe20a0-263a-4be8-96d2-8d9c665c544b_isrForCrosstalkSources_855416_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3dfe20a0-263a-4be8-96d2-8d9c665c544b_isrForCrosstalkSources_855416_42.stderr parsl.dataflow.dflow INFO: Task 1956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b47152d26bbd627e22de0585aa5675c1 with result from task 1956 parsl.dataflow.dflow DEBUG: Task 1956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8c35a59b-b7d9-4998-90f8-50be2ab1e551_isrForCrosstalkSources_855391_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/8c35a59b-b7d9-4998-90f8-50be2ab1e551_isrForCrosstalkSources_855391_20.stderr parsl.dataflow.dflow INFO: Task 1934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e846602ca252a9a168d0794d7ecd2001 with result from task 1934 parsl.dataflow.dflow DEBUG: Task 1934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/850245b8-8240-44b8-9efe-4b45563a8bcf_isrForCrosstalkSources_855436_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/850245b8-8240-44b8-9efe-4b45563a8bcf_isrForCrosstalkSources_855436_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:26 proc_lsst.multi:146 [INFO] found job 16964120 in provider checkpoint proc_lsst.multi INFO: found job 16964120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4eb912c1cca531813a347ebb58999a2 with result from task 2229 parsl.dataflow.dflow DEBUG: Task 2229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d0ffd2cf-6baa-411a-8373-f6ed9c2a3088_isrForCrosstalkSources_855433_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/d0ffd2cf-6baa-411a-8373-f6ed9c2a3088_isrForCrosstalkSources_855433_32.stderr parsl.dataflow.dflow INFO: Task 2059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 974f28ee28da6b7c96bf53a180353550 with result from task 2059 parsl.dataflow.dflow DEBUG: Task 2059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/51a61add-6fa8-4af3-a98f-5dc5db8e5d3c_isrForCrosstalkSources_855410_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/51a61add-6fa8-4af3-a98f-5dc5db8e5d3c_isrForCrosstalkSources_855410_62.stderr parsl.dataflow.dflow INFO: Task 2088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e126ec4c6be1aa192ea8ff2fe8090e97 with result from task 2088 parsl.dataflow.dflow DEBUG: Task 2088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d869b044-07a7-485b-9947-42d6436972a7_isrForCrosstalkSources_855457_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d869b044-07a7-485b-9947-42d6436972a7_isrForCrosstalkSources_855457_19.stderr parsl.dataflow.dflow INFO: Task 2025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a765c033999da806e24a1905ef507ac6 with result from task 2025 parsl.dataflow.dflow DEBUG: Task 2025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a74c2d4-bc4c-49e1-9f77-07ef58a9e4ef_isrForCrosstalkSources_855447_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a74c2d4-bc4c-49e1-9f77-07ef58a9e4ef_isrForCrosstalkSources_855447_5.stderr parsl.dataflow.dflow INFO: Task 2056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99ae4887f68d13d0884578e5f8a3f643 with result from task 2056 parsl.dataflow.dflow DEBUG: Task 2056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/05a93d35-7f63-447b-b2c9-628fa66a1105_isrForCrosstalkSources_855437_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/05a93d35-7f63-447b-b2c9-628fa66a1105_isrForCrosstalkSources_855437_58.stderr parsl.dataflow.dflow INFO: Task 2206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d631dfe9e9677ee20b45484dcc23ee66 with result from task 2206 parsl.dataflow.dflow DEBUG: Task 2206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/478a21d7-3b95-40c6-aaf5-7cf86cc932b9_isrForCrosstalkSources_855422_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/478a21d7-3b95-40c6-aaf5-7cf86cc932b9_isrForCrosstalkSources_855422_36.stderr parsl.dataflow.dflow INFO: Task 2028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aec05c4091a6e402df166092917a1a0 with result from task 2028 parsl.dataflow.dflow DEBUG: Task 2028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/59537ff1-7b88-4e0f-b4d6-d0cfaf174381_isrForCrosstalkSources_855389_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/59537ff1-7b88-4e0f-b4d6-d0cfaf174381_isrForCrosstalkSources_855389_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:27 proc_lsst.multi:146 [INFO] found job 16964121 in provider checkpoint proc_lsst.multi INFO: found job 16964121 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9f924fb5512b8e19d0e471140c0af8 with result from task 2138 parsl.dataflow.dflow DEBUG: Task 2138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bfeb9968-9c61-4a74-86f3-869c5d90e325_isrForCrosstalkSources_855386_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bfeb9968-9c61-4a74-86f3-869c5d90e325_isrForCrosstalkSources_855386_32.stderr parsl.dataflow.dflow INFO: Task 2091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69f3eff892c9e87c6d8b663f1d3e5953 with result from task 2091 parsl.dataflow.dflow DEBUG: Task 2091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/68d113d0-cdca-45c3-b799-ce2281e48795_isrForCrosstalkSources_855435_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/68d113d0-cdca-45c3-b799-ce2281e48795_isrForCrosstalkSources_855435_41.stderr parsl.dataflow.dflow INFO: Task 1926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 808202056f1674cde653c29894bfdae0 with result from task 1926 parsl.dataflow.dflow DEBUG: Task 1926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2264d4ab-f993-4b31-abd4-e90fbb39d7f9_isrForCrosstalkSources_855404_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2264d4ab-f993-4b31-abd4-e90fbb39d7f9_isrForCrosstalkSources_855404_48.stderr parsl.dataflow.dflow INFO: Task 1943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a92bf2020c618f02c886d42ac5444bde with result from task 1943 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:27 proc_lsst.multi:146 [INFO] found job 16964122 in provider checkpoint proc_lsst.multi INFO: found job 16964122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 1943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a129c6ed-7fa5-494d-84f3-8d42ea706761_isrForCrosstalkSources_855384_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/a129c6ed-7fa5-494d-84f3-8d42ea706761_isrForCrosstalkSources_855384_4.stderr parsl.dataflow.dflow INFO: Task 1929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b98f96c7f9bb8d57e06a2dc7fea1707 with result from task 1929 parsl.dataflow.dflow DEBUG: Task 1929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/65041afa-92e9-4393-b94e-b3e3c04f94df_isrForCrosstalkSources_855422_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/65041afa-92e9-4393-b94e-b3e3c04f94df_isrForCrosstalkSources_855422_8.stderr parsl.dataflow.dflow INFO: Task 2162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58208bfbbe4425f3b40fb69fc7c8f67 with result from task 2162 parsl.dataflow.dflow DEBUG: Task 2162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/807abd61-00e7-4437-bfe1-f9930bef95ce_isrForCrosstalkSources_855431_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/807abd61-00e7-4437-bfe1-f9930bef95ce_isrForCrosstalkSources_855431_20.stderr parsl.dataflow.dflow INFO: Task 2017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81162bf8c741810602906b4bc2144cfd with result from task 2017 parsl.dataflow.dflow DEBUG: Task 2017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d231f72c-75a7-4acc-b85c-3a6e3bc5dd3d_isrForCrosstalkSources_855428_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d231f72c-75a7-4acc-b85c-3a6e3bc5dd3d_isrForCrosstalkSources_855428_23.stderr parsl.dataflow.dflow INFO: Task 2136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4942778b2948bfd5d4d26ec2352e14d with result from task 2136 parsl.dataflow.dflow DEBUG: Task 2136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6d0e94f9-8354-4683-9438-dc9d3b8102bc_isrForCrosstalkSources_855408_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/6d0e94f9-8354-4683-9438-dc9d3b8102bc_isrForCrosstalkSources_855408_45.stderr parsl.dataflow.dflow INFO: Task 2031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4521763c45bc16f34cf2d88b9204ef with result from task 2031 parsl.dataflow.dflow DEBUG: Task 2031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/815a8a35-cfe3-42e6-a071-c63693401cd0_isrForCrosstalkSources_855394_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/815a8a35-cfe3-42e6-a071-c63693401cd0_isrForCrosstalkSources_855394_20.stderr parsl.dataflow.dflow INFO: Task 2251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2baa66d9be46895a505cebe1fb40a8f5 with result from task 2251 parsl.dataflow.dflow DEBUG: Task 2251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9bb318d3-bc92-4031-823d-7c168a97b85d_isrForCrosstalkSources_855383_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9bb318d3-bc92-4031-823d-7c168a97b85d_isrForCrosstalkSources_855383_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:28 proc_lsst.multi:146 [INFO] found job 16964123 in provider checkpoint proc_lsst.multi INFO: found job 16964123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7216daa81a1564c6757e9ee24796a3e9 with result from task 2234 parsl.dataflow.dflow DEBUG: Task 2234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ea36501b-ed1d-41a8-925d-7b151b99ac2a_isrForCrosstalkSources_855404_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ea36501b-ed1d-41a8-925d-7b151b99ac2a_isrForCrosstalkSources_855404_18.stderr parsl.dataflow.dflow INFO: Task 2101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a5e3cf50e7f3ab1c3fc8f79ba0c99de with result from task 2101 parsl.dataflow.dflow DEBUG: Task 2101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2df00e72-094c-4b5d-b60c-491be948dafd_isrForCrosstalkSources_855443_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/2df00e72-094c-4b5d-b60c-491be948dafd_isrForCrosstalkSources_855443_4.stderr parsl.dataflow.dflow INFO: Task 2210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eadb57ae329d64cbfac486d92abec3d with result from task 2210 parsl.dataflow.dflow DEBUG: Task 2210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5074cfc3-f6c3-44f9-a655-af1fab12913f_isrForCrosstalkSources_855384_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5074cfc3-f6c3-44f9-a655-af1fab12913f_isrForCrosstalkSources_855384_24.stderr parsl.dataflow.dflow INFO: Task 1852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f64f7c8730f35bfc9136748a055dfce9 with result from task 1852 parsl.dataflow.dflow DEBUG: Task 1852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/e6de8865-2b78-40d2-af65-cfe4c52da8b8_isrForCrosstalkSources_855415_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/e6de8865-2b78-40d2-af65-cfe4c52da8b8_isrForCrosstalkSources_855415_11.stderr parsl.dataflow.dflow INFO: Task 2182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bfe460e55c3303ae51e8b11a9d049a9 with result from task 2182 parsl.dataflow.dflow DEBUG: Task 2182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/33e42e57-9065-446e-801b-58681d3594df_isrForCrosstalkSources_855433_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/33e42e57-9065-446e-801b-58681d3594df_isrForCrosstalkSources_855433_6.stderr parsl.dataflow.dflow INFO: Task 2161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ff1a73d38eeda8fcf5f7bd6ae574658 with result from task 2161 parsl.dataflow.dflow DEBUG: Task 2161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35119359-d300-4283-b9b8-f0e94cd1a376_isrForCrosstalkSources_855409_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/35119359-d300-4283-b9b8-f0e94cd1a376_isrForCrosstalkSources_855409_5.stderr parsl.dataflow.dflow INFO: Task 1909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ff2f7ca27985a79f12348c0b22f5488 with result from task 1909 parsl.dataflow.dflow DEBUG: Task 1909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1a135b36-98b9-4092-b243-296e39482c65_isrForCrosstalkSources_855439_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1a135b36-98b9-4092-b243-296e39482c65_isrForCrosstalkSources_855439_55.stderr parsl.dataflow.dflow INFO: Task 1884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76f5cda087c621a211beafe8291462a5 with result from task 1884 parsl.dataflow.dflow DEBUG: Task 1884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6233f9e1-ebd6-478e-8692-2dcf34aff3dc_isrForCrosstalkSources_855440_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/6233f9e1-ebd6-478e-8692-2dcf34aff3dc_isrForCrosstalkSources_855440_58.stderr parsl.dataflow.dflow INFO: Task 2248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b896bb7a002d94eb5718d4ea95cf23cd with result from task 2248 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c8005905-f0d4-4bab-9aec-8af67af05c6f_isrForCrosstalkSources_855438_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/c8005905-f0d4-4bab-9aec-8af67af05c6f_isrForCrosstalkSources_855438_16.stderr parsl.dataflow.dflow INFO: Task 2185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68f9d88a3cc5441f08d66e418c9b979b with result from task 2185 parsl.dataflow.dflow DEBUG: Task 2185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/780227a6-056c-421c-ad36-1389ac85a770_isrForCrosstalkSources_855386_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/780227a6-056c-421c-ad36-1389ac85a770_isrForCrosstalkSources_855386_46.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:28 proc_lsst.multi:146 [INFO] found job 16964124 in provider checkpoint proc_lsst.multi INFO: found job 16964124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b23116d579bd1f5b67db5411ced228cf with result from task 1982 parsl.dataflow.dflow DEBUG: Task 1982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10911095-9f38-4e8c-85a2-01dc73263e6b_isrForCrosstalkSources_855412_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/10911095-9f38-4e8c-85a2-01dc73263e6b_isrForCrosstalkSources_855412_7.stderr parsl.dataflow.dflow INFO: Task 2113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30bd97355db82dc92a9deafe670b064a with result from task 2113 parsl.dataflow.dflow DEBUG: Task 2113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f6f118f6-52f9-42dc-b366-0202ecff9f1d_isrForCrosstalkSources_855399_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/f6f118f6-52f9-42dc-b366-0202ecff9f1d_isrForCrosstalkSources_855399_16.stderr parsl.dataflow.dflow INFO: Task 2174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfd9e5e95e3005352a7e841ee110ca8f with result from task 2174 parsl.dataflow.dflow DEBUG: Task 2174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e837e086-91f3-4b86-a9cd-a2ca43e75baf_isrForCrosstalkSources_855453_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/e837e086-91f3-4b86-a9cd-a2ca43e75baf_isrForCrosstalkSources_855453_36.stderr parsl.dataflow.dflow INFO: Task 1997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85992ebbedc5c4f48671b284f92c07f with result from task 1997 parsl.dataflow.dflow DEBUG: Task 1997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5aa24a98-c1fa-4fcf-b8e0-c09b0d1db49b_isrForCrosstalkSources_855425_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5aa24a98-c1fa-4fcf-b8e0-c09b0d1db49b_isrForCrosstalkSources_855425_53.stderr parsl.dataflow.dflow INFO: Task 2159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffa21a1c4e23747fca89c90e8d87ed29 with result from task 2159 parsl.dataflow.dflow DEBUG: Task 2159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/04e3e035-6c1c-477a-bf64-ebf78ec77bb9_isrForCrosstalkSources_855425_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/04e3e035-6c1c-477a-bf64-ebf78ec77bb9_isrForCrosstalkSources_855425_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:29 proc_lsst.multi:146 [INFO] found job 16964125 in provider checkpoint proc_lsst.multi INFO: found job 16964125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d7f0bbea69ad38b598366886e15fa07 with result from task 1938 parsl.dataflow.dflow DEBUG: Task 1938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e126bb40-ead6-4ad8-954a-d3dfc0165acd_isrForCrosstalkSources_855434_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e126bb40-ead6-4ad8-954a-d3dfc0165acd_isrForCrosstalkSources_855434_60.stderr parsl.dataflow.dflow INFO: Task 2172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 008488c7c5275a6771ecd738f54c7cb2 with result from task 2172 parsl.dataflow.dflow DEBUG: Task 2172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/664637fb-bbbe-4a50-a742-b88fd35d6571_isrForCrosstalkSources_855404_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/664637fb-bbbe-4a50-a742-b88fd35d6571_isrForCrosstalkSources_855404_44.stderr parsl.dataflow.dflow INFO: Task 2137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d78c4f1d90257e66e4b4a04df6c378df with result from task 2137 parsl.dataflow.dflow DEBUG: Task 2137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7aca9415-8032-45fd-8fc6-e77318ab86ea_isrForCrosstalkSources_855381_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7aca9415-8032-45fd-8fc6-e77318ab86ea_isrForCrosstalkSources_855381_11.stderr parsl.dataflow.dflow INFO: Task 1881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40f9d52799df65134b3d78090206b49e with result from task 1881 parsl.dataflow.dflow DEBUG: Task 1881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91e2138a-5fed-43d4-b2c4-bfcf03b7118d_isrForCrosstalkSources_855413_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91e2138a-5fed-43d4-b2c4-bfcf03b7118d_isrForCrosstalkSources_855413_54.stderr parsl.dataflow.dflow INFO: Task 2225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b554a77f01e30a691456eb64e9adb73 with result from task 2225 parsl.dataflow.dflow DEBUG: Task 2225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/91e08b9e-057b-4707-b392-b2334920011f_isrForCrosstalkSources_855430_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/91e08b9e-057b-4707-b392-b2334920011f_isrForCrosstalkSources_855430_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:29 proc_lsst.multi:146 [INFO] found job 16964126 in provider checkpoint proc_lsst.multi INFO: found job 16964126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cd2f3b3b4e688a477718faf8ebde384 with result from task 2179 parsl.dataflow.dflow DEBUG: Task 2179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/d69df90c-03ae-464b-b9a7-55862d1b05b7_isrForCrosstalkSources_855458_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/d69df90c-03ae-464b-b9a7-55862d1b05b7_isrForCrosstalkSources_855458_20.stderr parsl.dataflow.dflow INFO: Task 1913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b02e7341736d39bf99bf9d8735da53b0 with result from task 1913 parsl.dataflow.dflow DEBUG: Task 1913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e6e364e2-6d21-4928-ae7e-e487d6fe67a3_isrForCrosstalkSources_855406_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/e6e364e2-6d21-4928-ae7e-e487d6fe67a3_isrForCrosstalkSources_855406_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:29 proc_lsst.multi:146 [INFO] found job 16964127 in provider checkpoint proc_lsst.multi INFO: found job 16964127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc13f700ee53188c55e62ff759b39aa with result from task 2042 parsl.dataflow.dflow DEBUG: Task 2042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f50f3fbb-7157-461f-b43b-a37ae2128306_isrForCrosstalkSources_855409_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f50f3fbb-7157-461f-b43b-a37ae2128306_isrForCrosstalkSources_855409_48.stderr parsl.dataflow.dflow INFO: Task 1850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 878b4d5f43a656479a0bfb3bf0885dcf with result from task 1850 parsl.dataflow.dflow DEBUG: Task 1850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/6c0ad98b-8d39-45a6-be53-09d7cea48231_isrForCrosstalkSources_855454_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/6c0ad98b-8d39-45a6-be53-09d7cea48231_isrForCrosstalkSources_855454_31.stderr parsl.dataflow.dflow INFO: Task 1899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c390d8867bd2c4c87651957fdbca6820 with result from task 1899 parsl.dataflow.dflow DEBUG: Task 1899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/121a4633-0677-4480-a5de-d6f769170ccc_isrForCrosstalkSources_855414_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/121a4633-0677-4480-a5de-d6f769170ccc_isrForCrosstalkSources_855414_49.stderr parsl.dataflow.dflow INFO: Task 1927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f56117b218ab6a165120ce77c1bf64b with result from task 1927 parsl.dataflow.dflow DEBUG: Task 1927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2df23423-f927-4549-8b75-55533ee5073c_isrForCrosstalkSources_855441_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2df23423-f927-4549-8b75-55533ee5073c_isrForCrosstalkSources_855441_62.stderr parsl.dataflow.dflow INFO: Task 2166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc486888e3d953507267a90d2c1d17ac with result from task 2166 parsl.dataflow.dflow DEBUG: Task 2166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e43ad5bd-d650-4862-a7e3-8d5fa7b1f487_isrForCrosstalkSources_855382_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e43ad5bd-d650-4862-a7e3-8d5fa7b1f487_isrForCrosstalkSources_855382_25.stderr parsl.dataflow.dflow INFO: Task 2082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eab516c911cef3c1298e071a1f8be68a with result from task 2082 parsl.dataflow.dflow DEBUG: Task 2082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68a997c1-b08a-4ce4-8a2c-433765f04e2e_isrForCrosstalkSources_855455_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/68a997c1-b08a-4ce4-8a2c-433765f04e2e_isrForCrosstalkSources_855455_42.stderr parsl.dataflow.dflow INFO: Task 2006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4d90f3a86c74a4361f2bb272fbbc328 with result from task 2006 parsl.dataflow.dflow DEBUG: Task 2006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e751dbcf-9443-493b-b53e-caf469446971_isrForCrosstalkSources_855421_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e751dbcf-9443-493b-b53e-caf469446971_isrForCrosstalkSources_855421_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:30 proc_lsst.multi:146 [INFO] found job 16964128 in provider checkpoint proc_lsst.multi INFO: found job 16964128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c058b9dde0b487b88dd3ad1c6a9afc82 with result from task 2208 parsl.dataflow.dflow DEBUG: Task 2208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/77fe158a-e3c4-483f-b4b0-3ba282f9f704_isrForCrosstalkSources_855441_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/77fe158a-e3c4-483f-b4b0-3ba282f9f704_isrForCrosstalkSources_855441_18.stderr parsl.dataflow.dflow INFO: Task 1910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f92b4420ef220f20c58d63dfc91adca8 with result from task 1910 parsl.dataflow.dflow DEBUG: Task 1910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/3577d9c2-b7f8-4eff-8844-8a9123ad38fa_isrForCrosstalkSources_855384_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/3577d9c2-b7f8-4eff-8844-8a9123ad38fa_isrForCrosstalkSources_855384_52.stderr parsl.dataflow.dflow INFO: Task 1916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f94040856128cec5c331c86e6f0df529 with result from task 1916 parsl.dataflow.dflow DEBUG: Task 1916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/264f50e2-b534-4dbe-bb04-25b3e14cc334_isrForCrosstalkSources_855397_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/264f50e2-b534-4dbe-bb04-25b3e14cc334_isrForCrosstalkSources_855397_31.stderr parsl.dataflow.dflow INFO: Task 2285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 891ec19f143fced8f1fc16fc160375da with result from task 2285 parsl.dataflow.dflow DEBUG: Task 2285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f1980b97-b23b-4385-9a19-60a78e909e8f_isrForCrosstalkSources_855382_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f1980b97-b23b-4385-9a19-60a78e909e8f_isrForCrosstalkSources_855382_4.stderr parsl.dataflow.dflow INFO: Task 2152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b2f20a5f6499b619f8db332e95908db with result from task 2152 parsl.dataflow.dflow DEBUG: Task 2152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/08ab1ede-2be7-48fa-bc45-2a84d7b1ef6f_isrForCrosstalkSources_855393_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/08ab1ede-2be7-48fa-bc45-2a84d7b1ef6f_isrForCrosstalkSources_855393_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 CG 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2184 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry a66a4a17c3fdb304fe4bddc8f02d6ead with result from task 2184 parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2184 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7c1603b7-6007-48a5-b2a5-a7b85c47f321_isrForCrosstalkSources_855443_33.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7c1603b7-6007-48a5-b2a5-a7b85c47f321_isrForCrosstalkSources_855443_33.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:30 proc_lsst.multi:146 [INFO] found job 16964129 in provider checkpoint proc_lsst.multi INFO: found job 16964129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb2c30c76dfb584b066ee3f7b594014 with result from task 2220 parsl.dataflow.dflow DEBUG: Task 2220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/70f1ceec-460e-44e4-bed6-9183f023120d_isrForCrosstalkSources_855430_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/70f1ceec-460e-44e4-bed6-9183f023120d_isrForCrosstalkSources_855430_44.stderr parsl.dataflow.dflow INFO: Task 2203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5727bbf580fe971581f2230b923c28c1 with result from task 2203 parsl.dataflow.dflow DEBUG: Task 2203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/675c34f2-491b-4199-9daf-3866758bac0c_isrForCrosstalkSources_855409_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/675c34f2-491b-4199-9daf-3866758bac0c_isrForCrosstalkSources_855409_53.stderr parsl.dataflow.dflow INFO: Task 2057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a63816c6ba6a7b198cbae45b914c843 with result from task 2057 parsl.dataflow.dflow DEBUG: Task 2057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/1b8f074b-9f17-4d13-a46c-039b83c64c44_isrForCrosstalkSources_855410_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/1b8f074b-9f17-4d13-a46c-039b83c64c44_isrForCrosstalkSources_855410_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 CG 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:30 proc_lsst.multi:146 [INFO] found job 16964130 in provider checkpoint proc_lsst.multi INFO: found job 16964130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad0cd4f623aabd37ffd0a99d22f9b811 with result from task 2107 parsl.dataflow.dflow DEBUG: Task 2107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a2500e5-c52b-4085-9496-ec8100dab1d5_isrForCrosstalkSources_855400_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a2500e5-c52b-4085-9496-ec8100dab1d5_isrForCrosstalkSources_855400_2.stderr parsl.dataflow.dflow INFO: Task 2252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e343b0244fab152f83f919d6fde700f with result from task 2252 parsl.dataflow.dflow DEBUG: Task 2252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a1870018-368b-493c-acc5-7ca18bc3700d_isrForCrosstalkSources_855427_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a1870018-368b-493c-acc5-7ca18bc3700d_isrForCrosstalkSources_855427_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:31 proc_lsst.multi:146 [INFO] found job 16964131 in provider checkpoint proc_lsst.multi INFO: found job 16964131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a964039603136545746f4a5b5954e4 with result from task 2259 parsl.dataflow.dflow DEBUG: Task 2259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f8d60b2-8948-4a0b-9bee-5c75a5cde9ee_isrForCrosstalkSources_855409_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0f8d60b2-8948-4a0b-9bee-5c75a5cde9ee_isrForCrosstalkSources_855409_20.stderr parsl.dataflow.dflow INFO: Task 2258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 469f3c90b148c7f55e19233438997bc6 with result from task 2258 parsl.dataflow.dflow DEBUG: Task 2258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b00ecd27-77f6-4b46-816b-d25fec42dfe0_isrForCrosstalkSources_855427_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b00ecd27-77f6-4b46-816b-d25fec42dfe0_isrForCrosstalkSources_855427_10.stderr parsl.dataflow.dflow INFO: Task 2265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70289b270f9154b6eec67da7b4fdb047 with result from task 2265 parsl.dataflow.dflow DEBUG: Task 2265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed1ca0fd-8667-4c0e-b710-1ba00aeb59d5_isrForCrosstalkSources_855453_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed1ca0fd-8667-4c0e-b710-1ba00aeb59d5_isrForCrosstalkSources_855453_45.stderr parsl.dataflow.dflow INFO: Task 2181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a6a2afc9ba866b2659f251c17086a4a with result from task 2181 parsl.dataflow.dflow DEBUG: Task 2181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ad4a6dbc-d0c7-4832-ad45-e7ce3a58f69b_isrForCrosstalkSources_855441_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ad4a6dbc-d0c7-4832-ad45-e7ce3a58f69b_isrForCrosstalkSources_855441_61.stderr parsl.dataflow.dflow INFO: Task 2239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2946fb0fa322516c2aa3749fc02cb439 with result from task 2239 parsl.dataflow.dflow DEBUG: Task 2239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/69389098-2ebd-4047-b277-1a9bc2c1b778_isrForCrosstalkSources_855420_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/69389098-2ebd-4047-b277-1a9bc2c1b778_isrForCrosstalkSources_855420_19.stderr parsl.dataflow.dflow INFO: Task 2046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce58cb2e7a9bfffdd17d21ee2969dcce with result from task 2046 parsl.dataflow.dflow DEBUG: Task 2046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/acfa45a7-df25-4fe1-b448-a5a6d6633526_isrForCrosstalkSources_855391_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/acfa45a7-df25-4fe1-b448-a5a6d6633526_isrForCrosstalkSources_855391_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:31 proc_lsst.multi:146 [INFO] found job 16964132 in provider checkpoint proc_lsst.multi INFO: found job 16964132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e3e1b54136acace8954ae400ddfe80b with result from task 2165 parsl.dataflow.dflow DEBUG: Task 2165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c9fe0283-9c7f-471b-b751-42677cd2febc_isrForCrosstalkSources_855439_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c9fe0283-9c7f-471b-b751-42677cd2febc_isrForCrosstalkSources_855439_58.stderr parsl.dataflow.dflow INFO: Task 2175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 566cbf68ee180ab7596ae0a3fb4c55b3 with result from task 2175 parsl.dataflow.dflow DEBUG: Task 2175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/63ff24bb-a27b-409e-ace5-f9b42391331e_isrForCrosstalkSources_855425_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/63ff24bb-a27b-409e-ace5-f9b42391331e_isrForCrosstalkSources_855425_34.stderr parsl.dataflow.dflow INFO: Task 2039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792bb065d3b30cf27c572f993ba9478d with result from task 2039 parsl.dataflow.dflow DEBUG: Task 2039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9854d422-a423-4b0f-bdba-5d8eac545b96_isrForCrosstalkSources_855448_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9854d422-a423-4b0f-bdba-5d8eac545b96_isrForCrosstalkSources_855448_18.stderr parsl.dataflow.dflow INFO: Task 2086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cd2566fc0d1259d9576708337bfd9fb with result from task 2086 parsl.dataflow.dflow DEBUG: Task 2086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24ba6116-5896-4344-92da-805cd08e9b66_isrForCrosstalkSources_855444_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24ba6116-5896-4344-92da-805cd08e9b66_isrForCrosstalkSources_855444_46.stderr parsl.dataflow.dflow INFO: Task 2291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ceec98812113e79019c81f2eabd790 with result from task 2291 parsl.dataflow.dflow DEBUG: Task 2291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/db86bdf4-e115-4986-a69b-a49f3a2deeac_isrForCrosstalkSources_855404_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/db86bdf4-e115-4986-a69b-a49f3a2deeac_isrForCrosstalkSources_855404_5.stderr parsl.dataflow.dflow INFO: Task 2313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0b1146a0355a07ff5780076573f078e with result from task 2313 parsl.dataflow.dflow DEBUG: Task 2313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/06d18253-25bd-4e8e-8e7b-0c56e0724702_isrForCrosstalkSources_855440_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/06d18253-25bd-4e8e-8e7b-0c56e0724702_isrForCrosstalkSources_855440_40.stderr parsl.dataflow.dflow INFO: Task 1883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e2e36f373d02feb2902d1bf6896a1f9 with result from task 1883 parsl.dataflow.dflow DEBUG: Task 1883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fd49f2cd-90c7-4a1f-8f25-6fc35e55df28_isrForCrosstalkSources_855442_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fd49f2cd-90c7-4a1f-8f25-6fc35e55df28_isrForCrosstalkSources_855442_52.stderr parsl.dataflow.dflow INFO: Task 2004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaec8c0b43492714d1919d474d81bf70 with result from task 2004 parsl.dataflow.dflow DEBUG: Task 2004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/cd56e708-4268-44b4-ad57-3b40635c0da4_isrForCrosstalkSources_855424_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/cd56e708-4268-44b4-ad57-3b40635c0da4_isrForCrosstalkSources_855424_46.stderr parsl.dataflow.dflow INFO: Task 2054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fdc20e92cc9a9c31c70487ab1ba9e07 with result from task 2054 parsl.dataflow.dflow DEBUG: Task 2054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7a22be39-9b64-4bab-b717-d47e0365d750_isrForCrosstalkSources_855453_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7a22be39-9b64-4bab-b717-d47e0365d750_isrForCrosstalkSources_855453_62.stderr parsl.dataflow.dflow INFO: Task 2256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1f105965416034f76bd7a6b225b83d with result from task 2256 parsl.dataflow.dflow DEBUG: Task 2256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f617a65a-7beb-4b33-98c7-65923b2e6063_isrForCrosstalkSources_855452_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f617a65a-7beb-4b33-98c7-65923b2e6063_isrForCrosstalkSources_855452_26.stderr parsl.dataflow.dflow INFO: Task 1877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ddc856fb746c40681d7b6ed200ee69c with result from task 1877 parsl.dataflow.dflow DEBUG: Task 1877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ab13f348-851c-4d4d-9cb7-a487d5a9c5a1_isrForCrosstalkSources_855447_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ab13f348-851c-4d4d-9cb7-a487d5a9c5a1_isrForCrosstalkSources_855447_48.stderr parsl.dataflow.dflow INFO: Task 2298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d26d47ee6be7d9eb24b5957f5b58073e with result from task 2298 parsl.dataflow.dflow DEBUG: Task 2298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/645f91d1-98a9-462e-b209-34cb836c099c_isrForCrosstalkSources_855395_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/645f91d1-98a9-462e-b209-34cb836c099c_isrForCrosstalkSources_855395_14.stderr parsl.dataflow.dflow INFO: Task 2228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c1bbf932f4348438fcba7816354f84 with result from task 2228 parsl.dataflow.dflow DEBUG: Task 2228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/82122e32-1885-4816-8034-4b18414578b8_isrForCrosstalkSources_855416_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/82122e32-1885-4816-8034-4b18414578b8_isrForCrosstalkSources_855416_8.stderr parsl.dataflow.dflow INFO: Task 2212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60f52c6d1ea8789c8baa825dca2fe803 with result from task 2212 parsl.dataflow.dflow DEBUG: Task 2212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/320e01f0-d24c-4dff-8c94-c6f08c5cc92d_isrForCrosstalkSources_855459_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/320e01f0-d24c-4dff-8c94-c6f08c5cc92d_isrForCrosstalkSources_855459_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:31 proc_lsst.multi:146 [INFO] found job 16964133 in provider checkpoint proc_lsst.multi INFO: found job 16964133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39cd914d00f9297946e0e3672aa92f7d with result from task 2379 parsl.dataflow.dflow DEBUG: Task 2379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1987c67e-544f-4533-81ee-d0c2beab1384_isrForCrosstalkSources_855441_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1987c67e-544f-4533-81ee-d0c2beab1384_isrForCrosstalkSources_855441_24.stderr parsl.dataflow.dflow INFO: Task 2200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2ce1506be07fbb3f4a8085c796e6951 with result from task 2200 parsl.dataflow.dflow DEBUG: Task 2200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/149aa58c-c0bd-47d8-ad0c-dbbc6c317a14_isrForCrosstalkSources_855406_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/149aa58c-c0bd-47d8-ad0c-dbbc6c317a14_isrForCrosstalkSources_855406_20.stderr parsl.dataflow.dflow INFO: Task 2307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e50471854ee028df781f8655feb20950 with result from task 2307 parsl.dataflow.dflow DEBUG: Task 2307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8c137b8d-b082-4582-a2d5-9555c3b90f1a_isrForCrosstalkSources_855422_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/8c137b8d-b082-4582-a2d5-9555c3b90f1a_isrForCrosstalkSources_855422_27.stderr parsl.dataflow.dflow INFO: Task 2122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9773386965c411c3b369b02ac4f7e822 with result from task 2122 parsl.dataflow.dflow DEBUG: Task 2122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/add05326-0f4d-478b-b234-a34592b3f9ee_isrForCrosstalkSources_855380_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/add05326-0f4d-478b-b234-a34592b3f9ee_isrForCrosstalkSources_855380_38.stderr parsl.dataflow.dflow INFO: Task 2287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bfcd6d933ff2883a593d9945cb779cb with result from task 2287 parsl.dataflow.dflow DEBUG: Task 2287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f2f06b3b-fa93-41d8-840d-310c900fabf8_isrForCrosstalkSources_855396_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f2f06b3b-fa93-41d8-840d-310c900fabf8_isrForCrosstalkSources_855396_21.stderr parsl.dataflow.dflow INFO: Task 2398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f90648fb6eae9bf27ec9de97d97e85ee with result from task 2398 parsl.dataflow.dflow DEBUG: Task 2398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5a934b80-717a-469f-8ada-71284bcb0860_isrForCrosstalkSources_855451_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5a934b80-717a-469f-8ada-71284bcb0860_isrForCrosstalkSources_855451_29.stderr parsl.dataflow.dflow INFO: Task 2262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bd483f98d8f541b994c7aa75cedd7d3 with result from task 2262 parsl.dataflow.dflow DEBUG: Task 2262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0c654d09-4e78-4284-8f30-563540cba64f_isrForCrosstalkSources_855408_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0c654d09-4e78-4284-8f30-563540cba64f_isrForCrosstalkSources_855408_20.stderr parsl.dataflow.dflow INFO: Task 1849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc7b2a9b22b6f52713098345107fad64 with result from task 1849 parsl.dataflow.dflow DEBUG: Task 1849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8b299c75-6a97-48ab-846e-59201d6f69ff_isrForCrosstalkSources_855386_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8b299c75-6a97-48ab-846e-59201d6f69ff_isrForCrosstalkSources_855386_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:32 proc_lsst.multi:146 [INFO] found job 16964134 in provider checkpoint proc_lsst.multi INFO: found job 16964134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dee0ac057efb6f4276a39456ee83b37 with result from task 2264 parsl.dataflow.dflow DEBUG: Task 2264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0692b536-0ef7-4951-9510-a58a9045eff3_isrForCrosstalkSources_855395_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0692b536-0ef7-4951-9510-a58a9045eff3_isrForCrosstalkSources_855395_9.stderr parsl.dataflow.dflow INFO: Task 1915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdb16412fd2b038e7d5837e8651d3d0b with result from task 1915 parsl.dataflow.dflow DEBUG: Task 1915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/b1c8267a-cbdb-44a1-81a8-c0ce77b00bfa_isrForCrosstalkSources_855387_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/b1c8267a-cbdb-44a1-81a8-c0ce77b00bfa_isrForCrosstalkSources_855387_55.stderr parsl.dataflow.dflow INFO: Task 2141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50618d649b3fe8c5649fc2353e408a87 with result from task 2141 parsl.dataflow.dflow DEBUG: Task 2141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/aedda8e7-bbae-4be6-8780-31dfcd3b57a9_isrForCrosstalkSources_855412_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/aedda8e7-bbae-4be6-8780-31dfcd3b57a9_isrForCrosstalkSources_855412_30.stderr parsl.dataflow.dflow INFO: Task 2343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1db4e841ccd04ca8e8fcb6a69de2d3b with result from task 2343 parsl.dataflow.dflow DEBUG: Task 2343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ac294c7d-323c-44ce-b94e-f4357d86159d_isrForCrosstalkSources_855403_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ac294c7d-323c-44ce-b94e-f4357d86159d_isrForCrosstalkSources_855403_46.stderr parsl.dataflow.dflow INFO: Task 2105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c6e8db1abb51fa686bbda291c68006c with result from task 2105 parsl.dataflow.dflow DEBUG: Task 2105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/82955c78-fa1a-409c-9306-18a7a537b10b_isrForCrosstalkSources_855436_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/82955c78-fa1a-409c-9306-18a7a537b10b_isrForCrosstalkSources_855436_45.stderr parsl.dataflow.dflow INFO: Task 2147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25ce78e5419872784a600d62c844b09c with result from task 2147 parsl.dataflow.dflow DEBUG: Task 2147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bc1604f0-5276-42de-befd-b4acc8f42011_isrForCrosstalkSources_855397_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bc1604f0-5276-42de-befd-b4acc8f42011_isrForCrosstalkSources_855397_14.stderr parsl.dataflow.dflow INFO: Task 2314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85a2269a90ba856a14d0ee87320cd1c5 with result from task 2314 parsl.dataflow.dflow DEBUG: Task 2314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/30dd5d45-cf70-471d-9fe4-207998930282_isrForCrosstalkSources_855433_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/30dd5d45-cf70-471d-9fe4-207998930282_isrForCrosstalkSources_855433_46.stderr parsl.dataflow.dflow INFO: Task 2123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8c6d52c7617ac253736ed8d168a42c with result from task 2123 parsl.dataflow.dflow DEBUG: Task 2123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cbfdf39d-b2ce-482e-bdb5-4937ed1f74d2_isrForCrosstalkSources_855445_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/cbfdf39d-b2ce-482e-bdb5-4937ed1f74d2_isrForCrosstalkSources_855445_27.stderr parsl.dataflow.dflow INFO: Task 2148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10635ad7de62ac74bb5179b35f4ba5d8 with result from task 2148 parsl.dataflow.dflow DEBUG: Task 2148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/33fdadc6-1463-465a-bd88-961655bf045b_isrForCrosstalkSources_855411_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/33fdadc6-1463-465a-bd88-961655bf045b_isrForCrosstalkSources_855411_14.stderr parsl.dataflow.dflow INFO: Task 2261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3874f4dffd33cc0a8fcee3fc9ef1306 with result from task 2261 parsl.dataflow.dflow DEBUG: Task 2261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5e95f288-f475-4fc7-abc5-67b6de053c20_isrForCrosstalkSources_855413_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/5e95f288-f475-4fc7-abc5-67b6de053c20_isrForCrosstalkSources_855413_10.stderr parsl.dataflow.dflow INFO: Task 2231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b01bb69c46300cd7f201c6658b666870 with result from task 2231 parsl.dataflow.dflow DEBUG: Task 2231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72d448e7-6db5-4a6b-8016-9f8e7324e621_isrForCrosstalkSources_855437_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/72d448e7-6db5-4a6b-8016-9f8e7324e621_isrForCrosstalkSources_855437_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:32 proc_lsst.multi:146 [INFO] found job 16964135 in provider checkpoint proc_lsst.multi INFO: found job 16964135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de2bcdcc2d95d1194670926777235ea with result from task 2211 parsl.dataflow.dflow DEBUG: Task 2211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec521dd7-6b2d-4f01-b767-be95ff5ec0dc_isrForCrosstalkSources_855413_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec521dd7-6b2d-4f01-b767-be95ff5ec0dc_isrForCrosstalkSources_855413_5.stderr parsl.dataflow.dflow INFO: Task 2096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5fcd81b9c9dec7e23c4ea6466750032 with result from task 2096 parsl.dataflow.dflow DEBUG: Task 2096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/798bd937-5adc-4f99-b5ab-ffffbb03b8f8_isrForCrosstalkSources_855414_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/798bd937-5adc-4f99-b5ab-ffffbb03b8f8_isrForCrosstalkSources_855414_55.stderr parsl.dataflow.dflow INFO: Task 2143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2370cc75ee4487b437f97ed2d4aae669 with result from task 2143 parsl.dataflow.dflow DEBUG: Task 2143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7af609e0-9a3c-4e1d-bfc8-9d6b5824c607_isrForCrosstalkSources_855422_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7af609e0-9a3c-4e1d-bfc8-9d6b5824c607_isrForCrosstalkSources_855422_21.stderr parsl.dataflow.dflow INFO: Task 2202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46b2fa38689b5a61f2bc294ce502cf2 with result from task 2202 parsl.dataflow.dflow DEBUG: Task 2202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/38e44ff3-3552-458d-be70-e71f713d0c6c_isrForCrosstalkSources_855403_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/38e44ff3-3552-458d-be70-e71f713d0c6c_isrForCrosstalkSources_855403_45.stderr parsl.dataflow.dflow INFO: Task 2266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571ef8288800675d54fb60e5efb95eb7 with result from task 2266 parsl.dataflow.dflow DEBUG: Task 2266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/166c15ac-6f73-4da2-912d-f6d1f3a8cd6c_isrForCrosstalkSources_855453_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/166c15ac-6f73-4da2-912d-f6d1f3a8cd6c_isrForCrosstalkSources_855453_4.stderr parsl.dataflow.dflow INFO: Task 2293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77563abd113b5bbd6598132736d237aa with result from task 2293 parsl.dataflow.dflow DEBUG: Task 2293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/aabad0a7-467c-454e-9928-af64b949907d_isrForCrosstalkSources_855448_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/aabad0a7-467c-454e-9928-af64b949907d_isrForCrosstalkSources_855448_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 2383 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 3a2fea57af307cc6068e4841b22ca93e with result from task 2383 parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2383 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:33 proc_lsst.multi:146 [INFO] found job 16964136 in provider checkpoint parsl.dataflow.dflow INFO: Done checkpointing 1 tasks proc_lsst.multi INFO: found job 16964136 in provider checkpoint parsl.dataflow.dflow INFO: Standard output for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/fd2f2765-e4af-4eca-bf6a-884da302469f_isrForCrosstalkSources_855443_20.stdout parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Standard error for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/fd2f2765-e4af-4eca-bf6a-884da302469f_isrForCrosstalkSources_855443_20.stderr parsl.dataflow.dflow INFO: Task 2170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d11ef85c37383c37e9e9b9fa018518c1 with result from task 2170 parsl.dataflow.dflow DEBUG: Task 2170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/8f88c479-9371-4ab8-8f11-657b19756223_isrForCrosstalkSources_855438_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/8f88c479-9371-4ab8-8f11-657b19756223_isrForCrosstalkSources_855438_19.stderr parsl.dataflow.dflow INFO: Task 2033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06de22bfed01ced7615218817ac62635 with result from task 2033 parsl.dataflow.dflow DEBUG: Task 2033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ae1c6241-b1d5-4c32-b456-85acd61d526c_isrForCrosstalkSources_855398_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ae1c6241-b1d5-4c32-b456-85acd61d526c_isrForCrosstalkSources_855398_28.stderr parsl.dataflow.dflow INFO: Task 2286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d00b97954ae352116a63232b6957f9ac with result from task 2286 parsl.dataflow.dflow DEBUG: Task 2286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e7c5f42-9b96-4bbb-b8bf-a762e6c0dd14_isrForCrosstalkSources_855429_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e7c5f42-9b96-4bbb-b8bf-a762e6c0dd14_isrForCrosstalkSources_855429_28.stderr parsl.dataflow.dflow INFO: Task 2111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52dfa9b3b9b165e1aa5ff799bcd3cdc7 with result from task 2111 parsl.dataflow.dflow DEBUG: Task 2111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/58a0562d-3b18-4dea-9e10-6c33a07c46ce_isrForCrosstalkSources_855400_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/58a0562d-3b18-4dea-9e10-6c33a07c46ce_isrForCrosstalkSources_855400_60.stderr parsl.dataflow.dflow INFO: Task 2275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f4ba3b931dce232403daaa63a993b1 with result from task 2275 parsl.dataflow.dflow DEBUG: Task 2275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/15dbcb47-16fb-412e-8bd5-5abc5e2bb13c_isrForCrosstalkSources_855390_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/15dbcb47-16fb-412e-8bd5-5abc5e2bb13c_isrForCrosstalkSources_855390_32.stderr parsl.dataflow.dflow INFO: Task 2288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa3c6b070941f72ebebaffb9c3486c4b with result from task 2288 parsl.dataflow.dflow DEBUG: Task 2288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e3266562-35ca-4930-8aa1-cc077a227152_isrForCrosstalkSources_855404_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e3266562-35ca-4930-8aa1-cc077a227152_isrForCrosstalkSources_855404_43.stderr parsl.dataflow.dflow INFO: Task 2069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d96bc47a7d0eabb294d42d720d3672b with result from task 2069 parsl.dataflow.dflow DEBUG: Task 2069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e22899e9-671d-4643-be6d-0b381a6a82d2_isrForCrosstalkSources_855391_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e22899e9-671d-4643-be6d-0b381a6a82d2_isrForCrosstalkSources_855391_27.stderr parsl.dataflow.dflow INFO: Task 2245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc0b0f4a6dac1abf0b29eaa475cceabe with result from task 2245 parsl.dataflow.dflow DEBUG: Task 2245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/00bfc4ad-2fe7-45cc-9085-f0be19f40b8f_isrForCrosstalkSources_855407_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/00bfc4ad-2fe7-45cc-9085-f0be19f40b8f_isrForCrosstalkSources_855407_9.stderr parsl.dataflow.dflow INFO: Task 2093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edf9fca314f92200fb85026486f16c2b with result from task 2093 parsl.dataflow.dflow DEBUG: Task 2093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e7070891-ee00-4ee7-a254-142e9f8d3e59_isrForCrosstalkSources_855436_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e7070891-ee00-4ee7-a254-142e9f8d3e59_isrForCrosstalkSources_855436_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:33 proc_lsst.multi:146 [INFO] found job 16964137 in provider checkpoint proc_lsst.multi INFO: found job 16964137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 1970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f38251a88a22716b839de4f916636ce with result from task 1970 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 1970 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/1b42d58d-1b77-4b95-abfa-ea64bbccd5df_isrForCrosstalkSources_855449_23.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/1b42d58d-1b77-4b95-abfa-ea64bbccd5df_isrForCrosstalkSources_855449_23.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 1854 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry ff02ef03a6ea8ff4a026170e0fb8cfb9 with result from task 1854 parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 1854 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/894365fa-3212-4f2b-8b81-432a1f323612_isrForCrosstalkSources_855433_55.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/894365fa-3212-4f2b-8b81-432a1f323612_isrForCrosstalkSources_855433_55.stderr parsl.dataflow.dflow INFO: Task 2134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1e133ecd669b54b86502e051deeacd1 with result from task 2134 parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2134 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5c2ee636-3d0a-4399-ba70-90774a7a1a08_isrForCrosstalkSources_855406_46.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5c2ee636-3d0a-4399-ba70-90774a7a1a08_isrForCrosstalkSources_855406_46.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55790f6c5437482f591a9debe43f6fe5 with result from task 2222 parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2222 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/c72df4ec-5657-44dd-ad1a-2c3ee302924e_isrForCrosstalkSources_855447_52.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/c72df4ec-5657-44dd-ad1a-2c3ee302924e_isrForCrosstalkSources_855447_52.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2316 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 4fed90bb621643c6fdfeeed1a87fea83 with result from task 2316 parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2316 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7bb8ba3a-7160-4f1d-8fc3-9a5da8494272_isrForCrosstalkSources_855448_35.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/7bb8ba3a-7160-4f1d-8fc3-9a5da8494272_isrForCrosstalkSources_855448_35.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:33 proc_lsst.multi:146 [INFO] found job 16964138 in provider checkpoint proc_lsst.multi INFO: found job 16964138 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed4187d3f05e14af68f5fa0fc8b916c4 with result from task 2205 parsl.dataflow.dflow DEBUG: Task 2205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6eb4bf60-5ec7-4c6d-97df-67a5b7e85573_isrForCrosstalkSources_855456_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6eb4bf60-5ec7-4c6d-97df-67a5b7e85573_isrForCrosstalkSources_855456_16.stderr parsl.dataflow.dflow INFO: Task 2151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c48c47ecfb686831d8b118aba08a70e with result from task 2151 parsl.dataflow.dflow DEBUG: Task 2151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7885c3c6-3d32-47a5-b7d4-8fa94525f9f7_isrForCrosstalkSources_855436_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/7885c3c6-3d32-47a5-b7d4-8fa94525f9f7_isrForCrosstalkSources_855436_13.stderr parsl.dataflow.dflow INFO: Task 1966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7980031865fa7e23ab07d9733b125c35 with result from task 1966 parsl.dataflow.dflow DEBUG: Task 1966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/826cb2c0-8134-4c3b-b546-bd325759507e_isrForCrosstalkSources_855453_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/826cb2c0-8134-4c3b-b546-bd325759507e_isrForCrosstalkSources_855453_57.stderr parsl.dataflow.dflow INFO: Task 2386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82937068c4c569bdae5eba958a1a6d84 with result from task 2386 parsl.dataflow.dflow DEBUG: Task 2386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/120fd0a3-607f-4553-8993-9f2eb2ff929d_isrForCrosstalkSources_855391_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/120fd0a3-607f-4553-8993-9f2eb2ff929d_isrForCrosstalkSources_855391_15.stderr parsl.dataflow.dflow INFO: Task 2247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bd32dff7c1976de5761662b99b4a9ad with result from task 2247 parsl.dataflow.dflow DEBUG: Task 2247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/6fa0e290-1379-42c4-9b16-158f9426a3b6_isrForCrosstalkSources_855438_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/6fa0e290-1379-42c4-9b16-158f9426a3b6_isrForCrosstalkSources_855438_14.stderr parsl.dataflow.dflow INFO: Task 1993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ed6ddeca0855ced9265b9f9b50bd8b0 with result from task 1993 parsl.dataflow.dflow DEBUG: Task 1993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/fd320069-d73d-45d6-af2f-7f3e4865041e_isrForCrosstalkSources_855446_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/fd320069-d73d-45d6-af2f-7f3e4865041e_isrForCrosstalkSources_855446_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2375 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry bcbfcd40304bf69a0d025c3a100ed287 with result from task 2375 parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2375 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8895118a-0566-425d-8c57-b4440fb32a06_isrForCrosstalkSources_855382_41.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/8895118a-0566-425d-8c57-b4440fb32a06_isrForCrosstalkSources_855382_41.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:34 proc_lsst.multi:146 [INFO] found job 16964139 in provider checkpoint proc_lsst.multi INFO: found job 16964139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb5c74e2587274f78c19d85fd42fb22b with result from task 2135 parsl.dataflow.dflow DEBUG: Task 2135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/b68caddc-c4b6-4004-bc8b-97de5f14369c_isrForCrosstalkSources_855431_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/b68caddc-c4b6-4004-bc8b-97de5f14369c_isrForCrosstalkSources_855431_14.stderr parsl.dataflow.dflow INFO: Task 2074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73be6d22febeb9cc0e23260c833dd686 with result from task 2074 parsl.dataflow.dflow DEBUG: Task 2074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c23d3c86-24ef-4225-8e6c-bf5bcedd23e7_isrForCrosstalkSources_855383_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/c23d3c86-24ef-4225-8e6c-bf5bcedd23e7_isrForCrosstalkSources_855383_48.stderr parsl.dataflow.dflow INFO: Task 2163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5763c87aa4acaecba155f382b208dfa3 with result from task 2163 parsl.dataflow.dflow DEBUG: Task 2163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2bf82d1c-a2fc-45da-a0f0-9530180c88d5_isrForCrosstalkSources_855400_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2bf82d1c-a2fc-45da-a0f0-9530180c88d5_isrForCrosstalkSources_855400_26.stderr parsl.dataflow.dflow INFO: Task 2102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28dfc5fcc29fafa13393dba21920728e with result from task 2102 parsl.dataflow.dflow DEBUG: Task 2102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/d797de75-16c3-45cc-8160-dbf0b865075f_isrForCrosstalkSources_855453_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/d797de75-16c3-45cc-8160-dbf0b865075f_isrForCrosstalkSources_855453_13.stderr parsl.dataflow.dflow INFO: Task 2241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12edc420053bc86d9acebc81d987fdf9 with result from task 2241 parsl.dataflow.dflow DEBUG: Task 2241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/15dce3a0-be8c-45ca-8828-715afcdd0b80_isrForCrosstalkSources_855424_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/15dce3a0-be8c-45ca-8828-715afcdd0b80_isrForCrosstalkSources_855424_18.stderr parsl.dataflow.dflow INFO: Task 2029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dab7ea5ed423ec6d36a97ecb6bdcbfcd with result from task 2029 parsl.dataflow.dflow DEBUG: Task 2029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/0ea58fd6-b16b-4cde-aeca-440f315a4d20_isrForCrosstalkSources_855397_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/0ea58fd6-b16b-4cde-aeca-440f315a4d20_isrForCrosstalkSources_855397_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2406 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 02eae4d489d6855e9905ce96e6e12d63 with result from task 2406 parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2406 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9be537aa-f4e7-4b49-8a31-b4fae92de33d_isrForCrosstalkSources_855438_42.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/9be537aa-f4e7-4b49-8a31-b4fae92de33d_isrForCrosstalkSources_855438_42.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:34 proc_lsst.multi:146 [INFO] found job 16964140 in provider checkpoint proc_lsst.multi INFO: found job 16964140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ffc75cac8d87c32970dcc71e4c93582 with result from task 2263 parsl.dataflow.dflow DEBUG: Task 2263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a94700fc-f45f-4a97-b3b8-1175aa11e590_isrForCrosstalkSources_855429_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/a94700fc-f45f-4a97-b3b8-1175aa11e590_isrForCrosstalkSources_855429_45.stderr parsl.dataflow.dflow INFO: Task 2260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3ff686cba2cdd4dc485dc12a98ebf13 with result from task 2260 parsl.dataflow.dflow DEBUG: Task 2260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/fdc804dc-2387-4ef9-aa27-77e831cfc25c_isrForCrosstalkSources_855397_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/fdc804dc-2387-4ef9-aa27-77e831cfc25c_isrForCrosstalkSources_855397_10.stderr parsl.dataflow.dflow INFO: Task 2377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25d0b5e2f1555c129fae3bc1bfe621ab with result from task 2377 parsl.dataflow.dflow DEBUG: Task 2377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a6b6bcf4-0c50-4767-b5e7-aac165ff81af_isrForCrosstalkSources_855385_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a6b6bcf4-0c50-4767-b5e7-aac165ff81af_isrForCrosstalkSources_855385_13.stderr parsl.dataflow.dflow INFO: Task 2303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8f87a746dd62c4e1dc07bc4367c494 with result from task 2303 parsl.dataflow.dflow DEBUG: Task 2303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7917b0ce-ea61-40af-8c03-ab361cc2b5ad_isrForCrosstalkSources_855379_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7917b0ce-ea61-40af-8c03-ab361cc2b5ad_isrForCrosstalkSources_855379_22.stderr parsl.dataflow.dflow INFO: Task 2315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 393caa88c008694612922910b35d721b with result from task 2315 parsl.dataflow.dflow DEBUG: Task 2315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4b7ccec6-0eda-4995-8643-6816d7706ebd_isrForCrosstalkSources_855409_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4b7ccec6-0eda-4995-8643-6816d7706ebd_isrForCrosstalkSources_855409_56.stderr parsl.dataflow.dflow INFO: Task 2131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 344d31ee4610e65d5f6d96bc338de2f6 with result from task 2131 parsl.dataflow.dflow DEBUG: Task 2131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e494d765-b3b5-4cd3-9463-e9d749c2cfed_isrForCrosstalkSources_855418_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e494d765-b3b5-4cd3-9463-e9d749c2cfed_isrForCrosstalkSources_855418_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:34 proc_lsst.multi:146 [INFO] found job 16964141 in provider checkpoint proc_lsst.multi INFO: found job 16964141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b2c01d48f28297f60c8b0b0bd5bdb04 with result from task 2199 parsl.dataflow.dflow DEBUG: Task 2199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/293b0fea-4bae-4aa1-b059-78b0e67c63b0_isrForCrosstalkSources_855453_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/293b0fea-4bae-4aa1-b059-78b0e67c63b0_isrForCrosstalkSources_855453_48.stderr parsl.dataflow.dflow INFO: Task 2246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ff798d7c32797f37e6af92bdce9c8d5 with result from task 2246 parsl.dataflow.dflow DEBUG: Task 2246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5d43df9c-39d8-437f-9b1b-b7234fb49de0_isrForCrosstalkSources_855440_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/5d43df9c-39d8-437f-9b1b-b7234fb49de0_isrForCrosstalkSources_855440_9.stderr parsl.dataflow.dflow INFO: Task 2376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 315a2da9b7aad333a970bb1ab5a61460 with result from task 2376 parsl.dataflow.dflow DEBUG: Task 2376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/7bc30094-e9ec-48f4-86d7-e90f760df552_isrForCrosstalkSources_855418_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/7bc30094-e9ec-48f4-86d7-e90f760df552_isrForCrosstalkSources_855418_60.stderr parsl.dataflow.dflow INFO: Task 2411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ca91846f0bc0037d4e4fa0f5763f276 with result from task 2411 parsl.dataflow.dflow DEBUG: Task 2411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bf5eff79-59cb-433f-aab4-3e2ac2baa38d_isrForCrosstalkSources_855381_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bf5eff79-59cb-433f-aab4-3e2ac2baa38d_isrForCrosstalkSources_855381_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:35 proc_lsst.multi:146 [INFO] found job 16964142 in provider checkpoint proc_lsst.multi INFO: found job 16964142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98f186b0e0b50126e316bc93227f0ce4 with result from task 2027 parsl.dataflow.dflow DEBUG: Task 2027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b4585347-7155-4f0c-87f6-b0095c553b81_isrForCrosstalkSources_855458_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b4585347-7155-4f0c-87f6-b0095c553b81_isrForCrosstalkSources_855458_6.stderr parsl.dataflow.dflow INFO: Task 2007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7c73f02c2c16e9e73ee535fa451e646 with result from task 2007 parsl.dataflow.dflow DEBUG: Task 2007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7e19c8a0-402c-419f-8265-b1c05020a675_isrForCrosstalkSources_855381_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/7e19c8a0-402c-419f-8265-b1c05020a675_isrForCrosstalkSources_855381_60.stderr parsl.dataflow.dflow INFO: Task 2156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c68160da5d6145b2040eb31922bad8 with result from task 2156 parsl.dataflow.dflow DEBUG: Task 2156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3af54afd-0fc7-4fa9-8dcc-738b444aba12_isrForCrosstalkSources_855386_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3af54afd-0fc7-4fa9-8dcc-738b444aba12_isrForCrosstalkSources_855386_38.stderr parsl.dataflow.dflow INFO: Task 2221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d42cd4fc8650b13a62dc1b55c13b1c4 with result from task 2221 parsl.dataflow.dflow DEBUG: Task 2221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/38cfd2a8-e61e-470f-96a8-e67116c300c8_isrForCrosstalkSources_855402_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/38cfd2a8-e61e-470f-96a8-e67116c300c8_isrForCrosstalkSources_855402_35.stderr parsl.dataflow.dflow INFO: Task 2359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80321bf7dbc3f0b6a35bcfd62381d41e with result from task 2359 parsl.dataflow.dflow DEBUG: Task 2359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ff396927-fabd-47aa-9247-41293c89ff1e_isrForCrosstalkSources_855453_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ff396927-fabd-47aa-9247-41293c89ff1e_isrForCrosstalkSources_855453_58.stderr parsl.dataflow.dflow INFO: Task 2284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd35ba3586954e6101a85e136b9d09e3 with result from task 2284 parsl.dataflow.dflow DEBUG: Task 2284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9bc96fe8-21e3-4841-bf15-61c71379f8bf_isrForCrosstalkSources_855395_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9bc96fe8-21e3-4841-bf15-61c71379f8bf_isrForCrosstalkSources_855395_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:35 proc_lsst.multi:146 [INFO] found job 16964143 in provider checkpoint proc_lsst.multi INFO: found job 16964143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c08823281e9b6803aa0496d7fa31f83 with result from task 2024 parsl.dataflow.dflow DEBUG: Task 2024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/28b4293e-de17-441e-aaac-d6fb9b6b1300_isrForCrosstalkSources_855442_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/28b4293e-de17-441e-aaac-d6fb9b6b1300_isrForCrosstalkSources_855442_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:35 proc_lsst.multi:146 [INFO] found job 16964144 in provider checkpoint proc_lsst.multi INFO: found job 16964144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2253e63d276476b51d698a7cf395a344 with result from task 2201 parsl.dataflow.dflow DEBUG: Task 2201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7339f5af-9527-44c9-ace4-a34863b3927a_isrForCrosstalkSources_855386_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7339f5af-9527-44c9-ace4-a34863b3927a_isrForCrosstalkSources_855386_42.stderr parsl.dataflow.dflow INFO: Task 2145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36f98afe7fb0f09936b7cf8c4c07ec02 with result from task 2145 parsl.dataflow.dflow DEBUG: Task 2145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e5979b25-587a-4302-ba58-104f34ac9ed6_isrForCrosstalkSources_855456_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e5979b25-587a-4302-ba58-104f34ac9ed6_isrForCrosstalkSources_855456_21.stderr parsl.dataflow.dflow INFO: Task 1952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4925311349e7055b4598c74467fd0179 with result from task 1952 parsl.dataflow.dflow DEBUG: Task 1952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f6d529e3-ec54-4df3-9139-6dc76cc84244_isrForCrosstalkSources_855384_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f6d529e3-ec54-4df3-9139-6dc76cc84244_isrForCrosstalkSources_855384_43.stderr parsl.dataflow.dflow INFO: Task 2077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d45b7311e4d73a1709b0694b0196b35b with result from task 2077 parsl.dataflow.dflow DEBUG: Task 2077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/08ba01d6-8b6a-4d3d-bdc6-d9c7f9a04b6a_isrForCrosstalkSources_855404_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/08ba01d6-8b6a-4d3d-bdc6-d9c7f9a04b6a_isrForCrosstalkSources_855404_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:36 proc_lsst.multi:146 [INFO] found job 16964145 in provider checkpoint proc_lsst.multi INFO: found job 16964145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57cea68d927c7c6eedd81b96798139b2 with result from task 2240 parsl.dataflow.dflow DEBUG: Task 2240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/881f6011-e669-44f1-a996-5b8fea089224_isrForCrosstalkSources_855415_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/881f6011-e669-44f1-a996-5b8fea089224_isrForCrosstalkSources_855415_58.stderr parsl.dataflow.dflow INFO: Task 1962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f25fd89b71fff385e4cb917be858cbb0 with result from task 1962 parsl.dataflow.dflow DEBUG: Task 1962 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0ac35c29-a51a-4ee8-82e0-af617690bb2b_isrForCrosstalkSources_855430_5.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0ac35c29-a51a-4ee8-82e0-af617690bb2b_isrForCrosstalkSources_855430_5.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2267 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry fc62885015d6ed98b7cd85e8425fd6b5 with result from task 2267 parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2267 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/99c54937-1a43-457e-b345-ad51b9b68e6b_isrForCrosstalkSources_855447_60.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/99c54937-1a43-457e-b345-ad51b9b68e6b_isrForCrosstalkSources_855447_60.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2150 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 96cc86f66a615412b2c2f46184b8b2b5 with result from task 2150 parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2150 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/da9c9462-dff2-4330-893b-9d98f42f9d85_isrForCrosstalkSources_855439_24.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/da9c9462-dff2-4330-893b-9d98f42f9d85_isrForCrosstalkSources_855439_24.stderr parsl.dataflow.dflow INFO: Task 2237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19ab40f91024f26e027a5fb893b19ce9 with result from task 2237 parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2237 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f490dff2-dcab-4246-b94c-0a96d9c0a74c_isrForCrosstalkSources_855451_62.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f490dff2-dcab-4246-b94c-0a96d9c0a74c_isrForCrosstalkSources_855451_62.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:36 proc_lsst.multi:146 [INFO] found job 16964146 in provider checkpoint proc_lsst.multi INFO: found job 16964146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96eac7e80ae010b269a9ed3acb34bdd7 with result from task 2041 parsl.dataflow.dflow DEBUG: Task 2041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06f8acf7-155e-49c6-8a9d-dfc09b52e608_isrForCrosstalkSources_855454_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06f8acf7-155e-49c6-8a9d-dfc09b52e608_isrForCrosstalkSources_855454_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:36 proc_lsst.multi:146 [INFO] found job 16964147 in provider checkpoint proc_lsst.multi INFO: found job 16964147 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48cf5dff5ff163542e5d1f4ec57a011a with result from task 2282 parsl.dataflow.dflow DEBUG: Task 2282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4e0979a2-e84b-4acb-ac01-bb0baaa6e461_isrForCrosstalkSources_855383_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4e0979a2-e84b-4acb-ac01-bb0baaa6e461_isrForCrosstalkSources_855383_28.stderr parsl.dataflow.dflow INFO: Task 2081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee536651437ea88a6edfcdaff94d4173 with result from task 2081 parsl.dataflow.dflow DEBUG: Task 2081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4894d996-f2d6-440f-8d52-8599122314d0_isrForCrosstalkSources_855400_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4894d996-f2d6-440f-8d52-8599122314d0_isrForCrosstalkSources_855400_37.stderr parsl.dataflow.dflow INFO: Task 2119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63778f1181776553b66ccb9e8180f7f7 with result from task 2119 parsl.dataflow.dflow DEBUG: Task 2119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1a79629f-03ee-49fa-9984-45434c30668f_isrForCrosstalkSources_855447_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/1a79629f-03ee-49fa-9984-45434c30668f_isrForCrosstalkSources_855447_21.stderr parsl.dataflow.dflow INFO: Task 2244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4536f2f3245c25a4d1cc9c05b3ed07a6 with result from task 2244 parsl.dataflow.dflow DEBUG: Task 2244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/367bb177-9512-434f-a7f2-d668daac1f5c_isrForCrosstalkSources_855428_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/367bb177-9512-434f-a7f2-d668daac1f5c_isrForCrosstalkSources_855428_28.stderr parsl.dataflow.dflow INFO: Task 2277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a33f51f5084bc536d641cfaa168a516 with result from task 2277 parsl.dataflow.dflow DEBUG: Task 2277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b4c3f855-b2ae-4615-be5b-fa1d4a4409e2_isrForCrosstalkSources_855423_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b4c3f855-b2ae-4615-be5b-fa1d4a4409e2_isrForCrosstalkSources_855423_31.stderr parsl.dataflow.dflow INFO: Task 2319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 109c5a5a596f2903b2099b6d4c0a6c01 with result from task 2319 parsl.dataflow.dflow DEBUG: Task 2319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/1c82362d-34cf-4667-bc1a-97dcfec1086e_isrForCrosstalkSources_855413_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/1c82362d-34cf-4667-bc1a-97dcfec1086e_isrForCrosstalkSources_855413_2.stderr parsl.dataflow.dflow INFO: Task 2296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05c2d36e9e38ded0b2eb708524657bfd with result from task 2296 parsl.dataflow.dflow DEBUG: Task 2296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/56dd1b58-893b-4add-bbd3-f0823a946421_isrForCrosstalkSources_855432_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/56dd1b58-893b-4add-bbd3-f0823a946421_isrForCrosstalkSources_855432_17.stderr parsl.dataflow.dflow INFO: Task 2254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d9953d6ddf13155ef72a03eb20d911c with result from task 2254 parsl.dataflow.dflow DEBUG: Task 2254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/528002b4-5226-45e7-bbda-1ec4b6258ade_isrForCrosstalkSources_855419_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/528002b4-5226-45e7-bbda-1ec4b6258ade_isrForCrosstalkSources_855419_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:37 proc_lsst.multi:146 [INFO] found job 16964148 in provider checkpoint proc_lsst.multi INFO: found job 16964148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79eef9ded790c15540a80846f4c5e748 with result from task 2126 parsl.dataflow.dflow DEBUG: Task 2126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/b82ace8d-f11b-49b8-8e40-090b288ae8a1_isrForCrosstalkSources_855417_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/b82ace8d-f11b-49b8-8e40-090b288ae8a1_isrForCrosstalkSources_855417_18.stderr parsl.dataflow.dflow INFO: Task 2189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e978e90bc28663eb31bbae8b3a003da with result from task 2189 parsl.dataflow.dflow DEBUG: Task 2189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c9e0bf92-bae3-4cb0-8053-b1af1d98d2e3_isrForCrosstalkSources_855450_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c9e0bf92-bae3-4cb0-8053-b1af1d98d2e3_isrForCrosstalkSources_855450_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:37 proc_lsst.multi:146 [INFO] found job 16964149 in provider checkpoint proc_lsst.multi INFO: found job 16964149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17eb2ffa1497443cc72dc59cdedb78e6 with result from task 2198 parsl.dataflow.dflow DEBUG: Task 2198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df101f5c-5cb9-4676-ac98-b695c27104d5_isrForCrosstalkSources_855422_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/df101f5c-5cb9-4676-ac98-b695c27104d5_isrForCrosstalkSources_855422_16.stderr parsl.dataflow.dflow INFO: Task 2144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94b9e717e0bc20d65d6720bdc196d4ad with result from task 2144 parsl.dataflow.dflow DEBUG: Task 2144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8f3d366e-613c-4610-ad5f-cf21321c8678_isrForCrosstalkSources_855450_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8f3d366e-613c-4610-ad5f-cf21321c8678_isrForCrosstalkSources_855450_55.stderr parsl.dataflow.dflow INFO: Task 1998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08185ec1cee834382cd40469adc59ee9 with result from task 1998 parsl.dataflow.dflow DEBUG: Task 1998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8cc765be-0e5f-48ed-ab5b-2212ec6febac_isrForCrosstalkSources_855398_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8cc765be-0e5f-48ed-ab5b-2212ec6febac_isrForCrosstalkSources_855398_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:37 proc_lsst.multi:146 [INFO] found job 16964150 in provider checkpoint proc_lsst.multi INFO: found job 16964150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36bdc47874cb5c0ed5b3409d70c46c22 with result from task 2255 parsl.dataflow.dflow DEBUG: Task 2255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b0374f13-3a89-4a4f-9d67-c41e37ace78f_isrForCrosstalkSources_855399_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b0374f13-3a89-4a4f-9d67-c41e37ace78f_isrForCrosstalkSources_855399_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:37 proc_lsst.multi:146 [INFO] found job 16964151 in provider checkpoint proc_lsst.multi INFO: found job 16964151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d87a9934a79539475ffa4e8e144b906 with result from task 2217 parsl.dataflow.dflow DEBUG: Task 2217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/40e4df54-4f6a-4c92-b66e-c7220300d49c_isrForCrosstalkSources_855446_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/40e4df54-4f6a-4c92-b66e-c7220300d49c_isrForCrosstalkSources_855446_4.stderr parsl.dataflow.dflow INFO: Task 2274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad06044013facbc2a7c3022a9a668d4d with result from task 2274 parsl.dataflow.dflow DEBUG: Task 2274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1161d7b3-e471-4450-903d-74eb9ef71828_isrForCrosstalkSources_855435_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1161d7b3-e471-4450-903d-74eb9ef71828_isrForCrosstalkSources_855435_47.stderr parsl.dataflow.dflow INFO: Task 2120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f00cc6e07f03933e9df02e48d65410 with result from task 2120 parsl.dataflow.dflow DEBUG: Task 2120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/12b64628-cc59-4c13-b63a-a7da76d755cd_isrForCrosstalkSources_855420_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/12b64628-cc59-4c13-b63a-a7da76d755cd_isrForCrosstalkSources_855420_22.stderr parsl.dataflow.dflow INFO: Task 2125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry effcedcc0a7fd2e0c849f37de2e52b4e with result from task 2125 parsl.dataflow.dflow DEBUG: Task 2125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6182ed0c-a33d-4ad1-b55d-a9b0ceb8dfd9_isrForCrosstalkSources_855425_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/6182ed0c-a33d-4ad1-b55d-a9b0ceb8dfd9_isrForCrosstalkSources_855425_7.stderr parsl.dataflow.dflow INFO: Task 2187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d22b213a38abdaa604c0d45c44767a4 with result from task 2187 parsl.dataflow.dflow DEBUG: Task 2187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6fdc2b77-532c-498f-b8ea-49ef8510ebe7_isrForCrosstalkSources_855380_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6fdc2b77-532c-498f-b8ea-49ef8510ebe7_isrForCrosstalkSources_855380_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:38 proc_lsst.multi:146 [INFO] found job 16964152 in provider checkpoint proc_lsst.multi INFO: found job 16964152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1601410de90a6ef55a4eabe417b76c7 with result from task 2363 parsl.dataflow.dflow DEBUG: Task 2363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/88b612f0-ae8b-4980-8ad8-7e93aa6e709d_isrForCrosstalkSources_855399_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/88b612f0-ae8b-4980-8ad8-7e93aa6e709d_isrForCrosstalkSources_855399_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:38 proc_lsst.multi:146 [INFO] found job 16964153 in provider checkpoint proc_lsst.multi INFO: found job 16964153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bc99acaa338b713728a741f1fe012f8 with result from task 2305 parsl.dataflow.dflow DEBUG: Task 2305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01905662-aedb-49de-9459-d4bf4cf76c59_isrForCrosstalkSources_855417_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/01905662-aedb-49de-9459-d4bf4cf76c59_isrForCrosstalkSources_855417_3.stderr parsl.dataflow.dflow INFO: Task 2066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d167ec207f26f507708bfc0726affbf3 with result from task 2066 parsl.dataflow.dflow DEBUG: Task 2066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ac295d63-b765-474f-b039-0ccd71f1fa6b_isrForCrosstalkSources_855390_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ac295d63-b765-474f-b039-0ccd71f1fa6b_isrForCrosstalkSources_855390_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:38 proc_lsst.multi:146 [INFO] found job 16964154 in provider checkpoint proc_lsst.multi INFO: found job 16964154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd6159fe1a449e1b3933c7a4f8ea4613 with result from task 2419 parsl.dataflow.dflow DEBUG: Task 2419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab10fedd-531f-419d-a1a1-f56d0a92b0d1_isrForCrosstalkSources_855437_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ab10fedd-531f-419d-a1a1-f56d0a92b0d1_isrForCrosstalkSources_855437_45.stderr parsl.dataflow.dflow INFO: Task 2312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d1f75ba249e4fe8743097ec8716ecb with result from task 2312 parsl.dataflow.dflow DEBUG: Task 2312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/96709b38-8441-495c-a093-4241bb95706e_isrForCrosstalkSources_855450_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/96709b38-8441-495c-a093-4241bb95706e_isrForCrosstalkSources_855450_45.stderr parsl.dataflow.dflow INFO: Task 2177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3deb2ee4ad906e82d47d91d45d6777fc with result from task 2177 parsl.dataflow.dflow DEBUG: Task 2177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2da315fc-47b7-4d48-a299-754e0ae0cded_isrForCrosstalkSources_855419_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2da315fc-47b7-4d48-a299-754e0ae0cded_isrForCrosstalkSources_855419_59.stderr parsl.dataflow.dflow INFO: Task 2193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe08f8bfcbfd17c36e360f64fa2396a with result from task 2193 parsl.dataflow.dflow DEBUG: Task 2193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c947b04e-baef-4c96-bd90-968558c853c7_isrForCrosstalkSources_855393_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c947b04e-baef-4c96-bd90-968558c853c7_isrForCrosstalkSources_855393_15.stderr parsl.dataflow.dflow INFO: Task 1972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9267df5cc8d037bb859848b873201cdf with result from task 1972 parsl.dataflow.dflow DEBUG: Task 1972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a82587f8-603a-4ca7-97a3-443abad24d43_isrForCrosstalkSources_855415_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a82587f8-603a-4ca7-97a3-443abad24d43_isrForCrosstalkSources_855415_31.stderr parsl.dataflow.dflow INFO: Task 2180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d9e674d6de3c879085d960e380003c0 with result from task 2180 parsl.dataflow.dflow DEBUG: Task 2180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea4989c1-6813-465a-a583-69eb525c053c_isrForCrosstalkSources_855390_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea4989c1-6813-465a-a583-69eb525c053c_isrForCrosstalkSources_855390_21.stderr parsl.dataflow.dflow INFO: Task 2342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9079409a6da34e8ad6021bbf24f80c8 with result from task 2342 parsl.dataflow.dflow DEBUG: Task 2342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/590825ca-fdca-4682-9250-ea22174a8814_isrForCrosstalkSources_855380_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/590825ca-fdca-4682-9250-ea22174a8814_isrForCrosstalkSources_855380_7.stderr parsl.dataflow.dflow INFO: Task 2306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eae07189e92b81da9a806cfcc6cc152 with result from task 2306 parsl.dataflow.dflow DEBUG: Task 2306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/12ea2219-55ae-4399-bc79-e250462d1e7c_isrForCrosstalkSources_855432_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/12ea2219-55ae-4399-bc79-e250462d1e7c_isrForCrosstalkSources_855432_22.stderr parsl.dataflow.dflow INFO: Task 2146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 117ddd224ff0e9d72f6ad46ef1931079 with result from task 2146 parsl.dataflow.dflow DEBUG: Task 2146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/4ea6cc67-fe22-40cc-b85f-6ba47067b0b6_isrForCrosstalkSources_855396_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/4ea6cc67-fe22-40cc-b85f-6ba47067b0b6_isrForCrosstalkSources_855396_22.stderr parsl.dataflow.dflow INFO: Task 2300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f347e51b63fc08bd5f0cfb18f518ceb3 with result from task 2300 parsl.dataflow.dflow DEBUG: Task 2300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e69a528c-f3dc-456e-83fa-95ef92d78f83_isrForCrosstalkSources_855421_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e69a528c-f3dc-456e-83fa-95ef92d78f83_isrForCrosstalkSources_855421_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:38 proc_lsst.multi:146 [INFO] found job 16964155 in provider checkpoint proc_lsst.multi INFO: found job 16964155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f42ecf45eb8f7aaad9a6986d048dc9d with result from task 2149 parsl.dataflow.dflow DEBUG: Task 2149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/95c51693-86ee-4210-bcf3-d34c93302693_isrForCrosstalkSources_855433_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/95c51693-86ee-4210-bcf3-d34c93302693_isrForCrosstalkSources_855433_42.stderr parsl.dataflow.dflow INFO: Task 1965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa27aa6a22fad22be2797d57e8ee283b with result from task 1965 parsl.dataflow.dflow DEBUG: Task 1965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e9e96450-7c54-47fa-a711-9a527c21f26c_isrForCrosstalkSources_855407_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/e9e96450-7c54-47fa-a711-9a527c21f26c_isrForCrosstalkSources_855407_30.stderr parsl.dataflow.dflow INFO: Task 2253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f516d9acea88958457882e939e1baa1c with result from task 2253 parsl.dataflow.dflow DEBUG: Task 2253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/62871ec3-1c9f-485f-828e-8e9b5f317719_isrForCrosstalkSources_855426_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/62871ec3-1c9f-485f-828e-8e9b5f317719_isrForCrosstalkSources_855426_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:39 proc_lsst.multi:146 [INFO] found job 16964156 in provider checkpoint proc_lsst.multi INFO: found job 16964156 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee895acb3fae3796ce25dd169c8b7aa2 with result from task 2249 parsl.dataflow.dflow DEBUG: Task 2249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/53b76f67-7f29-4657-8913-45d20bdd073b_isrForCrosstalkSources_855434_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/53b76f67-7f29-4657-8913-45d20bdd073b_isrForCrosstalkSources_855434_23.stderr parsl.dataflow.dflow INFO: Task 2358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fad1c7ed2dfccd15edbd6b428f7539c0 with result from task 2358 parsl.dataflow.dflow DEBUG: Task 2358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f0a3f829-d0e0-4a57-b445-3b998ff4e9aa_isrForCrosstalkSources_855437_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f0a3f829-d0e0-4a57-b445-3b998ff4e9aa_isrForCrosstalkSources_855437_13.stderr parsl.dataflow.dflow INFO: Task 2116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87b9a98adf1840572b7d04768ffd49ed with result from task 2116 parsl.dataflow.dflow DEBUG: Task 2116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5802e5fc-6368-4444-abb5-7220750a9414_isrForCrosstalkSources_855444_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5802e5fc-6368-4444-abb5-7220750a9414_isrForCrosstalkSources_855444_20.stderr parsl.dataflow.dflow INFO: Task 2129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07be29bd091113ca1ed2deeb9f0e2090 with result from task 2129 parsl.dataflow.dflow DEBUG: Task 2129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04cb6fc9-33c7-4457-96bd-cd068b6e13ee_isrForCrosstalkSources_855440_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04cb6fc9-33c7-4457-96bd-cd068b6e13ee_isrForCrosstalkSources_855440_21.stderr parsl.dataflow.dflow INFO: Task 2087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49e0539168124d1165f5709ca0341aa4 with result from task 2087 parsl.dataflow.dflow DEBUG: Task 2087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cc7299b6-5f81-4cfc-a811-b8b10562cd7e_isrForCrosstalkSources_855412_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cc7299b6-5f81-4cfc-a811-b8b10562cd7e_isrForCrosstalkSources_855412_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:39 proc_lsst.multi:146 [INFO] found job 16964157 in provider checkpoint proc_lsst.multi INFO: found job 16964157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b67ef1f1a5e883744da137f7355f52c4 with result from task 2243 parsl.dataflow.dflow DEBUG: Task 2243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ced89f93-1660-446d-b3d1-d039048d1645_isrForCrosstalkSources_855457_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ced89f93-1660-446d-b3d1-d039048d1645_isrForCrosstalkSources_855457_60.stderr parsl.dataflow.dflow INFO: Task 2195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7fd1eba183f3c143a0ec60a106c0d4b with result from task 2195 parsl.dataflow.dflow DEBUG: Task 2195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c545b8da-4fa6-420b-b196-d701340da825_isrForCrosstalkSources_855403_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c545b8da-4fa6-420b-b196-d701340da825_isrForCrosstalkSources_855403_9.stderr parsl.dataflow.dflow INFO: Task 2049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0acc5b3639e7921bfee313e2b028d88c with result from task 2049 parsl.dataflow.dflow DEBUG: Task 2049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/924fafe3-7379-43f5-9ca0-c389247014a5_isrForCrosstalkSources_855398_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/924fafe3-7379-43f5-9ca0-c389247014a5_isrForCrosstalkSources_855398_53.stderr parsl.dataflow.dflow INFO: Task 2302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26d0cd92127fe19286583259ef99ec39 with result from task 2302 parsl.dataflow.dflow DEBUG: Task 2302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e9b1c043-10ef-43a9-b01c-61906ba71dbd_isrForCrosstalkSources_855433_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e9b1c043-10ef-43a9-b01c-61906ba71dbd_isrForCrosstalkSources_855433_20.stderr parsl.dataflow.dflow INFO: Task 2153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f308212660a770a7942bb4b71fdc22 with result from task 2153 parsl.dataflow.dflow DEBUG: Task 2153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/342d3c40-463a-4d66-9676-5698ff60cc15_isrForCrosstalkSources_855436_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/342d3c40-463a-4d66-9676-5698ff60cc15_isrForCrosstalkSources_855436_37.stderr parsl.dataflow.dflow INFO: Task 2094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae4f436f20ebb272c070c025f235f6d with result from task 2094 parsl.dataflow.dflow DEBUG: Task 2094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4bef055c-15cf-47f9-8c71-1211adaff5f5_isrForCrosstalkSources_855401_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4bef055c-15cf-47f9-8c71-1211adaff5f5_isrForCrosstalkSources_855401_49.stderr parsl.dataflow.dflow INFO: Task 2062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0f162b2cfe4b622d6bcf2b9d6fb252 with result from task 2062 parsl.dataflow.dflow DEBUG: Task 2062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/bf548217-67d9-41e7-8cd0-28d5b454a586_isrForCrosstalkSources_855411_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/bf548217-67d9-41e7-8cd0-28d5b454a586_isrForCrosstalkSources_855411_41.stderr parsl.dataflow.dflow INFO: Task 2155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed38a014d39c491c4510ba375708e327 with result from task 2155 parsl.dataflow.dflow DEBUG: Task 2155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/87141871-97b7-4c07-b1ef-8aad7c17d70e_isrForCrosstalkSources_855457_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/87141871-97b7-4c07-b1ef-8aad7c17d70e_isrForCrosstalkSources_855457_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:39 proc_lsst.multi:146 [INFO] found job 16964158 in provider checkpoint proc_lsst.multi INFO: found job 16964158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c502f1e016015e5a2ada163b34dd6114 with result from task 2235 parsl.dataflow.dflow DEBUG: Task 2235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8bbea5fa-a479-4df0-982f-662700304a06_isrForCrosstalkSources_855431_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/8bbea5fa-a479-4df0-982f-662700304a06_isrForCrosstalkSources_855431_8.stderr parsl.dataflow.dflow INFO: Task 2236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09c3c5ae3bf8ab84624919c24906412f with result from task 2236 parsl.dataflow.dflow DEBUG: Task 2236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3b6206a4-c8f2-4398-9fb6-e9ec9ac29ce4_isrForCrosstalkSources_855405_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3b6206a4-c8f2-4398-9fb6-e9ec9ac29ce4_isrForCrosstalkSources_855405_36.stderr parsl.dataflow.dflow INFO: Task 2127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97c90f751ab6e153f86f938637e2cc3f with result from task 2127 parsl.dataflow.dflow DEBUG: Task 2127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/83cf2b45-3431-4bd9-a790-217c41d957b2_isrForCrosstalkSources_855436_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/83cf2b45-3431-4bd9-a790-217c41d957b2_isrForCrosstalkSources_855436_7.stderr parsl.dataflow.dflow INFO: Task 2281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f23227e5c9ae378ba1c998ce6344ade with result from task 2281 parsl.dataflow.dflow DEBUG: Task 2281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/1ed9e1f0-513c-4128-96d9-f7117ae9b17d_isrForCrosstalkSources_855411_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/1ed9e1f0-513c-4128-96d9-f7117ae9b17d_isrForCrosstalkSources_855411_54.stderr parsl.dataflow.dflow INFO: Task 2084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6066f2cad231c0b0f21794ecc00b5302 with result from task 2084 parsl.dataflow.dflow DEBUG: Task 2084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5bff9dd8-301a-427d-b435-3aac42e6a186_isrForCrosstalkSources_855391_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5bff9dd8-301a-427d-b435-3aac42e6a186_isrForCrosstalkSources_855391_10.stderr parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 135c26aa8a13a1a2ca25b36c054e4150 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/59733803-5aa2-4ea0-a20e-bc541644982b_isrForCrosstalkSources_855397_42.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/59733803-5aa2-4ea0-a20e-bc541644982b_isrForCrosstalkSources_855397_42.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 940fd429d83d5853bdcb530194ad0fdd with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9682fd69-eded-4c2e-b3ac-aa5fcaf4ab95_isrForCrosstalkSources_855411_32.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9682fd69-eded-4c2e-b3ac-aa5fcaf4ab95_isrForCrosstalkSources_855411_32.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1692c3ea7c4ec28989298663784cd3 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/5181e166-7186-42d4-b12e-3c168a7ff679_isrForCrosstalkSources_855387_11.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/5181e166-7186-42d4-b12e-3c168a7ff679_isrForCrosstalkSources_855387_11.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb1694785b3f841c083812e8ed4956b4 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c611efd7-9e00-4dcc-8f30-d53913701ee6_isrForCrosstalkSources_855390_9.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c611efd7-9e00-4dcc-8f30-d53913701ee6_isrForCrosstalkSources_855390_9.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0573f225e9e914e1f193ca5a075b1d with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9bfec801-c10c-4222-b9b4-1722d74bfc3d_isrForCrosstalkSources_855432_16.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9bfec801-c10c-4222-b9b4-1722d74bfc3d_isrForCrosstalkSources_855432_16.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa4c338e4fcaac6af1a2b2f948d75838 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/70f13dba-0553-4bbf-b2cd-9cbefbc02f8e_isrForCrosstalkSources_855418_6.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/70f13dba-0553-4bbf-b2cd-9cbefbc02f8e_isrForCrosstalkSources_855418_6.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c771fcd147bb4e242a5e315d8d4c179b with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/b13cf44c-6ab3-4e17-b0f5-8457bd371cd0_isrForCrosstalkSources_855412_1.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/b13cf44c-6ab3-4e17-b0f5-8457bd371cd0_isrForCrosstalkSources_855412_1.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574da1913abecd7d723c6ee757cb4644 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a62a820a-518b-462e-ae64-afaf59ffdba6_isrForCrosstalkSources_855383_13.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a62a820a-518b-462e-ae64-afaf59ffdba6_isrForCrosstalkSources_855383_13.stderr parsl.dataflow.dflow INFO: Task 2097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cbe530143c1e8ef3e009de64b291e1f with result from task 2097 parsl.dataflow.dflow DEBUG: Task 2097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0996d724-f46b-4a1a-af2d-40964ab2e514_isrForCrosstalkSources_855387_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0996d724-f46b-4a1a-af2d-40964ab2e514_isrForCrosstalkSources_855387_5.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5458ad511114e483d84cddcc89825211 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ea112bf-ddb9-4618-a40e-7774b17adc6e_isrForCrosstalkSources_855396_37.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ea112bf-ddb9-4618-a40e-7774b17adc6e_isrForCrosstalkSources_855396_37.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a423d8489ee2aa575e9f2fae8cfdc51 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/43205f56-7575-4cae-9b85-9d37074aa7b5_isrForCrosstalkSources_855404_21.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/43205f56-7575-4cae-9b85-9d37074aa7b5_isrForCrosstalkSources_855404_21.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86bde349b5d68d600bde2e391301d1f2 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3d6a3b07-1864-4af8-b063-a011b0e2b577_isrForCrosstalkSources_855447_2.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3d6a3b07-1864-4af8-b063-a011b0e2b577_isrForCrosstalkSources_855447_2.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d00b3c41045a6c56adbe95cbf3c5a339 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f88dbbe9-66ea-4f8d-96d9-6d54b12290d3_isrForCrosstalkSources_855437_20.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f88dbbe9-66ea-4f8d-96d9-6d54b12290d3_isrForCrosstalkSources_855437_20.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb5098a900fc8484422da894df0efe53 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/39e0b0ae-3d42-4b73-9189-635ceebe31aa_isrForCrosstalkSources_855383_45.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/39e0b0ae-3d42-4b73-9189-635ceebe31aa_isrForCrosstalkSources_855383_45.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9824c38738d9408971aead72a7725d4 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a27e47ff-a4df-4cb6-a308-c9e9e623a754_isrForCrosstalkSources_855401_59.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a27e47ff-a4df-4cb6-a308-c9e9e623a754_isrForCrosstalkSources_855401_59.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79de9835c19a6a947dfd159ae496845a with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5534c7df-9ceb-4a4d-98a1-a79a7bb2bb03_isrForCrosstalkSources_855451_36.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/5534c7df-9ceb-4a4d-98a1-a79a7bb2bb03_isrForCrosstalkSources_855451_36.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d07eaa5a954d441748ccf654c34032d2 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/20722a66-6e8e-4e62-8561-a5d65c051a10_isrForCrosstalkSources_855442_15.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/20722a66-6e8e-4e62-8561-a5d65c051a10_isrForCrosstalkSources_855442_15.stderr parsl.dataflow.dflow INFO: Task 2209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c862f54a86002b575ebc288c76955698 with result from task 2209 parsl.dataflow.dflow DEBUG: Task 2209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f3c5ebaf-535f-464d-89fd-db04f3be8677_isrForCrosstalkSources_855392_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f3c5ebaf-535f-464d-89fd-db04f3be8677_isrForCrosstalkSources_855392_22.stderr parsl.dataflow.dflow INFO: Task 2130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3afb9febb6b3126711d6d69f7135cccd with result from task 2130 parsl.dataflow.dflow DEBUG: Task 2130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/54674d2f-fc09-467c-bced-7258793982bb_isrForCrosstalkSources_855422_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/54674d2f-fc09-467c-bced-7258793982bb_isrForCrosstalkSources_855422_10.stderr parsl.dataflow.dflow INFO: Task 2164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f9d8fc728a6a2e0428fac1097a459b1 with result from task 2164 parsl.dataflow.dflow DEBUG: Task 2164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/363a2cee-6061-4100-88fb-e1a06b242fa8_isrForCrosstalkSources_855435_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/363a2cee-6061-4100-88fb-e1a06b242fa8_isrForCrosstalkSources_855435_22.stderr parsl.dataflow.dflow INFO: Task 2272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a0475a9ef88c6b825939ab28a1a65f with result from task 2272 parsl.dataflow.dflow DEBUG: Task 2272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e03790f-bee2-4b54-adcd-88fda00bdffb_isrForCrosstalkSources_855380_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e03790f-bee2-4b54-adcd-88fda00bdffb_isrForCrosstalkSources_855380_16.stderr parsl.dataflow.dflow INFO: Task 2169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b95d28af1f383ec57f5fa92ab6faea9 with result from task 2169 parsl.dataflow.dflow DEBUG: Task 2169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b9329f7b-85a8-4711-8c68-f9b586957f3e_isrForCrosstalkSources_855399_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/b9329f7b-85a8-4711-8c68-f9b586957f3e_isrForCrosstalkSources_855399_36.stderr parsl.dataflow.dflow INFO: Task 2268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa4db97a41d592bb13fbfed5368157f7 with result from task 2268 parsl.dataflow.dflow DEBUG: Task 2268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0d9f88ff-7435-43dc-8dee-c7fdd9761ba9_isrForCrosstalkSources_855418_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0d9f88ff-7435-43dc-8dee-c7fdd9761ba9_isrForCrosstalkSources_855418_33.stderr parsl.dataflow.dflow INFO: Task 2157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81c607f13e19771bf533f48e9af3edf8 with result from task 2157 parsl.dataflow.dflow DEBUG: Task 2157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/664b9362-dc40-4600-9b25-88560ed90756_isrForCrosstalkSources_855401_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/664b9362-dc40-4600-9b25-88560ed90756_isrForCrosstalkSources_855401_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:40 proc_lsst.multi:146 [INFO] found job 16964159 in provider checkpoint proc_lsst.multi INFO: found job 16964159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68621b2614737d3d02ecbd3f2b5444e with result from task 2339 parsl.dataflow.dflow DEBUG: Task 2339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7011596e-0fda-4dce-a284-6f44e8aa4da4_isrForCrosstalkSources_855401_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7011596e-0fda-4dce-a284-6f44e8aa4da4_isrForCrosstalkSources_855401_51.stderr parsl.dataflow.dflow INFO: Task 2109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bba360c61833907b2fa0aab37e2b3333 with result from task 2109 parsl.dataflow.dflow DEBUG: Task 2109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/0fa0bb6c-26b5-4343-8ef2-7f91f1141c0b_isrForCrosstalkSources_855453_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/0fa0bb6c-26b5-4343-8ef2-7f91f1141c0b_isrForCrosstalkSources_855453_30.stderr parsl.dataflow.dflow INFO: Task 2171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3c9145ae85b4d63ef4dff2b1f32d75 with result from task 2171 parsl.dataflow.dflow DEBUG: Task 2171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e58045cf-c247-454b-b2b3-142c7e860d76_isrForCrosstalkSources_855458_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e58045cf-c247-454b-b2b3-142c7e860d76_isrForCrosstalkSources_855458_55.stderr parsl.dataflow.dflow INFO: Task 2289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bac71f3323b77c8292a6a58a673422a with result from task 2289 parsl.dataflow.dflow DEBUG: Task 2289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/74ae3be6-e371-4a4f-88ac-46795a1c6876_isrForCrosstalkSources_855446_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/74ae3be6-e371-4a4f-88ac-46795a1c6876_isrForCrosstalkSources_855446_46.stderr parsl.dataflow.dflow INFO: Task 2389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f51b2bd83c6e27748d8d3f05678f90 with result from task 2389 parsl.dataflow.dflow DEBUG: Task 2389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c286d705-ad70-4099-8743-5209e1053847_isrForCrosstalkSources_855399_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c286d705-ad70-4099-8743-5209e1053847_isrForCrosstalkSources_855399_29.stderr parsl.dataflow.dflow INFO: Task 2276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd309e4077ec1447bdda219cdf856e24 with result from task 2276 parsl.dataflow.dflow DEBUG: Task 2276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/90d887cf-8d23-4919-bb97-8994563ff1ca_isrForCrosstalkSources_855433_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/90d887cf-8d23-4919-bb97-8994563ff1ca_isrForCrosstalkSources_855433_36.stderr parsl.dataflow.dflow INFO: Task 2118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 975b76bb72b95e534fef00a3d0d702e0 with result from task 2118 parsl.dataflow.dflow DEBUG: Task 2118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed6c8424-28ae-4c82-94c6-fdcc81bed214_isrForCrosstalkSources_855427_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/ed6c8424-28ae-4c82-94c6-fdcc81bed214_isrForCrosstalkSources_855427_3.stderr parsl.dataflow.dflow INFO: Task 2257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1adac9899b9d118aca14ec954ac45308 with result from task 2257 parsl.dataflow.dflow DEBUG: Task 2257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b5e5b0c4-6101-4050-be8b-55d0c94db78b_isrForCrosstalkSources_855427_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b5e5b0c4-6101-4050-be8b-55d0c94db78b_isrForCrosstalkSources_855427_28.stderr parsl.dataflow.dflow INFO: Task 2110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f501260affaea8a726e10213938343a with result from task 2110 parsl.dataflow.dflow DEBUG: Task 2110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c4524680-65b3-462f-9e1c-144029330489_isrForCrosstalkSources_855441_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c4524680-65b3-462f-9e1c-144029330489_isrForCrosstalkSources_855441_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:41 proc_lsst.multi:146 [INFO] found job 16964160 in provider checkpoint proc_lsst.multi INFO: found job 16964160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbb3e7dc389c1b860f5f55152895b717 with result from task 2168 parsl.dataflow.dflow DEBUG: Task 2168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/347a1464-336a-451d-8772-50ee6f540ef5_isrForCrosstalkSources_855422_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/347a1464-336a-451d-8772-50ee6f540ef5_isrForCrosstalkSources_855422_2.stderr parsl.dataflow.dflow INFO: Task 2304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41a490ccdd567acf9982084bf4c922a8 with result from task 2304 parsl.dataflow.dflow DEBUG: Task 2304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6094b6f4-8078-4e4b-9e05-fdcf2a3cc8d4_isrForCrosstalkSources_855391_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6094b6f4-8078-4e4b-9e05-fdcf2a3cc8d4_isrForCrosstalkSources_855391_36.stderr parsl.dataflow.dflow INFO: Task 2362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1d49e3fa22755de2cab4ac971d6064 with result from task 2362 parsl.dataflow.dflow DEBUG: Task 2362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/446a21b3-aeff-468e-a691-3ae15ac6083c_isrForCrosstalkSources_855380_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/446a21b3-aeff-468e-a691-3ae15ac6083c_isrForCrosstalkSources_855380_35.stderr parsl.dataflow.dflow INFO: Task 2408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210e8e6428c84428f5644017fa9cfd12 with result from task 2408 parsl.dataflow.dflow DEBUG: Task 2408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/42f26daa-0ede-4330-be4b-96d8842b2420_isrForCrosstalkSources_855400_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/42f26daa-0ede-4330-be4b-96d8842b2420_isrForCrosstalkSources_855400_10.stderr parsl.dataflow.dflow INFO: Task 2242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99aae585fdd2053702983883ac459039 with result from task 2242 parsl.dataflow.dflow DEBUG: Task 2242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81709fa0-8bc8-47a7-afbd-999d90dffc84_isrForCrosstalkSources_855443_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81709fa0-8bc8-47a7-afbd-999d90dffc84_isrForCrosstalkSources_855443_55.stderr parsl.dataflow.dflow INFO: Task 2346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e195f91c14c32048451f92f723bc6277 with result from task 2346 parsl.dataflow.dflow DEBUG: Task 2346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/57256f53-ba40-47da-9e2d-0130ce4afd69_isrForCrosstalkSources_855446_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/57256f53-ba40-47da-9e2d-0130ce4afd69_isrForCrosstalkSources_855446_40.stderr parsl.dataflow.dflow INFO: Task 2269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd81910d6c35be31760f00d2fb2071fb with result from task 2269 parsl.dataflow.dflow DEBUG: Task 2269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72f33fd1-2050-4c9f-8dbd-5251e22e557f_isrForCrosstalkSources_855417_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72f33fd1-2050-4c9f-8dbd-5251e22e557f_isrForCrosstalkSources_855417_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:41 proc_lsst.multi:146 [INFO] found job 16964161 in provider checkpoint proc_lsst.multi INFO: found job 16964161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792bc1a16e5298a794afe92d93187b3e with result from task 2357 parsl.dataflow.dflow DEBUG: Task 2357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/db2747b5-b320-4364-888b-97b7a88c213f_isrForCrosstalkSources_855449_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/db2747b5-b320-4364-888b-97b7a88c213f_isrForCrosstalkSources_855449_12.stderr parsl.dataflow.dflow INFO: Task 2347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 375f962949a917040fc39e08396ea8b2 with result from task 2347 parsl.dataflow.dflow DEBUG: Task 2347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4cb0a221-d636-44ea-9652-300cc01714ec_isrForCrosstalkSources_855425_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/4cb0a221-d636-44ea-9652-300cc01714ec_isrForCrosstalkSources_855425_52.stderr parsl.dataflow.dflow INFO: Task 2128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 781f97ddfa3dcaa3f730356de7823fb0 with result from task 2128 parsl.dataflow.dflow DEBUG: Task 2128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5174f3f5-39f8-4669-990d-133c31470342_isrForCrosstalkSources_855436_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5174f3f5-39f8-4669-990d-133c31470342_isrForCrosstalkSources_855436_16.stderr parsl.dataflow.dflow INFO: Task 2140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef04ba1594dcecf19ad0d20f22fe202f with result from task 2140 parsl.dataflow.dflow DEBUG: Task 2140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/524683ad-04da-4d11-bb13-693d4127ce61_isrForCrosstalkSources_855408_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/524683ad-04da-4d11-bb13-693d4127ce61_isrForCrosstalkSources_855408_59.stderr parsl.dataflow.dflow INFO: Task 2117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4abe1227f69d50b122c23833bd4afa90 with result from task 2117 parsl.dataflow.dflow DEBUG: Task 2117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/38058cb9-fe08-458c-8ff2-afab29e609e6_isrForCrosstalkSources_855429_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/38058cb9-fe08-458c-8ff2-afab29e609e6_isrForCrosstalkSources_855429_42.stderr parsl.dataflow.dflow INFO: Task 2317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6980c70488d3c1d1e0aaa95aa5079a98 with result from task 2317 parsl.dataflow.dflow DEBUG: Task 2317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/20be52d6-d698-402d-81c6-ddd5691f3a10_isrForCrosstalkSources_855406_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/20be52d6-d698-402d-81c6-ddd5691f3a10_isrForCrosstalkSources_855406_7.stderr parsl.dataflow.dflow INFO: Task 2154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 769bcd040da8aa75ca4177d72b6a5e75 with result from task 2154 parsl.dataflow.dflow DEBUG: Task 2154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bdf01289-64f1-490c-84a0-fe0d9692eecf_isrForCrosstalkSources_855410_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bdf01289-64f1-490c-84a0-fe0d9692eecf_isrForCrosstalkSources_855410_5.stderr parsl.dataflow.dflow INFO: Task 2283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b862a836b2d48fdfc83c40d41115344f with result from task 2283 parsl.dataflow.dflow DEBUG: Task 2283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1303cfd8-7714-4ab9-8403-256704e5e503_isrForCrosstalkSources_855391_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1303cfd8-7714-4ab9-8403-256704e5e503_isrForCrosstalkSources_855391_4.stderr parsl.dataflow.dflow INFO: Task 2215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f505fd64a30645e4ce120e978ef57803 with result from task 2215 parsl.dataflow.dflow DEBUG: Task 2215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b0ec1c55-4019-435e-92af-d59dbf589e1a_isrForCrosstalkSources_855439_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b0ec1c55-4019-435e-92af-d59dbf589e1a_isrForCrosstalkSources_855439_8.stderr parsl.dataflow.dflow INFO: Task 2381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d111f0a3b6f65be71444e621d1dc9780 with result from task 2381 parsl.dataflow.dflow DEBUG: Task 2381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/bd3b100f-c840-40b0-9049-ebbf4726b77c_isrForCrosstalkSources_855389_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/bd3b100f-c840-40b0-9049-ebbf4726b77c_isrForCrosstalkSources_855389_39.stderr parsl.dataflow.dflow INFO: Task 2356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69bae0a3ec2c7a1fa33301063084cf0e with result from task 2356 parsl.dataflow.dflow DEBUG: Task 2356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cbfe18ed-3651-476d-9f74-0292e14100ee_isrForCrosstalkSources_855407_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cbfe18ed-3651-476d-9f74-0292e14100ee_isrForCrosstalkSources_855407_6.stderr parsl.dataflow.dflow INFO: Task 2186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c874ea4b3d30aa97b7ff2e5ba9f8e56 with result from task 2186 parsl.dataflow.dflow DEBUG: Task 2186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b2abcb4b-bc18-4304-aae0-8da58382202d_isrForCrosstalkSources_855428_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/b2abcb4b-bc18-4304-aae0-8da58382202d_isrForCrosstalkSources_855428_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:42 proc_lsst.multi:146 [INFO] found job 16964162 in provider checkpoint proc_lsst.multi INFO: found job 16964162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b213b03968f44dd2e18bdd2494c274b with result from task 2309 parsl.dataflow.dflow DEBUG: Task 2309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8c55c8a-3b3e-44dc-83b0-b36df2ba9439_isrForCrosstalkSources_855441_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8c55c8a-3b3e-44dc-83b0-b36df2ba9439_isrForCrosstalkSources_855441_30.stderr parsl.dataflow.dflow INFO: Task 2341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f22a58d7530b8a513005ac222a59cefb with result from task 2341 parsl.dataflow.dflow DEBUG: Task 2341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/4999fcc3-33e2-4c33-a09e-0c122fc35a8e_isrForCrosstalkSources_855380_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/4999fcc3-33e2-4c33-a09e-0c122fc35a8e_isrForCrosstalkSources_855380_28.stderr parsl.dataflow.dflow INFO: Task 2273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97233c537c13abb9f241be3d71ae4f52 with result from task 2273 parsl.dataflow.dflow DEBUG: Task 2273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/187d73bd-c05f-4f96-af65-d4876ecbbdea_isrForCrosstalkSources_855435_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/187d73bd-c05f-4f96-af65-d4876ecbbdea_isrForCrosstalkSources_855435_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:42 proc_lsst.multi:146 [INFO] found job 16964163 in provider checkpoint proc_lsst.multi INFO: found job 16964163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e83cbd46c3c33fb3d6c43e517d19e9b3 with result from task 2106 parsl.dataflow.dflow DEBUG: Task 2106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f606330f-c16c-4bea-95d6-8b1201a05685_isrForCrosstalkSources_855388_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f606330f-c16c-4bea-95d6-8b1201a05685_isrForCrosstalkSources_855388_36.stderr parsl.dataflow.dflow INFO: Task 2068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20169b051239e9a92cc58a04545e3b09 with result from task 2068 parsl.dataflow.dflow DEBUG: Task 2068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/533f88fa-62c0-47b1-9866-de08bf88777f_isrForCrosstalkSources_855418_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/533f88fa-62c0-47b1-9866-de08bf88777f_isrForCrosstalkSources_855418_59.stderr parsl.dataflow.dflow INFO: Task 2278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e530899d17e53a76a26e79f8a35eaa3 with result from task 2278 parsl.dataflow.dflow DEBUG: Task 2278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e81a3ca3-9ef1-4f17-a17a-1ebf688a41c7_isrForCrosstalkSources_855436_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/e81a3ca3-9ef1-4f17-a17a-1ebf688a41c7_isrForCrosstalkSources_855436_5.stderr parsl.dataflow.dflow INFO: Task 2112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f581e6d2ae1595ac0964fd242076de0f with result from task 2112 parsl.dataflow.dflow DEBUG: Task 2112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a50f6e6a-3be2-4e23-a7ae-834a507144fa_isrForCrosstalkSources_855409_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/a50f6e6a-3be2-4e23-a7ae-834a507144fa_isrForCrosstalkSources_855409_61.stderr parsl.dataflow.dflow INFO: Task 2070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17b95e2a299bc50350bfd1033ddd73fc with result from task 2070 parsl.dataflow.dflow DEBUG: Task 2070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/62425bb3-91b1-40dd-a922-6f1d51c6ae2e_isrForCrosstalkSources_855415_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/62425bb3-91b1-40dd-a922-6f1d51c6ae2e_isrForCrosstalkSources_855415_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:43 proc_lsst.multi:146 [INFO] found job 16964164 in provider checkpoint proc_lsst.multi INFO: found job 16964164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 413de9da3fd15d417341481f3a50e678 with result from task 2194 parsl.dataflow.dflow DEBUG: Task 2194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e87ffe98-44c8-4e9a-b14c-eed52eec43a3_isrForCrosstalkSources_855424_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e87ffe98-44c8-4e9a-b14c-eed52eec43a3_isrForCrosstalkSources_855424_59.stderr parsl.dataflow.dflow INFO: Task 2361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b068660fb2a9ae16930325e13ee1425 with result from task 2361 parsl.dataflow.dflow DEBUG: Task 2361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/a7be44d0-3d1f-459a-a153-da0fff078fc3_isrForCrosstalkSources_855424_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/a7be44d0-3d1f-459a-a153-da0fff078fc3_isrForCrosstalkSources_855424_2.stderr parsl.dataflow.dflow INFO: Task 2100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50eb3aecdc4b050229ddd596ede6dbac with result from task 2100 parsl.dataflow.dflow DEBUG: Task 2100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3b2a73c4-13ec-454b-8c88-92af2f508b8a_isrForCrosstalkSources_855452_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3b2a73c4-13ec-454b-8c88-92af2f508b8a_isrForCrosstalkSources_855452_47.stderr parsl.dataflow.dflow INFO: Task 2000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4b5a4404841f1d2fa93241fe8471ac1 with result from task 2000 parsl.dataflow.dflow DEBUG: Task 2000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/57ca9624-bd15-4ee6-81b9-8cc380f63c02_isrForCrosstalkSources_855438_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/57ca9624-bd15-4ee6-81b9-8cc380f63c02_isrForCrosstalkSources_855438_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:43 proc_lsst.multi:146 [INFO] found job 16964165 in provider checkpoint proc_lsst.multi INFO: found job 16964165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8a7fc60a8b660981cda5cc13f57828 with result from task 2014 parsl.dataflow.dflow DEBUG: Task 2014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/27821f74-9d97-4af6-80c8-b86ebad0dbf3_isrForCrosstalkSources_855386_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/27821f74-9d97-4af6-80c8-b86ebad0dbf3_isrForCrosstalkSources_855386_43.stderr parsl.dataflow.dflow INFO: Task 2586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ff57f8d4c388a8d9e761cd43adfae6d with result from task 2586 parsl.dataflow.dflow DEBUG: Task 2586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ed90738b-040a-4fcc-a467-7d068e82c3bd_isrForCrosstalkSources_855381_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ed90738b-040a-4fcc-a467-7d068e82c3bd_isrForCrosstalkSources_855381_17.stderr parsl.dataflow.dflow INFO: Task 2250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d58a69be5ee43579fc37f1cb244e38a with result from task 2250 parsl.dataflow.dflow DEBUG: Task 2250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/65479ed6-3a43-48b9-abe0-0fd5a891f427_isrForCrosstalkSources_855441_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/65479ed6-3a43-48b9-abe0-0fd5a891f427_isrForCrosstalkSources_855441_49.stderr parsl.dataflow.dflow INFO: Task 2232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c8a20548767274abe864c082c95dcea with result from task 2232 parsl.dataflow.dflow DEBUG: Task 2232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32f9d826-11b0-4989-9833-b4a738daed43_isrForCrosstalkSources_855380_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32f9d826-11b0-4989-9833-b4a738daed43_isrForCrosstalkSources_855380_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:43 proc_lsst.multi:146 [INFO] found job 16964166 in provider checkpoint proc_lsst.multi INFO: found job 16964166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a902e716dbed34676963347171e373f with result from task 2104 parsl.dataflow.dflow DEBUG: Task 2104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ef2d0b7-5ce1-41a7-aa8f-93787674a8ac_isrForCrosstalkSources_855427_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ef2d0b7-5ce1-41a7-aa8f-93787674a8ac_isrForCrosstalkSources_855427_7.stderr parsl.dataflow.dflow INFO: Task 2567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e0a031e8dfd6c62864eea2a24fe72d9 with result from task 2567 parsl.dataflow.dflow DEBUG: Task 2567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c4943726-336b-4af5-ba7e-b145950c69fb_isrForCrosstalkSources_855380_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/c4943726-336b-4af5-ba7e-b145950c69fb_isrForCrosstalkSources_855380_10.stderr parsl.dataflow.dflow INFO: Task 2431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b713797041994ba9a5dc213de48fd75c with result from task 2431 parsl.dataflow.dflow DEBUG: Task 2431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d48ce83d-a6b4-43d4-a81f-9e91bf7c1f55_isrForCrosstalkSources_855397_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d48ce83d-a6b4-43d4-a81f-9e91bf7c1f55_isrForCrosstalkSources_855397_25.stderr parsl.dataflow.dflow INFO: Task 2368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9670c598f68b31e7f8db57db854f8a1 with result from task 2368 parsl.dataflow.dflow DEBUG: Task 2368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4428289a-a4b4-4abd-bdb8-fd064a8766e2_isrForCrosstalkSources_855393_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/4428289a-a4b4-4abd-bdb8-fd064a8766e2_isrForCrosstalkSources_855393_4.stderr parsl.dataflow.dflow INFO: Task 2480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e3990d6bded9b9cd625cfa656c2068e with result from task 2480 parsl.dataflow.dflow DEBUG: Task 2480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/18c587e3-9859-4c41-b060-9dc0042cdd5e_isrForCrosstalkSources_855430_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/18c587e3-9859-4c41-b060-9dc0042cdd5e_isrForCrosstalkSources_855430_25.stderr parsl.dataflow.dflow INFO: Task 2348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0496fa74cfb2a68d0526a1e6f12f5936 with result from task 2348 parsl.dataflow.dflow DEBUG: Task 2348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/8dd49191-1303-4de2-afd2-c79d96924d23_isrForCrosstalkSources_855434_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/8dd49191-1303-4de2-afd2-c79d96924d23_isrForCrosstalkSources_855434_37.stderr parsl.dataflow.dflow INFO: Task 2438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b76e59fd84173d216dbb911aa4f08f5 with result from task 2438 parsl.dataflow.dflow DEBUG: Task 2438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fb764033-0a18-48d5-b10e-d06976ed7777_isrForCrosstalkSources_855387_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/fb764033-0a18-48d5-b10e-d06976ed7777_isrForCrosstalkSources_855387_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:44 proc_lsst.multi:146 [INFO] found job 16964167 in provider checkpoint proc_lsst.multi INFO: found job 16964167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b25ae3f106bfc144d7f39fa85afd72ff with result from task 2230 parsl.dataflow.dflow DEBUG: Task 2230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/013f87dc-f15c-4658-b4e2-00a692dc86d3_isrForCrosstalkSources_855385_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/013f87dc-f15c-4658-b4e2-00a692dc86d3_isrForCrosstalkSources_855385_53.stderr parsl.dataflow.dflow INFO: Task 2310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb3c162c6f83463342e3caee6bfe2627 with result from task 2310 parsl.dataflow.dflow DEBUG: Task 2310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4a5d9da5-f99b-4753-abba-4f4d94cb80b1_isrForCrosstalkSources_855434_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4a5d9da5-f99b-4753-abba-4f4d94cb80b1_isrForCrosstalkSources_855434_50.stderr parsl.dataflow.dflow INFO: Task 2071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a71410f76faac6a79206ef50639bdbfa with result from task 2071 parsl.dataflow.dflow DEBUG: Task 2071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c16f3700-e331-4f53-964e-0d8a604eba6d_isrForCrosstalkSources_855423_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/c16f3700-e331-4f53-964e-0d8a604eba6d_isrForCrosstalkSources_855423_1.stderr parsl.dataflow.dflow INFO: Task 2577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bbe790242a8cf0b1a1d61dab7c9ee74 with result from task 2577 parsl.dataflow.dflow DEBUG: Task 2577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e1a10005-8b30-4bde-8422-dcc0cf2962f2_isrForCrosstalkSources_855380_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e1a10005-8b30-4bde-8422-dcc0cf2962f2_isrForCrosstalkSources_855380_48.stderr parsl.dataflow.dflow INFO: Task 2448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae227f9b6745ced4776ababed6c73cf with result from task 2448 parsl.dataflow.dflow DEBUG: Task 2448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5a3c351a-7181-4799-a856-720fe28481b8_isrForCrosstalkSources_855404_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5a3c351a-7181-4799-a856-720fe28481b8_isrForCrosstalkSources_855404_4.stderr parsl.dataflow.dflow INFO: Task 2427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e441d125d41bc6c6fa53faa4a641fc05 with result from task 2427 parsl.dataflow.dflow DEBUG: Task 2427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/13e5a9e3-5113-4ddd-bf73-306c21814f1d_isrForCrosstalkSources_855454_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/13e5a9e3-5113-4ddd-bf73-306c21814f1d_isrForCrosstalkSources_855454_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:44 proc_lsst.multi:146 [INFO] found job 16964168 in provider checkpoint proc_lsst.multi INFO: found job 16964168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b10e261ea53cf1b2470e66978a607a3f with result from task 2452 parsl.dataflow.dflow DEBUG: Task 2452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1a865d0e-8843-4078-aeeb-4a7b999e3ae2_isrForCrosstalkSources_855425_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/1a865d0e-8843-4078-aeeb-4a7b999e3ae2_isrForCrosstalkSources_855425_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:44 proc_lsst.multi:146 [INFO] found job 16964169 in provider checkpoint proc_lsst.multi INFO: found job 16964169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba89465c459049b8b04b595aae9abe01 with result from task 2224 parsl.dataflow.dflow DEBUG: Task 2224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91f50958-895d-4517-abda-80e39161909f_isrForCrosstalkSources_855413_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/91f50958-895d-4517-abda-80e39161909f_isrForCrosstalkSources_855413_47.stderr parsl.dataflow.dflow INFO: Task 2499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e956e803cc58bdfb81f3bc5a2ed1d410 with result from task 2499 parsl.dataflow.dflow DEBUG: Task 2499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a809978a-640f-4d43-8da0-dd62843f55ec_isrForCrosstalkSources_855422_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a809978a-640f-4d43-8da0-dd62843f55ec_isrForCrosstalkSources_855422_40.stderr parsl.dataflow.dflow INFO: Task 2459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3b703c3eaaa41797179b87232f487b2 with result from task 2459 parsl.dataflow.dflow DEBUG: Task 2459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/42380226-790d-42cb-93a7-3c10c32ba870_isrForCrosstalkSources_855379_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/42380226-790d-42cb-93a7-3c10c32ba870_isrForCrosstalkSources_855379_4.stderr parsl.dataflow.dflow INFO: Task 2529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca00ac30889d6f8a3fa2fa66370755bf with result from task 2529 parsl.dataflow.dflow DEBUG: Task 2529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8de8c96d-1d9e-49c9-aa46-d85c1605728b_isrForCrosstalkSources_855408_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8de8c96d-1d9e-49c9-aa46-d85c1605728b_isrForCrosstalkSources_855408_3.stderr parsl.dataflow.dflow INFO: Task 2464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feaf81ccc31f7d4859d40c18ac82073d with result from task 2464 parsl.dataflow.dflow DEBUG: Task 2464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dc164cad-064c-4de9-b168-a3cab107cf7c_isrForCrosstalkSources_855387_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dc164cad-064c-4de9-b168-a3cab107cf7c_isrForCrosstalkSources_855387_10.stderr parsl.dataflow.dflow INFO: Task 2570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7200b6e44940b215a9c4c55d833c9668 with result from task 2570 parsl.dataflow.dflow DEBUG: Task 2570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c5981058-52aa-4211-841b-1272e1033133_isrForCrosstalkSources_855433_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/c5981058-52aa-4211-841b-1272e1033133_isrForCrosstalkSources_855433_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:45 proc_lsst.multi:146 [INFO] found job 16964170 in provider checkpoint proc_lsst.multi INFO: found job 16964170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6bfebe13ba6439ff0ea7f3f3587c9b8 with result from task 2418 parsl.dataflow.dflow DEBUG: Task 2418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/21c2338b-46e6-4ecd-8084-23cabfd6c772_isrForCrosstalkSources_855443_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/21c2338b-46e6-4ecd-8084-23cabfd6c772_isrForCrosstalkSources_855443_44.stderr parsl.dataflow.dflow INFO: Task 2394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72c7f1875221e405e3d80ab5b4093cc5 with result from task 2394 parsl.dataflow.dflow DEBUG: Task 2394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/a3e346e5-a925-4d80-a8d4-a15c1dedd131_isrForCrosstalkSources_855437_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/a3e346e5-a925-4d80-a8d4-a15c1dedd131_isrForCrosstalkSources_855437_10.stderr parsl.dataflow.dflow INFO: Task 2549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20bfec3a0b8f075733c32a47a60d9e42 with result from task 2549 parsl.dataflow.dflow DEBUG: Task 2549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/129d906e-92df-4ac6-af57-d00d9935eb3e_isrForCrosstalkSources_855425_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/129d906e-92df-4ac6-af57-d00d9935eb3e_isrForCrosstalkSources_855425_59.stderr parsl.dataflow.dflow INFO: Task 2592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f22c7d18413ca134d12d33ecb855db85 with result from task 2592 parsl.dataflow.dflow DEBUG: Task 2592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7cc4e343-3e87-4f90-aed5-110cfd68c5e8_isrForCrosstalkSources_855412_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7cc4e343-3e87-4f90-aed5-110cfd68c5e8_isrForCrosstalkSources_855412_59.stderr parsl.dataflow.dflow INFO: Task 2473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11a85039561ef0106e28d30ba575fe8f with result from task 2473 parsl.dataflow.dflow DEBUG: Task 2473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/daf91724-fe72-44ba-896e-acb5709b9939_isrForCrosstalkSources_855406_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/daf91724-fe72-44ba-896e-acb5709b9939_isrForCrosstalkSources_855406_56.stderr parsl.dataflow.dflow INFO: Task 2500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45638ac04310cfbb6fd519c841081325 with result from task 2500 parsl.dataflow.dflow DEBUG: Task 2500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/db417d29-03ca-46e3-a757-57ca676c08e3_isrForCrosstalkSources_855403_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/db417d29-03ca-46e3-a757-57ca676c08e3_isrForCrosstalkSources_855403_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:45 proc_lsst.multi:146 [INFO] found job 16964171 in provider checkpoint proc_lsst.multi INFO: found job 16964171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 457404b32b4e1c5144104d546517db71 with result from task 2270 parsl.dataflow.dflow DEBUG: Task 2270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9c715dde-cf14-4043-9766-8cd7c2603856_isrForCrosstalkSources_855392_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9c715dde-cf14-4043-9766-8cd7c2603856_isrForCrosstalkSources_855392_49.stderr parsl.dataflow.dflow INFO: Task 2587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c1de937a0c574d6e85820268d8e7a6e with result from task 2587 parsl.dataflow.dflow DEBUG: Task 2587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/934d3f97-da6e-4e89-904a-9cc5b363cdee_isrForCrosstalkSources_855429_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/934d3f97-da6e-4e89-904a-9cc5b363cdee_isrForCrosstalkSources_855429_54.stderr parsl.dataflow.dflow INFO: Task 2435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a90f724e38a0f7359df16f78e9ddfe3d with result from task 2435 parsl.dataflow.dflow DEBUG: Task 2435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/2e72eaf9-4a7d-4c3b-96f2-c9cd12ecdc72_isrForCrosstalkSources_855448_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/2e72eaf9-4a7d-4c3b-96f2-c9cd12ecdc72_isrForCrosstalkSources_855448_52.stderr parsl.dataflow.dflow INFO: Task 2350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9546d04cdff2c9a6b30f96382875b6d with result from task 2350 parsl.dataflow.dflow DEBUG: Task 2350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/84ebbbfb-f2e8-4f6c-9b66-a2c4473040e2_isrForCrosstalkSources_855418_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/84ebbbfb-f2e8-4f6c-9b66-a2c4473040e2_isrForCrosstalkSources_855418_37.stderr parsl.dataflow.dflow INFO: Task 2434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2be3b37fdd4f18188f6e57e93c96c5f1 with result from task 2434 parsl.dataflow.dflow DEBUG: Task 2434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/05d1650b-cb2c-4e07-86be-e6f26e804e1a_isrForCrosstalkSources_855401_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/05d1650b-cb2c-4e07-86be-e6f26e804e1a_isrForCrosstalkSources_855401_57.stderr parsl.dataflow.dflow INFO: Task 2432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8873b30e7c90530f6031f46bec30ce5b with result from task 2432 parsl.dataflow.dflow DEBUG: Task 2432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c9056e20-4aa0-47b1-a20a-a23acb680074_isrForCrosstalkSources_855409_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/c9056e20-4aa0-47b1-a20a-a23acb680074_isrForCrosstalkSources_855409_45.stderr parsl.dataflow.dflow INFO: Task 2439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15ad4d652118e2525befe868f7020cc with result from task 2439 parsl.dataflow.dflow DEBUG: Task 2439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/91530958-2efe-4521-ae82-3f688bfd953b_isrForCrosstalkSources_855410_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/91530958-2efe-4521-ae82-3f688bfd953b_isrForCrosstalkSources_855410_12.stderr parsl.dataflow.dflow INFO: Task 2400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a23050a45217d7623e50d9e5f5fb5225 with result from task 2400 parsl.dataflow.dflow DEBUG: Task 2400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a585649-3980-4733-9df8-b7236b3b83c1_isrForCrosstalkSources_855400_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/7a585649-3980-4733-9df8-b7236b3b83c1_isrForCrosstalkSources_855400_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:45 proc_lsst.multi:146 [INFO] found job 16964172 in provider checkpoint proc_lsst.multi INFO: found job 16964172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba89ab6afdc2dcea4d48ab8a36ebe49 with result from task 2451 parsl.dataflow.dflow DEBUG: Task 2451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/099c71a8-7b63-4f6c-a910-41dd2896c344_isrForCrosstalkSources_855423_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/099c71a8-7b63-4f6c-a910-41dd2896c344_isrForCrosstalkSources_855423_5.stderr parsl.dataflow.dflow INFO: Task 2483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71a2e766b72850e1da2e1fc2a2a59eb9 with result from task 2483 parsl.dataflow.dflow DEBUG: Task 2483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/73270931-a8ce-4493-9da6-6306a4f01ac6_isrForCrosstalkSources_855457_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/73270931-a8ce-4493-9da6-6306a4f01ac6_isrForCrosstalkSources_855457_36.stderr parsl.dataflow.dflow INFO: Task 2160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78e63dad178ea1245315844f283527d9 with result from task 2160 parsl.dataflow.dflow DEBUG: Task 2160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3aac6332-eb7c-4d09-b92a-bbdd3d4dcf44_isrForCrosstalkSources_855420_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3aac6332-eb7c-4d09-b92a-bbdd3d4dcf44_isrForCrosstalkSources_855420_42.stderr parsl.dataflow.dflow INFO: Task 2475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c79cb39dfbeab682a4dedc9d2677e1f with result from task 2475 parsl.dataflow.dflow DEBUG: Task 2475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1b66172-5ef1-43d6-8ce6-3d032ac92069_isrForCrosstalkSources_855458_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1b66172-5ef1-43d6-8ce6-3d032ac92069_isrForCrosstalkSources_855458_58.stderr parsl.dataflow.dflow INFO: Task 2463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 721303b0fa3e5986ac1a7d175c2d6717 with result from task 2463 parsl.dataflow.dflow DEBUG: Task 2463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9a60ddd8-13bb-42b6-a605-f4cb0e3da924_isrForCrosstalkSources_855452_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9a60ddd8-13bb-42b6-a605-f4cb0e3da924_isrForCrosstalkSources_855452_15.stderr parsl.dataflow.dflow INFO: Task 2447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e46090ad63263d9022f0b432c7cb4f35 with result from task 2447 parsl.dataflow.dflow DEBUG: Task 2447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d1bc0f8e-08ed-4ef4-acd0-cbac1f2fc6c1_isrForCrosstalkSources_855414_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d1bc0f8e-08ed-4ef4-acd0-cbac1f2fc6c1_isrForCrosstalkSources_855414_15.stderr parsl.dataflow.dflow INFO: Task 2280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb5caf7c7ab43be5ef009abd23451359 with result from task 2280 parsl.dataflow.dflow DEBUG: Task 2280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/63662c64-8044-40fb-936b-0219fc2032d3_isrForCrosstalkSources_855458_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/63662c64-8044-40fb-936b-0219fc2032d3_isrForCrosstalkSources_855458_62.stderr parsl.dataflow.dflow INFO: Task 2481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab79d843b69bf6566039a0adebf370ee with result from task 2481 parsl.dataflow.dflow DEBUG: Task 2481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9719a20c-4cdf-47a5-84e0-cf3f842d0226_isrForCrosstalkSources_855446_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9719a20c-4cdf-47a5-84e0-cf3f842d0226_isrForCrosstalkSources_855446_53.stderr parsl.dataflow.dflow INFO: Task 2510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 312a5b1eb0b3ea3b0452491e869ff339 with result from task 2510 parsl.dataflow.dflow DEBUG: Task 2510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c85b9d48-bc1d-4b3b-b650-94ec3bb66669_isrForCrosstalkSources_855446_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/c85b9d48-bc1d-4b3b-b650-94ec3bb66669_isrForCrosstalkSources_855446_11.stderr parsl.dataflow.dflow INFO: Task 2391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d17f9a7e12390560ddb8ea58705d2c82 with result from task 2391 parsl.dataflow.dflow DEBUG: Task 2391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0945eccd-cf0f-4a3b-bb49-2b2148b2eb6a_isrForCrosstalkSources_855406_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0945eccd-cf0f-4a3b-bb49-2b2148b2eb6a_isrForCrosstalkSources_855406_44.stderr parsl.dataflow.dflow INFO: Task 2223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1da6d0df94b7c6792ff0e4eafda57e1 with result from task 2223 parsl.dataflow.dflow DEBUG: Task 2223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c3b8ca0d-a181-4c20-8181-4dda739e5841_isrForCrosstalkSources_855421_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c3b8ca0d-a181-4c20-8181-4dda739e5841_isrForCrosstalkSources_855421_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:46 proc_lsst.multi:146 [INFO] found job 16964173 in provider checkpoint proc_lsst.multi INFO: found job 16964173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b890bc0d2dfa1fa6418d359285a1554 with result from task 2423 parsl.dataflow.dflow DEBUG: Task 2423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a41c0e84-4975-4e1f-b9f7-a3f77fbaccd4_isrForCrosstalkSources_855422_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a41c0e84-4975-4e1f-b9f7-a3f77fbaccd4_isrForCrosstalkSources_855422_62.stderr parsl.dataflow.dflow INFO: Task 2478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b08e9632dca68836caf6e84e6d79d5a6 with result from task 2478 parsl.dataflow.dflow DEBUG: Task 2478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3640b6-802e-48f5-a463-d4166e51f79e_isrForCrosstalkSources_855445_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3640b6-802e-48f5-a463-d4166e51f79e_isrForCrosstalkSources_855445_38.stderr parsl.dataflow.dflow INFO: Task 2497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbcd8f96cc18e192607f3f880ade14a8 with result from task 2497 parsl.dataflow.dflow DEBUG: Task 2497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/95d46ed0-9af5-46d4-bbd1-043f23f4094a_isrForCrosstalkSources_855419_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/95d46ed0-9af5-46d4-bbd1-043f23f4094a_isrForCrosstalkSources_855419_39.stderr parsl.dataflow.dflow INFO: Task 2428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bee530f3029529c8d1c288ed3a1e1e2 with result from task 2428 parsl.dataflow.dflow DEBUG: Task 2428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1cfac012-652b-4731-bf13-8285c9150480_isrForCrosstalkSources_855435_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1cfac012-652b-4731-bf13-8285c9150480_isrForCrosstalkSources_855435_4.stderr parsl.dataflow.dflow INFO: Task 2477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb6de448581f6a8880387256a5cc30c with result from task 2477 parsl.dataflow.dflow DEBUG: Task 2477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d53d0048-e981-4f91-a15d-ff1f705a4292_isrForCrosstalkSources_855410_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d53d0048-e981-4f91-a15d-ff1f705a4292_isrForCrosstalkSources_855410_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:46 proc_lsst.multi:146 [INFO] found job 16964174 in provider checkpoint proc_lsst.multi INFO: found job 16964174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b120ecb8f0c14067e63f64b4a07e5fee with result from task 2472 parsl.dataflow.dflow DEBUG: Task 2472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6c04b816-b521-422b-882b-6802a365adcb_isrForCrosstalkSources_855393_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6c04b816-b521-422b-882b-6802a365adcb_isrForCrosstalkSources_855393_3.stderr parsl.dataflow.dflow INFO: Task 2416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a783d76e73510f3e605cedf23e5c3b3e with result from task 2416 parsl.dataflow.dflow DEBUG: Task 2416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1fc4bf3e-8b0e-4ae3-a746-ada9a17ae6bd_isrForCrosstalkSources_855428_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/1fc4bf3e-8b0e-4ae3-a746-ada9a17ae6bd_isrForCrosstalkSources_855428_47.stderr parsl.dataflow.dflow INFO: Task 2468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73624d2385559090fa77acb775b69cfb with result from task 2468 parsl.dataflow.dflow DEBUG: Task 2468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a07909f5-2d3a-41a3-add5-e40304d2b1e1_isrForCrosstalkSources_855444_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a07909f5-2d3a-41a3-add5-e40304d2b1e1_isrForCrosstalkSources_855444_30.stderr parsl.dataflow.dflow INFO: Task 2436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afbec66a6e0965013b7f7d4616dedd7a with result from task 2436 parsl.dataflow.dflow DEBUG: Task 2436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8672af45-df8e-4801-ad3b-6ab9c0affe2c_isrForCrosstalkSources_855443_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8672af45-df8e-4801-ad3b-6ab9c0affe2c_isrForCrosstalkSources_855443_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:46 proc_lsst.multi:146 [INFO] found job 16964175 in provider checkpoint proc_lsst.multi INFO: found job 16964175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f700cae579e5d506da29569673fa8e with result from task 2552 parsl.dataflow.dflow DEBUG: Task 2552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4acf74c3-0a6e-471b-afef-a978b8d3089a_isrForCrosstalkSources_855435_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4acf74c3-0a6e-471b-afef-a978b8d3089a_isrForCrosstalkSources_855435_45.stderr parsl.dataflow.dflow INFO: Task 2488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 558d9b0df03e309909872e936b40db25 with result from task 2488 parsl.dataflow.dflow DEBUG: Task 2488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5517cb26-d4c2-479a-a648-9aca657b82c3_isrForCrosstalkSources_855453_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5517cb26-d4c2-479a-a648-9aca657b82c3_isrForCrosstalkSources_855453_26.stderr parsl.dataflow.dflow INFO: Task 2530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6adf35d36c951cdc8182df3e2cfc36d3 with result from task 2530 parsl.dataflow.dflow DEBUG: Task 2530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/26863c32-ddd8-4a9d-8738-c36f35a3ec0a_isrForCrosstalkSources_855393_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/26863c32-ddd8-4a9d-8738-c36f35a3ec0a_isrForCrosstalkSources_855393_27.stderr parsl.dataflow.dflow INFO: Task 2581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12f37de86a9565a543785db2a5cea5bd with result from task 2581 parsl.dataflow.dflow DEBUG: Task 2581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/469803fc-f36d-4278-a715-8beb4fdfae5c_isrForCrosstalkSources_855406_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/469803fc-f36d-4278-a715-8beb4fdfae5c_isrForCrosstalkSources_855406_23.stderr parsl.dataflow.dflow INFO: Task 2124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9263ea19e65a88a8f1dc0ccd60297e10 with result from task 2124 parsl.dataflow.dflow DEBUG: Task 2124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ead198c-f371-4cbc-a5c5-8f1f4da1d082_isrForCrosstalkSources_855445_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ead198c-f371-4cbc-a5c5-8f1f4da1d082_isrForCrosstalkSources_855445_62.stderr parsl.dataflow.dflow INFO: Task 2387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bfb9a442ee277a7b674d998dbb93ccc with result from task 2387 parsl.dataflow.dflow DEBUG: Task 2387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/857d40bf-f758-4460-af23-5daea88c9a1a_isrForCrosstalkSources_855422_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/857d40bf-f758-4460-af23-5daea88c9a1a_isrForCrosstalkSources_855422_22.stderr parsl.dataflow.dflow INFO: Task 2396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9ddb678c506c77dd18e94ccf8b14da with result from task 2396 parsl.dataflow.dflow DEBUG: Task 2396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b2ad14e2-91eb-4763-a2c1-ef4f48df0220_isrForCrosstalkSources_855396_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b2ad14e2-91eb-4763-a2c1-ef4f48df0220_isrForCrosstalkSources_855396_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:47 proc_lsst.multi:146 [INFO] found job 16964176 in provider checkpoint proc_lsst.multi INFO: found job 16964176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 244aa1cd8d191b3d1f39c9aee6a0e407 with result from task 2142 parsl.dataflow.dflow DEBUG: Task 2142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fbe313e0-5187-4e9c-8786-03c703bc41ed_isrForCrosstalkSources_855442_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/fbe313e0-5187-4e9c-8786-03c703bc41ed_isrForCrosstalkSources_855442_31.stderr parsl.dataflow.dflow INFO: Task 2098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb7370bd8a16c28276f5772e6e7c32d0 with result from task 2098 parsl.dataflow.dflow DEBUG: Task 2098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/34ff6d7e-27e7-468d-b69b-e06461b18172_isrForCrosstalkSources_855402_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/34ff6d7e-27e7-468d-b69b-e06461b18172_isrForCrosstalkSources_855402_57.stderr parsl.dataflow.dflow INFO: Task 2536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 170b193c6dffca71431edcb3a4780f83 with result from task 2536 parsl.dataflow.dflow DEBUG: Task 2536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9714cd7f-d9d2-44c1-a2b7-8b9e3ca49afa_isrForCrosstalkSources_855387_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9714cd7f-d9d2-44c1-a2b7-8b9e3ca49afa_isrForCrosstalkSources_855387_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:48 proc_lsst.multi:146 [INFO] found job 16964177 in provider checkpoint proc_lsst.multi INFO: found job 16964177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8e7ffc7ec8563172f34aca739c05a5c with result from task 2564 parsl.dataflow.dflow DEBUG: Task 2564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/43c2c253-7a4b-42ce-b338-46fc31ccab39_isrForCrosstalkSources_855386_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/43c2c253-7a4b-42ce-b338-46fc31ccab39_isrForCrosstalkSources_855386_26.stderr parsl.dataflow.dflow INFO: Task 2553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ec8df5b44b32478d05e06c668f37065 with result from task 2553 parsl.dataflow.dflow DEBUG: Task 2553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d1de614c-4398-44bf-8cb2-c70d8c699b5a_isrForCrosstalkSources_855409_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d1de614c-4398-44bf-8cb2-c70d8c699b5a_isrForCrosstalkSources_855409_25.stderr parsl.dataflow.dflow INFO: Task 2476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92fe20366010f7a3fd16ecec93617d8a with result from task 2476 parsl.dataflow.dflow DEBUG: Task 2476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/153de814-c0ae-43e3-b93a-0fe8d033202f_isrForCrosstalkSources_855447_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/153de814-c0ae-43e3-b93a-0fe8d033202f_isrForCrosstalkSources_855447_46.stderr parsl.dataflow.dflow INFO: Task 2407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52d84a8b95450b1b812bf6312d5b589e with result from task 2407 parsl.dataflow.dflow DEBUG: Task 2407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/30f75b6e-5d45-4092-b51e-932f8c573911_isrForCrosstalkSources_855429_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/30f75b6e-5d45-4092-b51e-932f8c573911_isrForCrosstalkSources_855429_22.stderr parsl.dataflow.dflow INFO: Task 2425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 508a9befde0c240495d4ec01ba635121 with result from task 2425 parsl.dataflow.dflow DEBUG: Task 2425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fa7380a2-0dbe-4689-91c6-63aa30de0975_isrForCrosstalkSources_855382_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fa7380a2-0dbe-4689-91c6-63aa30de0975_isrForCrosstalkSources_855382_44.stderr parsl.dataflow.dflow INFO: Task 2551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e949251d03e7ad0a7de7c6c5a24a969c with result from task 2551 parsl.dataflow.dflow DEBUG: Task 2551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f35f9170-0262-4df5-b360-e41e6209d19b_isrForCrosstalkSources_855439_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/f35f9170-0262-4df5-b360-e41e6209d19b_isrForCrosstalkSources_855439_13.stderr parsl.dataflow.dflow INFO: Task 2409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea2596cf6118f49ada6019d27bb48240 with result from task 2409 parsl.dataflow.dflow DEBUG: Task 2409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b7cfab68-9d78-40f0-9d9f-6de8893ab605_isrForCrosstalkSources_855379_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/b7cfab68-9d78-40f0-9d9f-6de8893ab605_isrForCrosstalkSources_855379_5.stderr parsl.dataflow.dflow INFO: Task 2450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4538a8ee3ddf7947a041f2a57adf0f3a with result from task 2450 parsl.dataflow.dflow DEBUG: Task 2450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e9562881-01aa-4786-995d-67ffd1280231_isrForCrosstalkSources_855414_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e9562881-01aa-4786-995d-67ffd1280231_isrForCrosstalkSources_855414_26.stderr parsl.dataflow.dflow INFO: Task 2593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a9dfea077589611a06fca81e4d51906 with result from task 2593 parsl.dataflow.dflow DEBUG: Task 2593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82c64291-e258-4afa-b5fc-3f444a21ea9b_isrForCrosstalkSources_855440_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82c64291-e258-4afa-b5fc-3f444a21ea9b_isrForCrosstalkSources_855440_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2603 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:48 proc_lsst.multi:146 [INFO] found job 16964178 in provider checkpoint proc_lsst.multi INFO: found job 16964178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.memoization DEBUG: Storing app cache entry 5611b40844da93460b51178edd8ca2e8 with result from task 2603 parsl.dataflow.dflow DEBUG: Task 2603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/1d325502-6326-4eeb-8e88-20e02d4b4b60_isrForCrosstalkSources_855379_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/1d325502-6326-4eeb-8e88-20e02d4b4b60_isrForCrosstalkSources_855379_24.stderr parsl.dataflow.dflow INFO: Task 2467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24471df705bab1e5bbfe4599382d0605 with result from task 2467 parsl.dataflow.dflow DEBUG: Task 2467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6c92e573-fc6e-4440-a1e4-2c1e16101cfc_isrForCrosstalkSources_855415_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6c92e573-fc6e-4440-a1e4-2c1e16101cfc_isrForCrosstalkSources_855415_9.stderr parsl.dataflow.dflow INFO: Task 2393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f985c7a378640a9ae9d34310cb75fe42 with result from task 2393 parsl.dataflow.dflow DEBUG: Task 2393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a6a15bc7-94c9-4147-a92e-324bef78b2f6_isrForCrosstalkSources_855418_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a6a15bc7-94c9-4147-a92e-324bef78b2f6_isrForCrosstalkSources_855418_24.stderr parsl.dataflow.dflow INFO: Task 2485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d23b7147748d4fa83a6a02dea8550e78 with result from task 2485 parsl.dataflow.dflow DEBUG: Task 2485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6068ca33-d2fc-44a9-9370-ef0405c01a37_isrForCrosstalkSources_855383_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6068ca33-d2fc-44a9-9370-ef0405c01a37_isrForCrosstalkSources_855383_7.stderr parsl.dataflow.dflow INFO: Task 2412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4203ab142ff6ba25284e0d91a3b197ac with result from task 2412 parsl.dataflow.dflow DEBUG: Task 2412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1bc0b450-0338-4ba4-b6b1-8debae80609f_isrForCrosstalkSources_855445_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1bc0b450-0338-4ba4-b6b1-8debae80609f_isrForCrosstalkSources_855445_8.stderr parsl.dataflow.dflow INFO: Task 2599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379b00d1b599380672e292e059aa968d with result from task 2599 parsl.dataflow.dflow DEBUG: Task 2599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f7ad424a-23b4-44c2-ab0e-4b4e60381c5b_isrForCrosstalkSources_855420_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f7ad424a-23b4-44c2-ab0e-4b4e60381c5b_isrForCrosstalkSources_855420_26.stderr parsl.dataflow.dflow INFO: Task 2544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fac20b3a1da084dd9e4841051315d793 with result from task 2544 parsl.dataflow.dflow DEBUG: Task 2544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/bbfd66eb-9d3d-4d46-9c8b-7b685dc3d704_isrForCrosstalkSources_855395_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/bbfd66eb-9d3d-4d46-9c8b-7b685dc3d704_isrForCrosstalkSources_855395_32.stderr parsl.dataflow.dflow INFO: Task 2568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a82242c92d58165413b0a7b1fdba0202 with result from task 2568 parsl.dataflow.dflow DEBUG: Task 2568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/b3ee987b-21ac-42ba-884c-af2ac8cc2bd8_isrForCrosstalkSources_855449_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/b3ee987b-21ac-42ba-884c-af2ac8cc2bd8_isrForCrosstalkSources_855449_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2388 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 8b7ffc8bf562df0f4b876408af1b5ac9 with result from task 2388 parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:49 proc_lsst.multi:146 [INFO] found job 16964179 in provider checkpoint proc_lsst.multi INFO: found job 16964179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/864881cc-7c09-4086-8b45-d003c0c6b8eb_isrForCrosstalkSources_855402_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/864881cc-7c09-4086-8b45-d003c0c6b8eb_isrForCrosstalkSources_855402_30.stderr parsl.dataflow.dflow INFO: Task 2430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bcb88aca955a7dc71a29dedaf05b8ad with result from task 2430 parsl.dataflow.dflow DEBUG: Task 2430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9a66b52e-460d-4af3-80fa-158b260b665d_isrForCrosstalkSources_855387_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9a66b52e-460d-4af3-80fa-158b260b665d_isrForCrosstalkSources_855387_18.stderr parsl.dataflow.dflow INFO: Task 2399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f6f3443ba70c3fb7e6edf5e93fa223c with result from task 2399 parsl.dataflow.dflow DEBUG: Task 2399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ce41ac0-2179-45a5-bc0b-94c02222903a_isrForCrosstalkSources_855379_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/2ce41ac0-2179-45a5-bc0b-94c02222903a_isrForCrosstalkSources_855379_49.stderr parsl.dataflow.dflow INFO: Task 2595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35a9a08c98555ebb114d88115b6a118c with result from task 2595 parsl.dataflow.dflow DEBUG: Task 2595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2d1cc952-b318-4a5e-8c05-425b648fef96_isrForCrosstalkSources_855447_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2d1cc952-b318-4a5e-8c05-425b648fef96_isrForCrosstalkSources_855447_11.stderr parsl.dataflow.dflow INFO: Task 2578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a25234f808ee399f8b1b00fbe777941a with result from task 2578 parsl.dataflow.dflow DEBUG: Task 2578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d409610a-8d69-4a28-9df2-ac10c521f0a8_isrForCrosstalkSources_855428_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d409610a-8d69-4a28-9df2-ac10c521f0a8_isrForCrosstalkSources_855428_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:50 proc_lsst.multi:146 [INFO] found job 16964180 in provider checkpoint proc_lsst.multi INFO: found job 16964180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30a9088d1f30a41f11ea12e5fc6d5a94 with result from task 2652 parsl.dataflow.dflow DEBUG: Task 2652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/63625687-3f84-4379-8eb1-fd2af4f327fc_isrForCrosstalkSources_855389_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/63625687-3f84-4379-8eb1-fd2af4f327fc_isrForCrosstalkSources_855389_23.stderr parsl.dataflow.dflow INFO: Task 2410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c3f68099efa696870f827018ea253f5 with result from task 2410 parsl.dataflow.dflow DEBUG: Task 2410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7b8f17c5-0f47-48e9-851f-1f354b90764e_isrForCrosstalkSources_855403_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7b8f17c5-0f47-48e9-851f-1f354b90764e_isrForCrosstalkSources_855403_33.stderr parsl.dataflow.dflow INFO: Task 2601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2033c3048626bdf9c1f5d5d5f2e7e438 with result from task 2601 parsl.dataflow.dflow DEBUG: Task 2601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/89048f1e-2bc8-4dfb-8871-70fba59dd2b2_isrForCrosstalkSources_855380_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/89048f1e-2bc8-4dfb-8871-70fba59dd2b2_isrForCrosstalkSources_855380_14.stderr parsl.dataflow.dflow INFO: Task 2562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 524cff39a29323f1aeda440054705c4d with result from task 2562 parsl.dataflow.dflow DEBUG: Task 2562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/30b5f41a-19db-4c12-96f9-babbea6f6112_isrForCrosstalkSources_855451_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/30b5f41a-19db-4c12-96f9-babbea6f6112_isrForCrosstalkSources_855451_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:50 proc_lsst.multi:146 [INFO] found job 16964181 in provider checkpoint proc_lsst.multi INFO: found job 16964181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2186822b35bcc7366664ec08db484287 with result from task 2579 parsl.dataflow.dflow DEBUG: Task 2579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5bdc3a2a-1304-4e01-bccb-2a45dc5e5209_isrForCrosstalkSources_855396_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/5bdc3a2a-1304-4e01-bccb-2a45dc5e5209_isrForCrosstalkSources_855396_25.stderr parsl.dataflow.dflow INFO: Task 2426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6bd4f6653d325d9eecda02d0ea4c378 with result from task 2426 parsl.dataflow.dflow DEBUG: Task 2426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/076b7b8e-70f1-413f-95c3-2b17c696b402_isrForCrosstalkSources_855409_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/076b7b8e-70f1-413f-95c3-2b17c696b402_isrForCrosstalkSources_855409_34.stderr parsl.dataflow.dflow INFO: Task 2591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3c7d92277afa8d5af507c6a1a243851 with result from task 2591 parsl.dataflow.dflow DEBUG: Task 2591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/568d35f2-a798-42c2-bb96-ff235504ac5d_isrForCrosstalkSources_855386_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/568d35f2-a798-42c2-bb96-ff235504ac5d_isrForCrosstalkSources_855386_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:51 proc_lsst.multi:146 [INFO] found job 16964182 in provider checkpoint proc_lsst.multi INFO: found job 16964182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 698a7fae3c76c4c5619191c961b75695 with result from task 2403 parsl.dataflow.dflow DEBUG: Task 2403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/e3075180-b866-4d92-a3ce-4a8558bf50c2_isrForCrosstalkSources_855425_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/e3075180-b866-4d92-a3ce-4a8558bf50c2_isrForCrosstalkSources_855425_43.stderr parsl.dataflow.dflow INFO: Task 2621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba3d642598cf47598cc947d49dad61e3 with result from task 2621 parsl.dataflow.dflow DEBUG: Task 2621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eb9b7a0b-fb71-45b6-8300-5467ed3d9538_isrForCrosstalkSources_855423_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eb9b7a0b-fb71-45b6-8300-5467ed3d9538_isrForCrosstalkSources_855423_27.stderr parsl.dataflow.dflow INFO: Task 2402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67804ec3da307fa18cf17001b7b70049 with result from task 2402 parsl.dataflow.dflow DEBUG: Task 2402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65a30e4e-beab-49a7-91fb-cdba2541702e_isrForCrosstalkSources_855413_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65a30e4e-beab-49a7-91fb-cdba2541702e_isrForCrosstalkSources_855413_34.stderr parsl.dataflow.dflow INFO: Task 2537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 173630ac502011ee1859fa7975452a6d with result from task 2537 parsl.dataflow.dflow DEBUG: Task 2537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/76e377bf-a14b-497a-b7d2-f1661d4f7546_isrForCrosstalkSources_855428_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/76e377bf-a14b-497a-b7d2-f1661d4f7546_isrForCrosstalkSources_855428_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:51 proc_lsst.multi:146 [INFO] found job 16964183 in provider checkpoint proc_lsst.multi INFO: found job 16964183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7dfd9a3bbc46375d800b4903f584db5 with result from task 2585 parsl.dataflow.dflow DEBUG: Task 2585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a4be636e-785d-4eab-935f-108332bf4ccb_isrForCrosstalkSources_855386_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a4be636e-785d-4eab-935f-108332bf4ccb_isrForCrosstalkSources_855386_30.stderr parsl.dataflow.dflow INFO: Task 2429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a474d5cc63229c03063a743b3ed8a4ae with result from task 2429 parsl.dataflow.dflow DEBUG: Task 2429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f9db8dfb-5a9d-45a8-b1a0-0d3c62be9095_isrForCrosstalkSources_855421_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/f9db8dfb-5a9d-45a8-b1a0-0d3c62be9095_isrForCrosstalkSources_855421_22.stderr parsl.dataflow.dflow INFO: Task 2482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e667942f988c9248ef57f0cb663ca5a with result from task 2482 parsl.dataflow.dflow DEBUG: Task 2482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c63893d9-f769-4264-a6e6-6780d6e0ac28_isrForCrosstalkSources_855434_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c63893d9-f769-4264-a6e6-6780d6e0ac28_isrForCrosstalkSources_855434_13.stderr parsl.dataflow.dflow INFO: Task 2526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d97cf74f3372d6ce3d24b3792ff222be with result from task 2526 parsl.dataflow.dflow DEBUG: Task 2526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/d78ff4f8-50fb-4f30-887f-b6d9c811c738_isrForCrosstalkSources_855396_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/d78ff4f8-50fb-4f30-887f-b6d9c811c738_isrForCrosstalkSources_855396_50.stderr parsl.dataflow.dflow INFO: Task 2540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b3fc0043debe5bddcc4fa172ee66fda with result from task 2540 parsl.dataflow.dflow DEBUG: Task 2540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/edccc128-d39d-4ee9-8b6c-df8703bbc9c6_isrForCrosstalkSources_855450_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/edccc128-d39d-4ee9-8b6c-df8703bbc9c6_isrForCrosstalkSources_855450_20.stderr parsl.dataflow.dflow INFO: Task 2629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6e7fe6a3baafc06bb664f6499398887 with result from task 2629 parsl.dataflow.dflow DEBUG: Task 2629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/217442bb-0da0-48a7-8503-95da57427e05_isrForCrosstalkSources_855412_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/217442bb-0da0-48a7-8503-95da57427e05_isrForCrosstalkSources_855412_5.stderr parsl.dataflow.dflow INFO: Task 2466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ceb96c3cf219e92c32f08446e61a8ca with result from task 2466 parsl.dataflow.dflow DEBUG: Task 2466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6a460e78-92db-4269-b7e3-d884088a2bb3_isrForCrosstalkSources_855387_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6a460e78-92db-4269-b7e3-d884088a2bb3_isrForCrosstalkSources_855387_30.stderr parsl.dataflow.dflow INFO: Task 2479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a08a8d9c2a60c196df7cf7c7b22f9dd with result from task 2479 parsl.dataflow.dflow DEBUG: Task 2479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f59a73ed-dd66-4bd0-ba00-a64755fd60aa_isrForCrosstalkSources_855416_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f59a73ed-dd66-4bd0-ba00-a64755fd60aa_isrForCrosstalkSources_855416_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:52 proc_lsst.multi:146 [INFO] found job 16964184 in provider checkpoint proc_lsst.multi INFO: found job 16964184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee816bb85f41156d74732741f80d21b4 with result from task 2456 parsl.dataflow.dflow DEBUG: Task 2456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8fcc1070-325a-4df1-98f1-6ede50274f23_isrForCrosstalkSources_855419_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8fcc1070-325a-4df1-98f1-6ede50274f23_isrForCrosstalkSources_855419_44.stderr parsl.dataflow.dflow INFO: Task 2575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d63f4c61c51b0490271a5fa817e6885 with result from task 2575 parsl.dataflow.dflow DEBUG: Task 2575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d7fb63a0-d8d5-4d86-a1a3-c69fa60bd599_isrForCrosstalkSources_855409_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/d7fb63a0-d8d5-4d86-a1a3-c69fa60bd599_isrForCrosstalkSources_855409_41.stderr parsl.dataflow.dflow INFO: Task 2690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49e0400149b78718ca76ffc6600e2057 with result from task 2690 parsl.dataflow.dflow DEBUG: Task 2690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7ecc3dc4-2653-4c38-894c-c49a277ce2b4_isrForCrosstalkSources_855401_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7ecc3dc4-2653-4c38-894c-c49a277ce2b4_isrForCrosstalkSources_855401_25.stderr parsl.dataflow.dflow INFO: Task 2523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8e41e63bfeff6b9cc5169d1a501cd94 with result from task 2523 parsl.dataflow.dflow DEBUG: Task 2523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/af4ef68a-607c-45ae-b1c5-85d3a8e11342_isrForCrosstalkSources_855386_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/af4ef68a-607c-45ae-b1c5-85d3a8e11342_isrForCrosstalkSources_855386_48.stderr parsl.dataflow.dflow INFO: Task 2684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e15db26a8b53f3c7404152ddb21ecf6e with result from task 2684 parsl.dataflow.dflow DEBUG: Task 2684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/728b185e-f293-487c-b945-e1fbcdd6006f_isrForCrosstalkSources_855412_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/728b185e-f293-487c-b945-e1fbcdd6006f_isrForCrosstalkSources_855412_58.stderr parsl.dataflow.dflow INFO: Task 2404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d7270d496efbd9b3ee6f8b6b59d222 with result from task 2404 parsl.dataflow.dflow DEBUG: Task 2404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/edb32e10-4572-4155-a26f-268d2e7c32bb_isrForCrosstalkSources_855456_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/edb32e10-4572-4155-a26f-268d2e7c32bb_isrForCrosstalkSources_855456_15.stderr parsl.dataflow.dflow INFO: Task 2528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2c03eea5fea590986a3356952075faa with result from task 2528 parsl.dataflow.dflow DEBUG: Task 2528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8ca9fdae-5048-479b-9f5b-bec7f01783f6_isrForCrosstalkSources_855393_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8ca9fdae-5048-479b-9f5b-bec7f01783f6_isrForCrosstalkSources_855393_36.stderr parsl.dataflow.dflow INFO: Task 2470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 492e4932e33871d9ec16cc0d313fdba0 with result from task 2470 parsl.dataflow.dflow DEBUG: Task 2470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/00d102e9-0340-4dc9-9c0a-f341531d50db_isrForCrosstalkSources_855410_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/00d102e9-0340-4dc9-9c0a-f341531d50db_isrForCrosstalkSources_855410_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2657 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 67b21f0b6cf281ef318cebd0a9e03fb0 with result from task 2657 parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2657 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2b9c421d-9453-4d1d-837e-114476e6dafe_isrForCrosstalkSources_855421_14.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2b9c421d-9453-4d1d-837e-114476e6dafe_isrForCrosstalkSources_855421_14.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2424 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 077343924ffd50f2a044dc265a06fb21 with result from task 2424 parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2424 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccd108f1-fdb3-488e-b23d-bb3a81a0f957_isrForCrosstalkSources_855403_55.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccd108f1-fdb3-488e-b23d-bb3a81a0f957_isrForCrosstalkSources_855403_55.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2604 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 836570b2d4364235193bd5c05af89c37 with result from task 2604 parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2604 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/14711fb2-21db-4c9a-af4d-2c39961a5def_isrForCrosstalkSources_855390_11.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/14711fb2-21db-4c9a-af4d-2c39961a5def_isrForCrosstalkSources_855390_11.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2708 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry b46c578aec35b8b61791714b594af737 with result from task 2708 parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2708 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:53 proc_lsst.multi:146 [INFO] found job 16964185 in provider checkpoint proc_lsst.multi INFO: found job 16964185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Standard output for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9316aed5-edf1-4755-a4ee-b6dcb24b8a4d_isrForCrosstalkSources_855450_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/9316aed5-edf1-4755-a4ee-b6dcb24b8a4d_isrForCrosstalkSources_855450_13.stderr parsl.dataflow.dflow INFO: Task 2484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f721278baed680998913815c0be0028 with result from task 2484 parsl.dataflow.dflow DEBUG: Task 2484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/048edd6e-75e7-463f-ba26-4fe51d2a0d82_isrForCrosstalkSources_855422_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/048edd6e-75e7-463f-ba26-4fe51d2a0d82_isrForCrosstalkSources_855422_53.stderr parsl.dataflow.dflow INFO: Task 2506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13dcf499580181b65bd46a70ec9b2ef5 with result from task 2506 parsl.dataflow.dflow DEBUG: Task 2506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/f0258b47-8a9e-473d-bf7d-d3ce125c8638_isrForCrosstalkSources_855386_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/f0258b47-8a9e-473d-bf7d-d3ce125c8638_isrForCrosstalkSources_855386_52.stderr parsl.dataflow.dflow INFO: Task 2525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5726636c68e8b7b932f11efcb8fcc2 with result from task 2525 parsl.dataflow.dflow DEBUG: Task 2525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/53f9b073-76c6-4198-a0a0-d881bfe2340c_isrForCrosstalkSources_855425_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/53f9b073-76c6-4198-a0a0-d881bfe2340c_isrForCrosstalkSources_855425_31.stderr parsl.dataflow.dflow INFO: Task 2524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03edc826b3c68841c1dedb1fe7b63137 with result from task 2524 parsl.dataflow.dflow DEBUG: Task 2524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/918bca7c-8929-459c-b78a-ad31137193da_isrForCrosstalkSources_855447_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/918bca7c-8929-459c-b78a-ad31137193da_isrForCrosstalkSources_855447_43.stderr parsl.dataflow.dflow INFO: Task 2632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 233cddacce7200c32f8e79f56d6edec9 with result from task 2632 parsl.dataflow.dflow DEBUG: Task 2632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/007786ed-30fd-4415-901e-1c60e28c22a4_isrForCrosstalkSources_855404_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/007786ed-30fd-4415-901e-1c60e28c22a4_isrForCrosstalkSources_855404_24.stderr parsl.dataflow.dflow INFO: Task 2504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75d98e4fdbdce11f3da67e95659c87ad with result from task 2504 parsl.dataflow.dflow DEBUG: Task 2504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d0207ac-7596-4fa4-a801-5568395b4324_isrForCrosstalkSources_855388_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d0207ac-7596-4fa4-a801-5568395b4324_isrForCrosstalkSources_855388_57.stderr parsl.dataflow.dflow INFO: Task 2572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33e2b7f956cb807ade675c7226ae7ad7 with result from task 2572 parsl.dataflow.dflow DEBUG: Task 2572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bbe6ed9d-2502-43f5-b169-9a78c0a6a225_isrForCrosstalkSources_855380_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/bbe6ed9d-2502-43f5-b169-9a78c0a6a225_isrForCrosstalkSources_855380_55.stderr parsl.dataflow.dflow INFO: Task 2590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0969fd98fc700204e24d26f9c556fde1 with result from task 2590 parsl.dataflow.dflow DEBUG: Task 2590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1229a30a-ca6c-4f3a-8338-18a370c7c8a1_isrForCrosstalkSources_855412_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/1229a30a-ca6c-4f3a-8338-18a370c7c8a1_isrForCrosstalkSources_855412_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:54 proc_lsst.multi:146 [INFO] found job 16964186 in provider checkpoint proc_lsst.multi INFO: found job 16964186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfb86a89ebb12780f9b51dba297f353 with result from task 2618 parsl.dataflow.dflow DEBUG: Task 2618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/beddd4e5-2c32-4ad1-b0b0-3895368eec5d_isrForCrosstalkSources_855399_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/beddd4e5-2c32-4ad1-b0b0-3895368eec5d_isrForCrosstalkSources_855399_47.stderr parsl.dataflow.dflow INFO: Task 2460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d1417b018e2258395ec84cc2dab7b9e with result from task 2460 parsl.dataflow.dflow DEBUG: Task 2460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f527f74e-d0bd-4e3a-96ff-a2634bf84ce2_isrForCrosstalkSources_855420_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f527f74e-d0bd-4e3a-96ff-a2634bf84ce2_isrForCrosstalkSources_855420_31.stderr parsl.dataflow.dflow INFO: Task 2711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e100bfb4d12b36757460c40dce3898cf with result from task 2711 parsl.dataflow.dflow DEBUG: Task 2711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/dc82d449-a567-4d41-9af7-b11ae3c1dae4_isrForCrosstalkSources_855422_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/dc82d449-a567-4d41-9af7-b11ae3c1dae4_isrForCrosstalkSources_855422_14.stderr parsl.dataflow.dflow INFO: Task 2571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5fe742aecaa1ea6e0b6ab34309ee1ab with result from task 2571 parsl.dataflow.dflow DEBUG: Task 2571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82776559-131a-4fdf-bed4-55ee12ce42e2_isrForCrosstalkSources_855440_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/82776559-131a-4fdf-bed4-55ee12ce42e2_isrForCrosstalkSources_855440_3.stderr parsl.dataflow.dflow INFO: Task 2513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry add7e7c64112ef59b4f4d2329c05965d with result from task 2513 parsl.dataflow.dflow DEBUG: Task 2513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c654e4b3-6c7f-4fb5-a63b-84045b2ac2b2_isrForCrosstalkSources_855413_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c654e4b3-6c7f-4fb5-a63b-84045b2ac2b2_isrForCrosstalkSources_855413_61.stderr parsl.dataflow.dflow INFO: Task 2445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e9664a1560d741bfa562ff0e5b01762 with result from task 2445 parsl.dataflow.dflow DEBUG: Task 2445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/31dc3073-8a09-4b6f-858b-f50ba4946306_isrForCrosstalkSources_855444_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/31dc3073-8a09-4b6f-858b-f50ba4946306_isrForCrosstalkSources_855444_23.stderr parsl.dataflow.dflow INFO: Task 2665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0feff71e3b00a077e343116599c15b5e with result from task 2665 parsl.dataflow.dflow DEBUG: Task 2665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dcc0f1a4-1b0b-48b9-838d-48218d617e08_isrForCrosstalkSources_855456_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/dcc0f1a4-1b0b-48b9-838d-48218d617e08_isrForCrosstalkSources_855456_27.stderr parsl.dataflow.dflow INFO: Task 2511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd0f00ccf9746fe600f29b10719dadfa with result from task 2511 parsl.dataflow.dflow DEBUG: Task 2511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6cfa263b-643b-4c5a-a072-0d2851665d8e_isrForCrosstalkSources_855392_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6cfa263b-643b-4c5a-a072-0d2851665d8e_isrForCrosstalkSources_855392_42.stderr parsl.dataflow.dflow INFO: Task 2509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c8b0638154a5691c6f25fe71c6ddcdf with result from task 2509 parsl.dataflow.dflow DEBUG: Task 2509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ebe987a-4fa4-42e4-86a6-e3358006a93b_isrForCrosstalkSources_855401_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ebe987a-4fa4-42e4-86a6-e3358006a93b_isrForCrosstalkSources_855401_11.stderr parsl.dataflow.dflow INFO: Task 2557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0344cd396b27d7fcba34c147c9761ec9 with result from task 2557 parsl.dataflow.dflow DEBUG: Task 2557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f413a3cd-eb45-424e-b7df-ff458de72eec_isrForCrosstalkSources_855454_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f413a3cd-eb45-424e-b7df-ff458de72eec_isrForCrosstalkSources_855454_60.stderr parsl.dataflow.dflow INFO: Task 2703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a40354c37330ef7a37ca85ec70f2e5ff with result from task 2703 parsl.dataflow.dflow DEBUG: Task 2703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/913b6b8d-af6b-45af-acbd-47481a269765_isrForCrosstalkSources_855428_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/913b6b8d-af6b-45af-acbd-47481a269765_isrForCrosstalkSources_855428_51.stderr parsl.dataflow.dflow INFO: Task 2534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc0f932164cc14bf6ea5d7d73e4dd1fd with result from task 2534 parsl.dataflow.dflow DEBUG: Task 2534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/cf407e3a-49d1-4b21-b061-3787a143e7da_isrForCrosstalkSources_855404_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/cf407e3a-49d1-4b21-b061-3787a143e7da_isrForCrosstalkSources_855404_55.stderr parsl.dataflow.dflow INFO: Task 2505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640e1986b169b5711287932a455e5da8 with result from task 2505 parsl.dataflow.dflow DEBUG: Task 2505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7b1a3c5a-b52e-40e9-ac05-15a7ee1a38f4_isrForCrosstalkSources_855421_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/7b1a3c5a-b52e-40e9-ac05-15a7ee1a38f4_isrForCrosstalkSources_855421_2.stderr parsl.dataflow.dflow INFO: Task 2671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7077324106a329ec20ee555bee782975 with result from task 2671 parsl.dataflow.dflow DEBUG: Task 2671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/79273be5-bbed-47a9-8f96-002e62ee06bb_isrForCrosstalkSources_855437_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/79273be5-bbed-47a9-8f96-002e62ee06bb_isrForCrosstalkSources_855437_19.stderr parsl.dataflow.dflow INFO: Task 2660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305dc3b88c7fd65246af4aef1ad5fc3f with result from task 2660 parsl.dataflow.dflow DEBUG: Task 2660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/34a0477b-482f-4b82-abf6-66322e01e7da_isrForCrosstalkSources_855390_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/34a0477b-482f-4b82-abf6-66322e01e7da_isrForCrosstalkSources_855390_20.stderr parsl.dataflow.dflow INFO: Task 2624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1caefce6e06044992acc845d344e6a6 with result from task 2624 parsl.dataflow.dflow DEBUG: Task 2624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/7c144a72-932e-4025-8ce9-f1e43d10c1df_isrForCrosstalkSources_855432_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/7c144a72-932e-4025-8ce9-f1e43d10c1df_isrForCrosstalkSources_855432_12.stderr parsl.dataflow.dflow INFO: Task 2491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62d32c16d454ed1dd4b8902ee5668ff3 with result from task 2491 parsl.dataflow.dflow DEBUG: Task 2491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/bc5eb85b-bd76-46c3-93c1-087ba7510987_isrForCrosstalkSources_855451_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/bc5eb85b-bd76-46c3-93c1-087ba7510987_isrForCrosstalkSources_855451_2.stderr parsl.dataflow.dflow INFO: Task 2713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 242550a2dd6049bbf39cb545ff2106ed with result from task 2713 parsl.dataflow.dflow DEBUG: Task 2713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e3aa41c4-038e-48e3-a0c7-248f34ef7ddd_isrForCrosstalkSources_855433_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/e3aa41c4-038e-48e3-a0c7-248f34ef7ddd_isrForCrosstalkSources_855433_4.stderr parsl.dataflow.dflow INFO: Task 2676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8f0248e28e043ec878361c32b2f68cb with result from task 2676 parsl.dataflow.dflow DEBUG: Task 2676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8f64edcd-8389-4fbe-94ad-75dc2911c61c_isrForCrosstalkSources_855392_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/8f64edcd-8389-4fbe-94ad-75dc2911c61c_isrForCrosstalkSources_855392_21.stderr parsl.dataflow.dflow INFO: Task 2698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 822e245cf3ab435b29aea18b7a1425b8 with result from task 2698 parsl.dataflow.dflow DEBUG: Task 2698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8b46c0d7-6c64-4478-a658-18a7b602a0f1_isrForCrosstalkSources_855443_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8b46c0d7-6c64-4478-a658-18a7b602a0f1_isrForCrosstalkSources_855443_51.stderr parsl.dataflow.dflow INFO: Task 2535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f93c69444e27aa6018e55699eba18c with result from task 2535 parsl.dataflow.dflow DEBUG: Task 2535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fbb284ae-b300-4eea-8877-7ad0e9904994_isrForCrosstalkSources_855450_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fbb284ae-b300-4eea-8877-7ad0e9904994_isrForCrosstalkSources_855450_10.stderr parsl.dataflow.dflow INFO: Task 2503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b0465a07dff81c034cbb1034c79ef19 with result from task 2503 parsl.dataflow.dflow DEBUG: Task 2503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/acd859d9-16d1-4e43-ac46-6116d78fce00_isrForCrosstalkSources_855396_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/acd859d9-16d1-4e43-ac46-6116d78fce00_isrForCrosstalkSources_855396_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964162 PD 16964161 PD 16964160 PD 16964159 PD 16964158 PD 16964157 PD 16964156 PD 16964155 PD 16964154 PD 16964153 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:56 proc_lsst.multi:146 [INFO] found job 16964187 in provider checkpoint proc_lsst.multi INFO: found job 16964187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd5a14f9f4335f889128d14e2af05fa2 with result from task 2736 parsl.dataflow.dflow DEBUG: Task 2736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3d9072e0-83e0-4b4d-b2fe-4e53fac26b31_isrForCrosstalkSources_855388_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/3d9072e0-83e0-4b4d-b2fe-4e53fac26b31_isrForCrosstalkSources_855388_22.stderr parsl.dataflow.dflow INFO: Task 2661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92c0a6b3cf5bb773c384746ccfcf93ca with result from task 2661 parsl.dataflow.dflow DEBUG: Task 2661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e7d1f125-8966-4c52-ae66-f3a6b419e131_isrForCrosstalkSources_855384_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e7d1f125-8966-4c52-ae66-f3a6b419e131_isrForCrosstalkSources_855384_32.stderr parsl.dataflow.dflow INFO: Task 2748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df94f9b4398e76f6b1d33d17f7738265 with result from task 2748 parsl.dataflow.dflow DEBUG: Task 2748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/16fcdc64-8a36-48df-a04f-cd17f9edff13_isrForCrosstalkSources_855452_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/16fcdc64-8a36-48df-a04f-cd17f9edff13_isrForCrosstalkSources_855452_23.stderr parsl.dataflow.dflow INFO: Task 2743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d45c5eb1f1b0e7aa4df9f5d1f089eeb with result from task 2743 parsl.dataflow.dflow DEBUG: Task 2743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/3fda785f-277d-4ffb-892e-ac4a789445f7_isrForCrosstalkSources_855436_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/3fda785f-277d-4ffb-892e-ac4a789445f7_isrForCrosstalkSources_855436_9.stderr parsl.dataflow.dflow INFO: Task 2675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 445cafc79c9811491d6ce19e2be7791f with result from task 2675 parsl.dataflow.dflow DEBUG: Task 2675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3cda799c-5382-4425-97c2-68ae7ee6a2f8_isrForCrosstalkSources_855389_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3cda799c-5382-4425-97c2-68ae7ee6a2f8_isrForCrosstalkSources_855389_17.stderr parsl.dataflow.dflow INFO: Task 2723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7085bc0c80e78759cc6ec65230151e7e with result from task 2723 parsl.dataflow.dflow DEBUG: Task 2723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/358da787-9135-4110-8339-c665dddc3cda_isrForCrosstalkSources_855391_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/358da787-9135-4110-8339-c665dddc3cda_isrForCrosstalkSources_855391_41.stderr parsl.dataflow.dflow INFO: Task 2589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e9883ae0e42bb644a85ac4f277ffd5 with result from task 2589 parsl.dataflow.dflow DEBUG: Task 2589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f564721e-0ff8-4d28-8ad7-fc1a546cd8f1_isrForCrosstalkSources_855409_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/f564721e-0ff8-4d28-8ad7-fc1a546cd8f1_isrForCrosstalkSources_855409_60.stderr parsl.dataflow.dflow INFO: Task 2669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d3e0d07e044331c3ce813784bdd451 with result from task 2669 parsl.dataflow.dflow DEBUG: Task 2669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d149318b-9ce7-4ea7-8c1a-3ac417694eab_isrForCrosstalkSources_855416_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/d149318b-9ce7-4ea7-8c1a-3ac417694eab_isrForCrosstalkSources_855416_40.stderr parsl.dataflow.dflow INFO: Task 2658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e48ea1240305c28bd396f3c0bdebb2a1 with result from task 2658 parsl.dataflow.dflow DEBUG: Task 2658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b876dae-f02f-4d3f-a7c4-99d9858b59e2_isrForCrosstalkSources_855416_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b876dae-f02f-4d3f-a7c4-99d9858b59e2_isrForCrosstalkSources_855416_9.stderr parsl.dataflow.dflow INFO: Task 2749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86a29a97a7da7e1ffe85bc9fbc84d3d5 with result from task 2749 parsl.dataflow.dflow DEBUG: Task 2749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/3bd70d76-a39c-4618-89df-97391a089542_isrForCrosstalkSources_855422_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/3bd70d76-a39c-4618-89df-97391a089542_isrForCrosstalkSources_855422_15.stderr parsl.dataflow.dflow INFO: Task 2663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e951975dde71d4dcdb3e1d2466c7e9a with result from task 2663 parsl.dataflow.dflow DEBUG: Task 2663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea7baaf7-5ac0-4483-a398-352622121fb2_isrForCrosstalkSources_855390_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ea7baaf7-5ac0-4483-a398-352622121fb2_isrForCrosstalkSources_855390_1.stderr parsl.dataflow.dflow INFO: Task 2659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 239d5088575a2941375b8a2acbdd9abc with result from task 2659 parsl.dataflow.dflow DEBUG: Task 2659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/d14bb54f-2837-4c76-8b88-1cc00999e011_isrForCrosstalkSources_855412_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/d14bb54f-2837-4c76-8b88-1cc00999e011_isrForCrosstalkSources_855412_40.stderr parsl.dataflow.dflow INFO: Task 2680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41f20fc0dbe85f6c30db303c426544cc with result from task 2680 parsl.dataflow.dflow DEBUG: Task 2680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e5f518a5-e826-4136-ad01-c42021713a75_isrForCrosstalkSources_855384_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/e5f518a5-e826-4136-ad01-c42021713a75_isrForCrosstalkSources_855384_30.stderr parsl.dataflow.dflow INFO: Task 2674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d81ee32c222af40b3f9144f5ac4ae2 with result from task 2674 parsl.dataflow.dflow DEBUG: Task 2674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2efc5bc8-43eb-4c68-8424-b8a39325e0e3_isrForCrosstalkSources_855382_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2efc5bc8-43eb-4c68-8424-b8a39325e0e3_isrForCrosstalkSources_855382_21.stderr parsl.dataflow.dflow INFO: Task 2667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 848032236dba575c13d31f0fc5f84eda with result from task 2667 parsl.dataflow.dflow DEBUG: Task 2667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da1e43a8-985e-42bc-a0f7-fb18c72dadf6_isrForCrosstalkSources_855404_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/da1e43a8-985e-42bc-a0f7-fb18c72dadf6_isrForCrosstalkSources_855404_9.stderr parsl.dataflow.dflow INFO: Task 2727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ddebc014a79ca57985f2a724a1f853 with result from task 2727 parsl.dataflow.dflow DEBUG: Task 2727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ae43a756-0a39-4d8a-8b10-467024235906_isrForCrosstalkSources_855452_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/ae43a756-0a39-4d8a-8b10-467024235906_isrForCrosstalkSources_855452_38.stderr parsl.dataflow.dflow INFO: Task 2753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e7fb0c3aff144c48c554753cba178e5 with result from task 2753 parsl.dataflow.dflow DEBUG: Task 2753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/67000ad8-0f88-4785-8035-321e9ad489d8_isrForCrosstalkSources_855415_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/67000ad8-0f88-4785-8035-321e9ad489d8_isrForCrosstalkSources_855415_53.stderr parsl.dataflow.dflow INFO: Task 2745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 410de272dfa866357857aca94c3f3566 with result from task 2745 parsl.dataflow.dflow DEBUG: Task 2745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/44358b84-b1e7-447f-af5f-358f0dcda1e2_isrForCrosstalkSources_855382_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/44358b84-b1e7-447f-af5f-358f0dcda1e2_isrForCrosstalkSources_855382_28.stderr parsl.dataflow.dflow INFO: Task 2673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1cfb96bcf71e3ca4614ee4b0957044 with result from task 2673 parsl.dataflow.dflow DEBUG: Task 2673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c641d690-288c-4639-bb9c-e1ffec6b1231_isrForCrosstalkSources_855452_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c641d690-288c-4639-bb9c-e1ffec6b1231_isrForCrosstalkSources_855452_28.stderr parsl.dataflow.dflow INFO: Task 2662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7512a21f81694e721f5b5eebd4a6318 with result from task 2662 parsl.dataflow.dflow DEBUG: Task 2662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/47517000-238b-4994-aaa6-b7716f69f751_isrForCrosstalkSources_855446_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/47517000-238b-4994-aaa6-b7716f69f751_isrForCrosstalkSources_855446_18.stderr parsl.dataflow.dflow INFO: Task 2670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00ac30a88a7a9399ceac7937c1c9be75 with result from task 2670 parsl.dataflow.dflow DEBUG: Task 2670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/15af6f28-0b1e-4728-8687-2fd2d0b50ad3_isrForCrosstalkSources_855433_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/15af6f28-0b1e-4728-8687-2fd2d0b50ad3_isrForCrosstalkSources_855433_10.stderr parsl.dataflow.dflow INFO: Task 2335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 323fe4ca46587cd1ea736b8252961932 with result from task 2335 parsl.dataflow.dflow DEBUG: Task 2335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5299c219-a7df-4d8f-9908-9ede91d8cf75_isrForCrosstalkSources_855406_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/5299c219-a7df-4d8f-9908-9ede91d8cf75_isrForCrosstalkSources_855406_32.stderr parsl.dataflow.dflow INFO: Task 2366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e762ec81ff4d7cf215600ee315099802 with result from task 2366 parsl.dataflow.dflow DEBUG: Task 2366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2e4362cf-9e09-48ae-ad85-d80f8ff4f75b_isrForCrosstalkSources_855410_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2e4362cf-9e09-48ae-ad85-d80f8ff4f75b_isrForCrosstalkSources_855410_23.stderr parsl.dataflow.dflow INFO: Task 2354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf13e14b9aff847d15262df2efcde0e with result from task 2354 parsl.dataflow.dflow DEBUG: Task 2354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/b2a2b8e9-9c30-4fa6-8d57-9a13fd163f24_isrForCrosstalkSources_855430_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/b2a2b8e9-9c30-4fa6-8d57-9a13fd163f24_isrForCrosstalkSources_855430_19.stderr parsl.dataflow.dflow INFO: Task 2369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c16a3a2346e38bb6b51c6c08c2c804de with result from task 2369 parsl.dataflow.dflow DEBUG: Task 2369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7d3809c0-2c69-44fd-b2d2-c7ec424d94d6_isrForCrosstalkSources_855437_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7d3809c0-2c69-44fd-b2d2-c7ec424d94d6_isrForCrosstalkSources_855437_40.stderr parsl.dataflow.dflow INFO: Task 2370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89763a5a98d8470b5cc1f4316b466c77 with result from task 2370 parsl.dataflow.dflow DEBUG: Task 2370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5b242329-1331-4bb7-8166-33e823255f12_isrForCrosstalkSources_855394_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5b242329-1331-4bb7-8166-33e823255f12_isrForCrosstalkSources_855394_18.stderr parsl.dataflow.dflow INFO: Task 2664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb8ca2d50525378b870fc8a32e16690 with result from task 2664 parsl.dataflow.dflow DEBUG: Task 2664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b039d81a-c362-49c2-9afe-839061cf75ca_isrForCrosstalkSources_855435_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b039d81a-c362-49c2-9afe-839061cf75ca_isrForCrosstalkSources_855435_11.stderr parsl.dataflow.dflow INFO: Task 2519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fb26bb0afb4c043d6ace63eb4c8c40e with result from task 2519 parsl.dataflow.dflow DEBUG: Task 2519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/75f20608-51e0-4b1b-bc8d-26f6916767c8_isrForCrosstalkSources_855381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/75f20608-51e0-4b1b-bc8d-26f6916767c8_isrForCrosstalkSources_855381_2.stderr parsl.dataflow.dflow INFO: Task 2344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deeee4a2bf89bff7b79352faf980d1af with result from task 2344 parsl.dataflow.dflow DEBUG: Task 2344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7b3bf4e9-a4d1-46af-9b0f-f04afd1386a9_isrForCrosstalkSources_855456_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7b3bf4e9-a4d1-46af-9b0f-f04afd1386a9_isrForCrosstalkSources_855456_20.stderr parsl.dataflow.dflow INFO: Task 2516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56c6558e85ade6c0fc0c05896f4b63cc with result from task 2516 parsl.dataflow.dflow DEBUG: Task 2516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/83d00e3e-1408-4cb9-a8c4-743b90675aaf_isrForCrosstalkSources_855398_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/83d00e3e-1408-4cb9-a8c4-743b90675aaf_isrForCrosstalkSources_855398_45.stderr parsl.dataflow.dflow INFO: Task 2349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a88344f9bace09fb1304ae0426a7df11 with result from task 2349 parsl.dataflow.dflow DEBUG: Task 2349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/fb7ed4fa-4989-4b96-bf7a-bc2cb4075f6f_isrForCrosstalkSources_855419_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/fb7ed4fa-4989-4b96-bf7a-bc2cb4075f6f_isrForCrosstalkSources_855419_11.stderr parsl.dataflow.dflow INFO: Task 2636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9273fc14a9c9fb7b81c8b8788180557 with result from task 2636 parsl.dataflow.dflow DEBUG: Task 2636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/997ea58c-a6bc-43df-bac2-bb9752cb2448_isrForCrosstalkSources_855452_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/997ea58c-a6bc-43df-bac2-bb9752cb2448_isrForCrosstalkSources_855452_19.stderr parsl.dataflow.dflow INFO: Task 2543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f2bb4d2abb47fee92b47c9847d5f367 with result from task 2543 parsl.dataflow.dflow DEBUG: Task 2543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ff47cab0-625f-40be-b9fa-53b3f7dfbb2c_isrForCrosstalkSources_855438_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/ff47cab0-625f-40be-b9fa-53b3f7dfbb2c_isrForCrosstalkSources_855438_33.stderr parsl.dataflow.dflow INFO: Task 2364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d26be314f357a25f7f17af88ad2b8ec7 with result from task 2364 parsl.dataflow.dflow DEBUG: Task 2364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/bbca41a8-8bc4-4ab5-ab6b-4561ffd7e6ea_isrForCrosstalkSources_855448_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/bbca41a8-8bc4-4ab5-ab6b-4561ffd7e6ea_isrForCrosstalkSources_855448_60.stderr parsl.dataflow.dflow INFO: Task 2648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3be507614cc4a7dffa6678825e179dec with result from task 2648 parsl.dataflow.dflow DEBUG: Task 2648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a5ce2269-71ce-4cb7-91e4-e1b53d805de8_isrForCrosstalkSources_855434_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a5ce2269-71ce-4cb7-91e4-e1b53d805de8_isrForCrosstalkSources_855434_35.stderr parsl.dataflow.dflow INFO: Task 2080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7973840e21a04c12bfa666179a3bd19 with result from task 2080 parsl.dataflow.dflow DEBUG: Task 2080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/384c3f45-c087-425c-84fb-306d1f0ea26c_isrForCrosstalkSources_855428_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/384c3f45-c087-425c-84fb-306d1f0ea26c_isrForCrosstalkSources_855428_58.stderr parsl.dataflow.dflow INFO: Task 2092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919ad2a5fbbb1d9db19277504f6628b2 with result from task 2092 parsl.dataflow.dflow DEBUG: Task 2092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/466e163a-5b8e-47c7-a4a0-912f4f1d7780_isrForCrosstalkSources_855405_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/466e163a-5b8e-47c7-a4a0-912f4f1d7780_isrForCrosstalkSources_855405_60.stderr parsl.dataflow.dflow INFO: Task 2501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5561ff9c9e618afddd4599efc26b6ce7 with result from task 2501 parsl.dataflow.dflow DEBUG: Task 2501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/cf3286dd-5d6d-45bf-bdcc-e3cb0d7c1213_isrForCrosstalkSources_855389_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/cf3286dd-5d6d-45bf-bdcc-e3cb0d7c1213_isrForCrosstalkSources_855389_29.stderr parsl.dataflow.dflow INFO: Task 2355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00df4396fcecb8e3744ea52aed20bcd6 with result from task 2355 parsl.dataflow.dflow DEBUG: Task 2355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8091a086-4f7b-4324-8e57-c4b7ecb1a348_isrForCrosstalkSources_855453_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8091a086-4f7b-4324-8e57-c4b7ecb1a348_isrForCrosstalkSources_855453_50.stderr parsl.dataflow.dflow INFO: Task 2279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 798724dc3287ac6dd7dadac01ccac16f with result from task 2279 parsl.dataflow.dflow DEBUG: Task 2279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dddb7ade-b0c7-4887-a6d1-3e86ebce764c_isrForCrosstalkSources_855458_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/dddb7ade-b0c7-4887-a6d1-3e86ebce764c_isrForCrosstalkSources_855458_30.stderr parsl.dataflow.dflow INFO: Task 2114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 467db57c8e9f6397535674d2ec9cbb32 with result from task 2114 parsl.dataflow.dflow DEBUG: Task 2114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/42d05cec-0a1f-4176-a2fc-6cb9bc15aa83_isrForCrosstalkSources_855415_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/42d05cec-0a1f-4176-a2fc-6cb9bc15aa83_isrForCrosstalkSources_855415_28.stderr parsl.dataflow.dflow INFO: Task 2178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6599b195a00d84ad41d72b8a07d0707c with result from task 2178 parsl.dataflow.dflow DEBUG: Task 2178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/62584125-5889-46e3-a278-1ed1ef0e010e_isrForCrosstalkSources_855427_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/62584125-5889-46e3-a278-1ed1ef0e010e_isrForCrosstalkSources_855427_31.stderr parsl.dataflow.dflow INFO: Task 2390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 844871557f0b0a0b62429512af1ef73d with result from task 2390 parsl.dataflow.dflow DEBUG: Task 2390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1d8adb58-10de-40d5-a8be-edd59da73695_isrForCrosstalkSources_855455_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/1d8adb58-10de-40d5-a8be-edd59da73695_isrForCrosstalkSources_855455_49.stderr parsl.dataflow.dflow INFO: Task 1995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31101982eb8a0a1e843e2350eafa4218 with result from task 1995 parsl.dataflow.dflow DEBUG: Task 1995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d67df830-cf9c-4057-8e4a-59c711403029_isrForCrosstalkSources_855392_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d67df830-cf9c-4057-8e4a-59c711403029_isrForCrosstalkSources_855392_55.stderr parsl.dataflow.dflow INFO: Task 2502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebce370c9afc8792f056b025d238882c with result from task 2502 parsl.dataflow.dflow DEBUG: Task 2502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f079f0b8-e0b4-4482-94f4-e85371bb9bf5_isrForCrosstalkSources_855379_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f079f0b8-e0b4-4482-94f4-e85371bb9bf5_isrForCrosstalkSources_855379_29.stderr parsl.dataflow.dflow INFO: Task 2183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d69ec9b8a1a7219f4ab33cc74389408c with result from task 2183 parsl.dataflow.dflow DEBUG: Task 2183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c43c8c7f-7171-4f39-833f-fd909359be6e_isrForCrosstalkSources_855419_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c43c8c7f-7171-4f39-833f-fd909359be6e_isrForCrosstalkSources_855419_35.stderr parsl.dataflow.dflow INFO: Task 2294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f11434c91503638459bd203768ee10 with result from task 2294 parsl.dataflow.dflow DEBUG: Task 2294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31656eec-9e22-4845-a86f-4b9ffa1978e4_isrForCrosstalkSources_855409_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31656eec-9e22-4845-a86f-4b9ffa1978e4_isrForCrosstalkSources_855409_43.stderr parsl.dataflow.dflow INFO: Task 2078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c57f58314dc260c9502086da3ddeb8e with result from task 2078 parsl.dataflow.dflow DEBUG: Task 2078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/19f74548-f7b7-486d-913d-8b9d569971e8_isrForCrosstalkSources_855404_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/19f74548-f7b7-486d-913d-8b9d569971e8_isrForCrosstalkSources_855404_54.stderr parsl.dataflow.dflow INFO: Task 2373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22dfec75f79ff652b8f61af9f89c0af2 with result from task 2373 parsl.dataflow.dflow DEBUG: Task 2373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9d6db98a-70eb-4819-9e28-28326266fc01_isrForCrosstalkSources_855414_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9d6db98a-70eb-4819-9e28-28326266fc01_isrForCrosstalkSources_855414_46.stderr parsl.dataflow.dflow INFO: Task 2226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4370c344b263ee40cee256c0aea75320 with result from task 2226 parsl.dataflow.dflow DEBUG: Task 2226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d7299c2c-5faa-4e9b-a670-62aa2bfe4475_isrForCrosstalkSources_855389_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d7299c2c-5faa-4e9b-a670-62aa2bfe4475_isrForCrosstalkSources_855389_18.stderr parsl.dataflow.dflow INFO: Task 2139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0654a6f2823d83a259a9971e34619a42 with result from task 2139 parsl.dataflow.dflow DEBUG: Task 2139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/63ce73e6-90ba-481e-ba00-4ec48e5ef5f0_isrForCrosstalkSources_855385_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/63ce73e6-90ba-481e-ba00-4ec48e5ef5f0_isrForCrosstalkSources_855385_16.stderr parsl.dataflow.dflow INFO: Task 2392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a41d9d5a1af2f27f28c75fddf2a89dbe with result from task 2392 parsl.dataflow.dflow DEBUG: Task 2392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/b2b7fff4-528c-41f9-afd1-ddf9f73f2d55_isrForCrosstalkSources_855392_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/b2b7fff4-528c-41f9-afd1-ddf9f73f2d55_isrForCrosstalkSources_855392_62.stderr parsl.dataflow.dflow INFO: Task 2207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de441fa0be02154e399eece55507e3bc with result from task 2207 parsl.dataflow.dflow DEBUG: Task 2207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/36b8c5d7-2d0f-4795-85d5-cdc9842d7c55_isrForCrosstalkSources_855434_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/36b8c5d7-2d0f-4795-85d5-cdc9842d7c55_isrForCrosstalkSources_855434_52.stderr parsl.dataflow.dflow INFO: Task 2365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de62a001ad1e38f96b835dd4b78383ec with result from task 2365 parsl.dataflow.dflow DEBUG: Task 2365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/7e627552-aa91-4242-bf87-03e0793e2a92_isrForCrosstalkSources_855426_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/7e627552-aa91-4242-bf87-03e0793e2a92_isrForCrosstalkSources_855426_16.stderr parsl.dataflow.dflow INFO: Task 2238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 651be80f14b5afdf91442ba5070ab96f with result from task 2238 parsl.dataflow.dflow DEBUG: Task 2238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31d33b0c-edef-4299-86db-60e4cf5424ec_isrForCrosstalkSources_855409_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/31d33b0c-edef-4299-86db-60e4cf5424ec_isrForCrosstalkSources_855409_31.stderr parsl.dataflow.dflow INFO: Task 2297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 359d75a459ef4f682d0b940414462a33 with result from task 2297 parsl.dataflow.dflow DEBUG: Task 2297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/52307f2e-d68b-4858-8097-69032fcaf850_isrForCrosstalkSources_855387_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/52307f2e-d68b-4858-8097-69032fcaf850_isrForCrosstalkSources_855387_14.stderr parsl.dataflow.dflow INFO: Task 2083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc090749f1762c69bf4d0bc45d681ed with result from task 2083 parsl.dataflow.dflow DEBUG: Task 2083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8e520ba5-323e-4282-9261-d74d467e2150_isrForCrosstalkSources_855445_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8e520ba5-323e-4282-9261-d74d467e2150_isrForCrosstalkSources_855445_49.stderr parsl.dataflow.dflow INFO: Task 2167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 062e26e0f220684706d058e3caa06e54 with result from task 2167 parsl.dataflow.dflow DEBUG: Task 2167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/90cb8e99-7c3e-476a-8d8b-6214edd07c64_isrForCrosstalkSources_855392_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/90cb8e99-7c3e-476a-8d8b-6214edd07c64_isrForCrosstalkSources_855392_2.stderr parsl.dataflow.dflow INFO: Task 2295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ef07de31d7ed2c6e3fb4cf55bc168f0 with result from task 2295 parsl.dataflow.dflow DEBUG: Task 2295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/81e62d03-193f-4d2b-affd-d70e030751a2_isrForCrosstalkSources_855442_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/81e62d03-193f-4d2b-affd-d70e030751a2_isrForCrosstalkSources_855442_17.stderr parsl.dataflow.dflow INFO: Task 2158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cac8102bc88c04e14d28a3da21f9c87 with result from task 2158 parsl.dataflow.dflow DEBUG: Task 2158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/70a9f4e0-e2f2-4bab-8f3e-4efc3129da0c_isrForCrosstalkSources_855399_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/70a9f4e0-e2f2-4bab-8f3e-4efc3129da0c_isrForCrosstalkSources_855399_56.stderr parsl.dataflow.dflow INFO: Task 2290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dabdcf9c9e7d8311abb118a6f765366 with result from task 2290 parsl.dataflow.dflow DEBUG: Task 2290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9f93f760-2440-4a49-a041-1ba81d33e8a0_isrForCrosstalkSources_855386_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/9f93f760-2440-4a49-a041-1ba81d33e8a0_isrForCrosstalkSources_855386_18.stderr parsl.dataflow.dflow INFO: Task 2196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72c94013f5cfb1d4a25f735008e1835b with result from task 2196 parsl.dataflow.dflow DEBUG: Task 2196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f37a00a2-441d-43a2-b5c8-879d48a91b6d_isrForCrosstalkSources_855428_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f37a00a2-441d-43a2-b5c8-879d48a91b6d_isrForCrosstalkSources_855428_40.stderr parsl.dataflow.dflow INFO: Task 2099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b32b94b46b0a39ed67ae249afce2dbe with result from task 2099 parsl.dataflow.dflow DEBUG: Task 2099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/04664b55-bb29-4346-889a-584fb9cb1bb9_isrForCrosstalkSources_855448_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/04664b55-bb29-4346-889a-584fb9cb1bb9_isrForCrosstalkSources_855448_44.stderr parsl.dataflow.dflow INFO: Task 2176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88dd9b40ed02337b383c770cfdc4be17 with result from task 2176 parsl.dataflow.dflow DEBUG: Task 2176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/30cbd2d7-1afb-44a6-9b46-59a5894492fb_isrForCrosstalkSources_855431_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/30cbd2d7-1afb-44a6-9b46-59a5894492fb_isrForCrosstalkSources_855431_43.stderr parsl.dataflow.dflow INFO: Task 2672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70184acfcef2158aa59a9ece8440805d with result from task 2672 parsl.dataflow.dflow DEBUG: Task 2672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/58ea4a9f-d77c-4b48-bbf2-6c1b01f569cc_isrForCrosstalkSources_855429_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/58ea4a9f-d77c-4b48-bbf2-6c1b01f569cc_isrForCrosstalkSources_855429_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:58 proc_lsst.multi:146 [INFO] found job 16964188 in provider checkpoint proc_lsst.multi INFO: found job 16964188 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c29b077337def96f734088d2c8f8d3c with result from task 2522 parsl.dataflow.dflow DEBUG: Task 2522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/6094fce6-59f7-4deb-a28d-cc0d8a1cd15c_isrForCrosstalkSources_855416_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/6094fce6-59f7-4deb-a28d-cc0d8a1cd15c_isrForCrosstalkSources_855416_10.stderr parsl.dataflow.dflow INFO: Task 2644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 888b4672728e094ec0190dacfce20644 with result from task 2644 parsl.dataflow.dflow DEBUG: Task 2644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3959229d-27e3-4da7-9468-037664625658_isrForCrosstalkSources_855442_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3959229d-27e3-4da7-9468-037664625658_isrForCrosstalkSources_855442_2.stderr parsl.dataflow.dflow INFO: Task 2616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478f945e29f0fdf3e7ee5ea26b7cc34e with result from task 2616 parsl.dataflow.dflow DEBUG: Task 2616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/62bd80a3-79c0-4258-a11a-8c2a5593b02e_isrForCrosstalkSources_855386_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/62bd80a3-79c0-4258-a11a-8c2a5593b02e_isrForCrosstalkSources_855386_59.stderr parsl.dataflow.dflow INFO: Task 2668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b15ac171608f940d4d8f43249ef49977 with result from task 2668 parsl.dataflow.dflow DEBUG: Task 2668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0cda55cd-c7fe-4710-a60b-6d0ad36a8b90_isrForCrosstalkSources_855457_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0cda55cd-c7fe-4710-a60b-6d0ad36a8b90_isrForCrosstalkSources_855457_55.stderr parsl.dataflow.dflow INFO: Task 2548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95d6a5b16f68ae82e72208ba3ee4ddf4 with result from task 2548 parsl.dataflow.dflow DEBUG: Task 2548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1a2fa7cf-067b-4f56-b7ad-846acae846b5_isrForCrosstalkSources_855383_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1a2fa7cf-067b-4f56-b7ad-846acae846b5_isrForCrosstalkSources_855383_12.stderr parsl.dataflow.dflow INFO: Task 2320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 369f12b0d6506d2fea9a3a8068ef6375 with result from task 2320 parsl.dataflow.dflow DEBUG: Task 2320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9fcdccba-e818-470a-9fa7-f7c219ee8344_isrForCrosstalkSources_855444_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9fcdccba-e818-470a-9fa7-f7c219ee8344_isrForCrosstalkSources_855444_43.stderr parsl.dataflow.dflow INFO: Task 2545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c1f61ea66da5596a7cae88a35afab39 with result from task 2545 parsl.dataflow.dflow DEBUG: Task 2545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/755b2108-1702-4bb0-81b7-d28a92d6e3cb_isrForCrosstalkSources_855399_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/755b2108-1702-4bb0-81b7-d28a92d6e3cb_isrForCrosstalkSources_855399_41.stderr parsl.dataflow.dflow INFO: Task 2610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8cc85c0414959be84c4acd1646d5f47 with result from task 2610 parsl.dataflow.dflow DEBUG: Task 2610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be4cecd2-ff8b-4964-b25a-43cd7ff9f797_isrForCrosstalkSources_855425_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/be4cecd2-ff8b-4964-b25a-43cd7ff9f797_isrForCrosstalkSources_855425_2.stderr parsl.dataflow.dflow INFO: Task 2608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10a8d1fe04610d736a203b791969585c with result from task 2608 parsl.dataflow.dflow DEBUG: Task 2608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/61e031f6-1351-4242-9e7b-40c2584a30b2_isrForCrosstalkSources_855403_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/61e031f6-1351-4242-9e7b-40c2584a30b2_isrForCrosstalkSources_855403_50.stderr parsl.dataflow.dflow INFO: Task 2605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 871bfb8c381882942c317fea6ed119bc with result from task 2605 parsl.dataflow.dflow DEBUG: Task 2605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/37ad1e4e-b503-42ec-ac0d-3bdbefa52b59_isrForCrosstalkSources_855379_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/37ad1e4e-b503-42ec-ac0d-3bdbefa52b59_isrForCrosstalkSources_855379_62.stderr parsl.dataflow.dflow INFO: Task 2496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfb5eff276bfc9efb70f07c1eaf8a4fc with result from task 2496 parsl.dataflow.dflow DEBUG: Task 2496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b1e4fac6-5457-4fe4-ab8d-456bdf4cc0bd_isrForCrosstalkSources_855411_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b1e4fac6-5457-4fe4-ab8d-456bdf4cc0bd_isrForCrosstalkSources_855411_8.stderr parsl.dataflow.dflow INFO: Task 2702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cce15a342631d1f9446319d7e7e1c95e with result from task 2702 parsl.dataflow.dflow DEBUG: Task 2702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/15fb0303-717a-4434-97d8-01b7704c233f_isrForCrosstalkSources_855436_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/15fb0303-717a-4434-97d8-01b7704c233f_isrForCrosstalkSources_855436_19.stderr parsl.dataflow.dflow INFO: Task 2326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72bd093dcbef3fffe94e31c3270040c9 with result from task 2326 parsl.dataflow.dflow DEBUG: Task 2326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/bc341306-3e8b-4d9f-8763-d9c08a3b4ea9_isrForCrosstalkSources_855439_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/bc341306-3e8b-4d9f-8763-d9c08a3b4ea9_isrForCrosstalkSources_855439_12.stderr parsl.dataflow.dflow INFO: Task 2337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3130b9631da7ba44fa9456614c147fd5 with result from task 2337 parsl.dataflow.dflow DEBUG: Task 2337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/cb33ef6d-c505-40df-bb7c-f42ba8b9b4df_isrForCrosstalkSources_855398_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/cb33ef6d-c505-40df-bb7c-f42ba8b9b4df_isrForCrosstalkSources_855398_4.stderr parsl.dataflow.dflow INFO: Task 2622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99f9c1682c37396050e996e91e2906ee with result from task 2622 parsl.dataflow.dflow DEBUG: Task 2622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a162ada0-d948-4189-a7cf-289d98dd36b3_isrForCrosstalkSources_855380_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a162ada0-d948-4189-a7cf-289d98dd36b3_isrForCrosstalkSources_855380_33.stderr parsl.dataflow.dflow INFO: Task 2327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da30fefd66b4cb98f87d93509dec68d1 with result from task 2327 parsl.dataflow.dflow DEBUG: Task 2327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5c25d440-6301-471d-b916-c261fc6b82d4_isrForCrosstalkSources_855419_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5c25d440-6301-471d-b916-c261fc6b82d4_isrForCrosstalkSources_855419_37.stderr parsl.dataflow.dflow INFO: Task 2322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5ec657275d85580475f4ad7a85a5f4f with result from task 2322 parsl.dataflow.dflow DEBUG: Task 2322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c0c7f2ee-fe43-4c24-a8ab-07db0a81da72_isrForCrosstalkSources_855406_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/c0c7f2ee-fe43-4c24-a8ab-07db0a81da72_isrForCrosstalkSources_855406_40.stderr parsl.dataflow.dflow INFO: Task 2415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b129b2b2e7a96060ffbdb7fbfff41f1e with result from task 2415 parsl.dataflow.dflow DEBUG: Task 2415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8973f42b-49be-4559-85b4-998fdbde92f3_isrForCrosstalkSources_855446_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8973f42b-49be-4559-85b4-998fdbde92f3_isrForCrosstalkSources_855446_51.stderr parsl.dataflow.dflow INFO: Task 2332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf30b73ee027d65463fc76436468ff33 with result from task 2332 parsl.dataflow.dflow DEBUG: Task 2332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2cd5d636-08c1-4a6a-a747-86cbd0f7689d_isrForCrosstalkSources_855388_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2cd5d636-08c1-4a6a-a747-86cbd0f7689d_isrForCrosstalkSources_855388_37.stderr parsl.dataflow.dflow INFO: Task 2330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a917519a119e914f4e57796628dd12 with result from task 2330 parsl.dataflow.dflow DEBUG: Task 2330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/689d4ece-bede-45e1-8c10-99d92bac0cc1_isrForCrosstalkSources_855428_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/689d4ece-bede-45e1-8c10-99d92bac0cc1_isrForCrosstalkSources_855428_15.stderr parsl.dataflow.dflow INFO: Task 2352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b66d593e8ddf0e998167d537d008122d with result from task 2352 parsl.dataflow.dflow DEBUG: Task 2352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a9abf411-27d1-4a39-933a-1ab7b6c06a84_isrForCrosstalkSources_855391_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a9abf411-27d1-4a39-933a-1ab7b6c06a84_isrForCrosstalkSources_855391_28.stderr parsl.dataflow.dflow INFO: Task 2682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f520003b95a48ea25cb065c4fa28360 with result from task 2682 parsl.dataflow.dflow DEBUG: Task 2682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fa6edf02-39c7-4ada-902c-3ed4d97fed56_isrForCrosstalkSources_855417_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fa6edf02-39c7-4ada-902c-3ed4d97fed56_isrForCrosstalkSources_855417_10.stderr parsl.dataflow.dflow INFO: Task 2421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f363baa335183232be486d826e796f3a with result from task 2421 parsl.dataflow.dflow DEBUG: Task 2421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0140d1bb-43b8-437a-9f6f-7e481a30a21a_isrForCrosstalkSources_855417_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0140d1bb-43b8-437a-9f6f-7e481a30a21a_isrForCrosstalkSources_855417_16.stderr parsl.dataflow.dflow INFO: Task 2384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b55b3b1ab65f3e9627939eac3347c8c with result from task 2384 parsl.dataflow.dflow DEBUG: Task 2384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/68e20802-fa96-47ee-b345-7431d8fbfda8_isrForCrosstalkSources_855445_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/68e20802-fa96-47ee-b345-7431d8fbfda8_isrForCrosstalkSources_855445_52.stderr parsl.dataflow.dflow INFO: Task 2612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2348ab559200e1e6d7dde8d156d40ff4 with result from task 2612 parsl.dataflow.dflow DEBUG: Task 2612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ddcb12f2-a888-4810-a8e5-fff36e9d8239_isrForCrosstalkSources_855403_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ddcb12f2-a888-4810-a8e5-fff36e9d8239_isrForCrosstalkSources_855403_31.stderr parsl.dataflow.dflow INFO: Task 2695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6102a828006d478c2ea5c36c1d9363a with result from task 2695 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54547a89-ea3b-49a4-bd9a-d227fdc3778d_isrForCrosstalkSources_855445_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54547a89-ea3b-49a4-bd9a-d227fdc3778d_isrForCrosstalkSources_855445_18.stderr parsl.dataflow.dflow INFO: Task 2417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c81c21526ee3a60a47e93781203d5dfd with result from task 2417 parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:59 proc_lsst.multi:146 [INFO] found job 16964189 in provider checkpoint proc_lsst.multi INFO: found job 16964189 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/acef806a-cfe5-4f97-b702-8da529f7c0fe_isrForCrosstalkSources_855423_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/acef806a-cfe5-4f97-b702-8da529f7c0fe_isrForCrosstalkSources_855423_54.stderr parsl.dataflow.dflow INFO: Task 2325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c2758b2947cce888a8a6f5f77d3694 with result from task 2325 parsl.dataflow.dflow DEBUG: Task 2325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc7d9708-1510-4b3c-96f4-93ca422461c0_isrForCrosstalkSources_855403_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/bc7d9708-1510-4b3c-96f4-93ca422461c0_isrForCrosstalkSources_855403_52.stderr parsl.dataflow.dflow INFO: Task 2329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c81d2424de923439b327769815a36696 with result from task 2329 parsl.dataflow.dflow DEBUG: Task 2329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/344543a8-7806-441e-bf65-1a9d7e559ae2_isrForCrosstalkSources_855424_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/344543a8-7806-441e-bf65-1a9d7e559ae2_isrForCrosstalkSources_855424_17.stderr parsl.dataflow.dflow INFO: Task 2382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 632332b5a9ec367e27d67454f171edc8 with result from task 2382 parsl.dataflow.dflow DEBUG: Task 2382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d51c4726-7730-40d7-9ea6-8c27a6d63640_isrForCrosstalkSources_855423_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/d51c4726-7730-40d7-9ea6-8c27a6d63640_isrForCrosstalkSources_855423_2.stderr parsl.dataflow.dflow INFO: Task 2639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86dbb1701c34a989bb31e3005427a3a0 with result from task 2639 parsl.dataflow.dflow DEBUG: Task 2639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f8aaa21-1a7a-4ebd-b915-eb9d4589cd97_isrForCrosstalkSources_855441_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f8aaa21-1a7a-4ebd-b915-eb9d4589cd97_isrForCrosstalkSources_855441_43.stderr parsl.dataflow.dflow INFO: Task 2630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d838d99bf3a279e5145ec7ab464f08 with result from task 2630 parsl.dataflow.dflow DEBUG: Task 2630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/71d7052f-7a4c-47a3-80c0-4a69ebeb8615_isrForCrosstalkSources_855407_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/71d7052f-7a4c-47a3-80c0-4a69ebeb8615_isrForCrosstalkSources_855407_14.stderr parsl.dataflow.dflow INFO: Task 2697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7b6301cb21fd1d5540229a509acd979 with result from task 2697 parsl.dataflow.dflow DEBUG: Task 2697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/3a347184-1302-4745-9f16-c3e31b350a9e_isrForCrosstalkSources_855398_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/3a347184-1302-4745-9f16-c3e31b350a9e_isrForCrosstalkSources_855398_13.stderr parsl.dataflow.dflow INFO: Task 2508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da74dc92fb8d4de97bd314a1c118d6f8 with result from task 2508 parsl.dataflow.dflow DEBUG: Task 2508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b8d2b781-4cc1-4721-b5a3-e5354e92873b_isrForCrosstalkSources_855425_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b8d2b781-4cc1-4721-b5a3-e5354e92873b_isrForCrosstalkSources_855425_26.stderr parsl.dataflow.dflow INFO: Task 2331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce66ed810ededc1bccc2cc1ae8e38bce with result from task 2331 parsl.dataflow.dflow DEBUG: Task 2331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0893d78e-8bf0-4874-acdb-cb5279b1f84a_isrForCrosstalkSources_855405_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0893d78e-8bf0-4874-acdb-cb5279b1f84a_isrForCrosstalkSources_855405_39.stderr parsl.dataflow.dflow INFO: Task 2328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eafca3cb7cdb224143f1fa9eba73bffa with result from task 2328 parsl.dataflow.dflow DEBUG: Task 2328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/54a71b9b-d43a-4b7c-998c-0d604071dfe4_isrForCrosstalkSources_855431_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/54a71b9b-d43a-4b7c-998c-0d604071dfe4_isrForCrosstalkSources_855431_54.stderr parsl.dataflow.dflow INFO: Task 2541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e2e23cc849690e8843347a020017131 with result from task 2541 parsl.dataflow.dflow DEBUG: Task 2541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/39d206e5-1847-418e-a8a0-3cbb5f67b4cf_isrForCrosstalkSources_855413_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/39d206e5-1847-418e-a8a0-3cbb5f67b4cf_isrForCrosstalkSources_855413_55.stderr parsl.dataflow.dflow INFO: Task 2609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14545e78a7613b179a48cb78c2d63a95 with result from task 2609 parsl.dataflow.dflow DEBUG: Task 2609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f4d0fa0b-fd9c-42dc-9d81-9aa074612bc3_isrForCrosstalkSources_855451_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f4d0fa0b-fd9c-42dc-9d81-9aa074612bc3_isrForCrosstalkSources_855451_17.stderr parsl.dataflow.dflow INFO: Task 2547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8edf9d1d099bdec5f5e9362938ce30bd with result from task 2547 parsl.dataflow.dflow DEBUG: Task 2547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a39353c0-a578-48bf-8399-e55d3c81fb4a_isrForCrosstalkSources_855398_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a39353c0-a578-48bf-8399-e55d3c81fb4a_isrForCrosstalkSources_855398_1.stderr parsl.dataflow.dflow INFO: Task 2666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862fce69e013523992312d78f4ebe59b with result from task 2666 parsl.dataflow.dflow DEBUG: Task 2666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/3e10f6fd-e4f8-4039-9ce5-d3d4f84ec232_isrForCrosstalkSources_855428_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/3e10f6fd-e4f8-4039-9ce5-d3d4f84ec232_isrForCrosstalkSources_855428_16.stderr parsl.dataflow.dflow INFO: Task 2615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f5ed09f81323e3fba890f67bb7badb1 with result from task 2615 parsl.dataflow.dflow DEBUG: Task 2615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/785093c9-3116-4fad-9629-cc4f7e93c57a_isrForCrosstalkSources_855458_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/785093c9-3116-4fad-9629-cc4f7e93c57a_isrForCrosstalkSources_855458_48.stderr parsl.dataflow.dflow INFO: Task 2646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04d46b7fe9786d5b5cb82095d65fdfea with result from task 2646 parsl.dataflow.dflow DEBUG: Task 2646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f33c1c05-9c25-40fd-83fd-67f404e13b94_isrForCrosstalkSources_855428_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f33c1c05-9c25-40fd-83fd-67f404e13b94_isrForCrosstalkSources_855428_6.stderr parsl.dataflow.dflow INFO: Task 2607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ef56c3019ebc807a826b1784c6363ec with result from task 2607 parsl.dataflow.dflow DEBUG: Task 2607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab0d87a1-8d99-471e-aa05-aba3293509bc_isrForCrosstalkSources_855420_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab0d87a1-8d99-471e-aa05-aba3293509bc_isrForCrosstalkSources_855420_8.stderr parsl.dataflow.dflow INFO: Task 2336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39c3950ff7b1a36437937d15523b6dd with result from task 2336 parsl.dataflow.dflow DEBUG: Task 2336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/51ed1c51-12f7-4836-84b0-4cd2b5fb7a43_isrForCrosstalkSources_855437_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/51ed1c51-12f7-4836-84b0-4cd2b5fb7a43_isrForCrosstalkSources_855437_27.stderr parsl.dataflow.dflow INFO: Task 2324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc54df362e42719bebd6e1e802f9f6b with result from task 2324 parsl.dataflow.dflow DEBUG: Task 2324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/787b1826-4ba6-4e53-961a-05ddc3e9c917_isrForCrosstalkSources_855454_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/787b1826-4ba6-4e53-961a-05ddc3e9c917_isrForCrosstalkSources_855454_7.stderr parsl.dataflow.dflow INFO: Task 2645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a545224f7a8fa9c7b948ce83c704b23 with result from task 2645 parsl.dataflow.dflow DEBUG: Task 2645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fb1dfe06-9662-4a82-89d2-f77614de6f5d_isrForCrosstalkSources_855401_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fb1dfe06-9662-4a82-89d2-f77614de6f5d_isrForCrosstalkSources_855401_33.stderr parsl.dataflow.dflow INFO: Task 2405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90fa5e7de560a7b77edbd058e00dfa59 with result from task 2405 parsl.dataflow.dflow DEBUG: Task 2405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/171dec6d-b74c-4ff7-aa9e-f0bf86033de8_isrForCrosstalkSources_855427_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/171dec6d-b74c-4ff7-aa9e-f0bf86033de8_isrForCrosstalkSources_855427_37.stderr parsl.dataflow.dflow INFO: Task 2685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08fbc75ecb1c92765a6eff9057428e26 with result from task 2685 parsl.dataflow.dflow DEBUG: Task 2685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4cb625d7-df84-40b1-a313-8ab40ec33ed6_isrForCrosstalkSources_855379_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4cb625d7-df84-40b1-a313-8ab40ec33ed6_isrForCrosstalkSources_855379_50.stderr parsl.dataflow.dflow INFO: Task 2613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc2cd0cb1bd75a324d5a4c1c18d02d1 with result from task 2613 parsl.dataflow.dflow DEBUG: Task 2613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7c7c610f-9533-476f-8008-43c4440e875b_isrForCrosstalkSources_855459_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/7c7c610f-9533-476f-8008-43c4440e875b_isrForCrosstalkSources_855459_25.stderr parsl.dataflow.dflow INFO: Task 2353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61bbad7d69c1360631876a76dd30d023 with result from task 2353 parsl.dataflow.dflow DEBUG: Task 2353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1bff1e60-a5c9-49f2-b9c2-863620b0ece3_isrForCrosstalkSources_855392_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1bff1e60-a5c9-49f2-b9c2-863620b0ece3_isrForCrosstalkSources_855392_39.stderr parsl.dataflow.dflow INFO: Task 2318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39483f3cd871c3b068d42e45942670a4 with result from task 2318 parsl.dataflow.dflow DEBUG: Task 2318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/884c4e4d-107f-4897-a054-5d01c03c76b3_isrForCrosstalkSources_855427_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/884c4e4d-107f-4897-a054-5d01c03c76b3_isrForCrosstalkSources_855427_40.stderr parsl.dataflow.dflow INFO: Task 2340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 334d7b51e806c1123561668466013543 with result from task 2340 parsl.dataflow.dflow DEBUG: Task 2340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/904c2f1e-61cf-4668-8ba5-c6ec084388d4_isrForCrosstalkSources_855399_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/904c2f1e-61cf-4668-8ba5-c6ec084388d4_isrForCrosstalkSources_855399_9.stderr parsl.dataflow.dflow INFO: Task 2620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5193272aaaa0205c79b4a5cc34919e61 with result from task 2620 parsl.dataflow.dflow DEBUG: Task 2620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ed8f32c-558d-42db-8873-4fb0dded2c19_isrForCrosstalkSources_855396_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ed8f32c-558d-42db-8873-4fb0dded2c19_isrForCrosstalkSources_855396_47.stderr parsl.dataflow.dflow INFO: Task 2301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aeb67e23919fd86e737a23768e571c7 with result from task 2301 parsl.dataflow.dflow DEBUG: Task 2301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/11cafe19-4d3b-4252-8bd9-cee93b01a8a3_isrForCrosstalkSources_855412_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/11cafe19-4d3b-4252-8bd9-cee93b01a8a3_isrForCrosstalkSources_855412_49.stderr parsl.dataflow.dflow INFO: Task 2334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76546bbbc53d4d717b62a2f5d25a8e44 with result from task 2334 parsl.dataflow.dflow DEBUG: Task 2334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b21f96c9-10c2-4598-b0a9-a1f50e780f10_isrForCrosstalkSources_855446_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b21f96c9-10c2-4598-b0a9-a1f50e780f10_isrForCrosstalkSources_855446_20.stderr parsl.dataflow.dflow INFO: Task 2688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73bdc4aa3800f5b222f161bf39a03a82 with result from task 2688 parsl.dataflow.dflow DEBUG: Task 2688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1d8d4eb0-2d21-4b62-ac5b-cb528611696b_isrForCrosstalkSources_855423_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1d8d4eb0-2d21-4b62-ac5b-cb528611696b_isrForCrosstalkSources_855423_11.stderr parsl.dataflow.dflow INFO: Task 2360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a318d150d62986f052cb2feb161a4a98 with result from task 2360 parsl.dataflow.dflow DEBUG: Task 2360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/90f08de3-4d23-4f7d-b773-d8b0b70ec0a2_isrForCrosstalkSources_855387_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/90f08de3-4d23-4f7d-b773-d8b0b70ec0a2_isrForCrosstalkSources_855387_15.stderr parsl.dataflow.dflow INFO: Task 2345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e642375255c62b13d2f186f778298d92 with result from task 2345 parsl.dataflow.dflow DEBUG: Task 2345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/69cebed1-9e09-4616-88f2-fb811de48b7a_isrForCrosstalkSources_855455_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/69cebed1-9e09-4616-88f2-fb811de48b7a_isrForCrosstalkSources_855455_61.stderr parsl.dataflow.dflow INFO: Task 2518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 156720f7b75381271b6abe12a8f319d3 with result from task 2518 parsl.dataflow.dflow DEBUG: Task 2518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/61f39e8d-132d-44a8-b8c7-2bfa62959b54_isrForCrosstalkSources_855424_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/61f39e8d-132d-44a8-b8c7-2bfa62959b54_isrForCrosstalkSources_855424_15.stderr parsl.dataflow.dflow INFO: Task 2558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4db8e88c9ed00dc09ec3c555696750 with result from task 2558 parsl.dataflow.dflow DEBUG: Task 2558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a20a60f7-aaf7-417d-8859-ede176d8b590_isrForCrosstalkSources_855383_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a20a60f7-aaf7-417d-8859-ede176d8b590_isrForCrosstalkSources_855383_21.stderr parsl.dataflow.dflow INFO: Task 2333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069ac4cefe83e614b0966cf46828b8ca with result from task 2333 parsl.dataflow.dflow DEBUG: Task 2333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3094d68c-8e0d-49cc-af0c-09b65ed6ed8d_isrForCrosstalkSources_855403_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3094d68c-8e0d-49cc-af0c-09b65ed6ed8d_isrForCrosstalkSources_855403_32.stderr parsl.dataflow.dflow INFO: Task 2323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dc7007d72c28b7a235d91e8f1dd1a6c with result from task 2323 parsl.dataflow.dflow DEBUG: Task 2323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f3b7e1fb-49aa-4f0a-ba73-79e62f27c33b_isrForCrosstalkSources_855407_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f3b7e1fb-49aa-4f0a-ba73-79e62f27c33b_isrForCrosstalkSources_855407_43.stderr parsl.dataflow.dflow INFO: Task 2584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2353a4cc298ae5831aa164b346722b68 with result from task 2584 parsl.dataflow.dflow DEBUG: Task 2584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/35444b75-964f-4c87-b470-ef28fe88ddfd_isrForCrosstalkSources_855440_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/35444b75-964f-4c87-b470-ef28fe88ddfd_isrForCrosstalkSources_855440_55.stderr parsl.dataflow.dflow INFO: Task 2656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1cfb2b71ecf7b7b7e16985aeba5366b with result from task 2656 parsl.dataflow.dflow DEBUG: Task 2656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d35e5f44-169e-427b-a90e-991f7020afde_isrForCrosstalkSources_855428_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d35e5f44-169e-427b-a90e-991f7020afde_isrForCrosstalkSources_855428_7.stderr parsl.dataflow.dflow INFO: Task 2351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed0d089e1232d4cde5f0348c94d43ef4 with result from task 2351 parsl.dataflow.dflow DEBUG: Task 2351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/23d78b1f-6a8a-45dc-92b1-2a089f1a0d8c_isrForCrosstalkSources_855456_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/23d78b1f-6a8a-45dc-92b1-2a089f1a0d8c_isrForCrosstalkSources_855456_44.stderr parsl.dataflow.dflow INFO: Task 2683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feca53257fb49468538b252e487f2d52 with result from task 2683 parsl.dataflow.dflow DEBUG: Task 2683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/df1739c4-bdaa-4a23-901d-68131c907316_isrForCrosstalkSources_855379_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/df1739c4-bdaa-4a23-901d-68131c907316_isrForCrosstalkSources_855379_9.stderr parsl.dataflow.dflow INFO: Task 2655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7493dc983cc1d42a239316979c5295e5 with result from task 2655 parsl.dataflow.dflow DEBUG: Task 2655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c672642f-b9d7-457e-8f1a-c981a13417c5_isrForCrosstalkSources_855451_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c672642f-b9d7-457e-8f1a-c981a13417c5_isrForCrosstalkSources_855451_3.stderr parsl.dataflow.dflow INFO: Task 2643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b416f7e35c38f946176fd864cd332bb1 with result from task 2643 parsl.dataflow.dflow DEBUG: Task 2643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5b48c56d-bd0d-41cb-b03b-0cdaaa7cb52c_isrForCrosstalkSources_855435_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5b48c56d-bd0d-41cb-b03b-0cdaaa7cb52c_isrForCrosstalkSources_855435_28.stderr parsl.dataflow.dflow INFO: Task 2498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7fb2c73d6cc36c904d2fe8d69f4c13f with result from task 2498 parsl.dataflow.dflow DEBUG: Task 2498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e2d78afc-4e74-4dc2-95c3-17152a03d1d0_isrForCrosstalkSources_855399_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e2d78afc-4e74-4dc2-95c3-17152a03d1d0_isrForCrosstalkSources_855399_44.stderr parsl.dataflow.dflow INFO: Task 2372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93063b92349539b31bc56c567b0e897e with result from task 2372 parsl.dataflow.dflow DEBUG: Task 2372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/916de712-9e26-42b8-a82e-e91e5dd6a709_isrForCrosstalkSources_855455_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/916de712-9e26-42b8-a82e-e91e5dd6a709_isrForCrosstalkSources_855455_36.stderr parsl.dataflow.dflow INFO: Task 2367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f0e0a7762f9ff5e66a58fefa82fa13 with result from task 2367 parsl.dataflow.dflow DEBUG: Task 2367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/718f1584-8339-4cce-b492-9d3b28c3d64c_isrForCrosstalkSources_855456_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/718f1584-8339-4cce-b492-9d3b28c3d64c_isrForCrosstalkSources_855456_28.stderr parsl.dataflow.dflow INFO: Task 2539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816770326bee61a9ffc871b16ea81370 with result from task 2539 parsl.dataflow.dflow DEBUG: Task 2539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/72dd28ea-5ff5-41c1-893e-2dff6343b322_isrForCrosstalkSources_855442_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/72dd28ea-5ff5-41c1-893e-2dff6343b322_isrForCrosstalkSources_855442_43.stderr parsl.dataflow.dflow INFO: Task 2569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46307bd573b66c1044687d3435127dce with result from task 2569 parsl.dataflow.dflow DEBUG: Task 2569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5b795e20-e9ee-4461-affa-7f17c2329eb8_isrForCrosstalkSources_855422_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/5b795e20-e9ee-4461-affa-7f17c2329eb8_isrForCrosstalkSources_855422_52.stderr parsl.dataflow.dflow INFO: Task 2338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c07bb5ad211166a83acc0a2152a6f31 with result from task 2338 parsl.dataflow.dflow DEBUG: Task 2338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8deccc42-8486-4da0-a9bb-51e46b38cef8_isrForCrosstalkSources_855398_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8deccc42-8486-4da0-a9bb-51e46b38cef8_isrForCrosstalkSources_855398_20.stderr parsl.dataflow.dflow INFO: Task 2538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5db64b8b5da2f79591b2ee95a7a43d7 with result from task 2538 parsl.dataflow.dflow DEBUG: Task 2538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/3908a21e-cd0a-489c-821b-bf4558df7c56_isrForCrosstalkSources_855444_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/3908a21e-cd0a-489c-821b-bf4558df7c56_isrForCrosstalkSources_855444_7.stderr parsl.dataflow.dflow INFO: Task 2515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80a2c789d6cbd1906ed4857e0c420dbe with result from task 2515 parsl.dataflow.dflow DEBUG: Task 2515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b83c57f3-9e98-4e91-b1ec-d5c5eddccbd9_isrForCrosstalkSources_855452_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b83c57f3-9e98-4e91-b1ec-d5c5eddccbd9_isrForCrosstalkSources_855452_45.stderr parsl.dataflow.dflow INFO: Task 2588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 360b0bf900dabeb1d14e296b39cd529d with result from task 2588 parsl.dataflow.dflow DEBUG: Task 2588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8bbfdb9e-1539-4be5-9441-9982851ce31d_isrForCrosstalkSources_855408_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8bbfdb9e-1539-4be5-9441-9982851ce31d_isrForCrosstalkSources_855408_36.stderr parsl.dataflow.dflow INFO: Task 2397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75ff01a3e519638df87b3ac58eb70d20 with result from task 2397 parsl.dataflow.dflow DEBUG: Task 2397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c5640b09-d786-4725-ad15-c26340270411_isrForCrosstalkSources_855439_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/c5640b09-d786-4725-ad15-c26340270411_isrForCrosstalkSources_855439_48.stderr parsl.dataflow.dflow INFO: Task 2461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed54e5fc3bb933b93429f95423884c4 with result from task 2461 parsl.dataflow.dflow DEBUG: Task 2461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6ab4bb0c-0cd9-4e67-8079-7500138d4ba2_isrForCrosstalkSources_855413_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/6ab4bb0c-0cd9-4e67-8079-7500138d4ba2_isrForCrosstalkSources_855413_44.stderr parsl.dataflow.dflow INFO: Task 2308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478f474d9f591912719b45e310e1b856 with result from task 2308 parsl.dataflow.dflow DEBUG: Task 2308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b7c73c2e-2c9e-484a-88a2-ec892987f8d1_isrForCrosstalkSources_855396_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/b7c73c2e-2c9e-484a-88a2-ec892987f8d1_isrForCrosstalkSources_855396_36.stderr parsl.dataflow.dflow INFO: Task 2495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ef128c02e501e513fc1b20fe7c8ddbd with result from task 2495 parsl.dataflow.dflow DEBUG: Task 2495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/308799e6-c0ac-4305-8a69-2bd354aea7d4_isrForCrosstalkSources_855443_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/308799e6-c0ac-4305-8a69-2bd354aea7d4_isrForCrosstalkSources_855443_62.stderr parsl.dataflow.dflow INFO: Task 2299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c0e4ef72ab6072f2725423b97780ae1 with result from task 2299 parsl.dataflow.dflow DEBUG: Task 2299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/84b65478-d134-4b01-bc9b-0bc4bc82e301_isrForCrosstalkSources_855402_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/84b65478-d134-4b01-bc9b-0bc4bc82e301_isrForCrosstalkSources_855402_53.stderr parsl.dataflow.dflow INFO: Task 2414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a325289e024287e2d30f3196752b5547 with result from task 2414 parsl.dataflow.dflow DEBUG: Task 2414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/24ccd384-3ccc-4991-906d-16d55ab97148_isrForCrosstalkSources_855421_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/24ccd384-3ccc-4991-906d-16d55ab97148_isrForCrosstalkSources_855421_62.stderr parsl.dataflow.dflow INFO: Task 2514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abeb14d00572a01992e71be119ef9907 with result from task 2514 parsl.dataflow.dflow DEBUG: Task 2514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/71c3a46e-10f1-4768-8454-4cfba64a8531_isrForCrosstalkSources_855415_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/71c3a46e-10f1-4768-8454-4cfba64a8531_isrForCrosstalkSources_855415_6.stderr parsl.dataflow.dflow INFO: Task 2649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 987be4866672968908498d1ccc63ea97 with result from task 2649 parsl.dataflow.dflow DEBUG: Task 2649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d7dad6bf-e353-4717-bedb-7057a30d8add_isrForCrosstalkSources_855428_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d7dad6bf-e353-4717-bedb-7057a30d8add_isrForCrosstalkSources_855428_2.stderr parsl.dataflow.dflow INFO: Task 2385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa9bc436c4d858c02a6b322b22ffc51 with result from task 2385 parsl.dataflow.dflow DEBUG: Task 2385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4c24027-050e-47b8-afef-ad427dd0ff45_isrForCrosstalkSources_855416_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/c4c24027-050e-47b8-afef-ad427dd0ff45_isrForCrosstalkSources_855416_29.stderr parsl.dataflow.dflow INFO: Task 2555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d9f3a8a34598eb2a30697ab26a7ca35 with result from task 2555 parsl.dataflow.dflow DEBUG: Task 2555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/196d0f63-7d92-42ee-9718-efda3e9f2a17_isrForCrosstalkSources_855457_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/196d0f63-7d92-42ee-9718-efda3e9f2a17_isrForCrosstalkSources_855457_20.stderr parsl.dataflow.dflow INFO: Task 2493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b92533bbfd32abc862103167ca9f4cf with result from task 2493 parsl.dataflow.dflow DEBUG: Task 2493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c378fb7b-18c3-4994-8692-56a91122f305_isrForCrosstalkSources_855448_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c378fb7b-18c3-4994-8692-56a91122f305_isrForCrosstalkSources_855448_16.stderr parsl.dataflow.dflow INFO: Task 2378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da4c6470409ea8c9ee8d569530cf1a96 with result from task 2378 parsl.dataflow.dflow DEBUG: Task 2378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/934d996a-2bca-4d4f-b6c5-209ccaf5d274_isrForCrosstalkSources_855407_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/934d996a-2bca-4d4f-b6c5-209ccaf5d274_isrForCrosstalkSources_855407_54.stderr parsl.dataflow.dflow INFO: Task 2197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cb72c99053812d12d46f9b38f2985e5 with result from task 2197 parsl.dataflow.dflow DEBUG: Task 2197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/ed191250-5e4b-407f-95c4-df8bf3d6348a_isrForCrosstalkSources_855408_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/ed191250-5e4b-407f-95c4-df8bf3d6348a_isrForCrosstalkSources_855408_61.stderr parsl.dataflow.dflow INFO: Task 2565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cddcd824833efba9f784c79d40b15069 with result from task 2565 parsl.dataflow.dflow DEBUG: Task 2565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dbf1a0a9-40f6-45b4-bd22-96d5d4c213ec_isrForCrosstalkSources_855441_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dbf1a0a9-40f6-45b4-bd22-96d5d4c213ec_isrForCrosstalkSources_855441_34.stderr parsl.dataflow.dflow INFO: Task 2413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8b5ba1ba21c8f90852fa0ed60be6fc with result from task 2413 parsl.dataflow.dflow DEBUG: Task 2413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9e248002-4b1a-4034-981a-1c92022ebc0c_isrForCrosstalkSources_855427_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9e248002-4b1a-4034-981a-1c92022ebc0c_isrForCrosstalkSources_855427_54.stderr parsl.dataflow.dflow INFO: Task 2582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cec39cd79b987c6b226a765c03bf326d with result from task 2582 parsl.dataflow.dflow DEBUG: Task 2582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/530abfac-689c-44cd-8738-0205cc179bbb_isrForCrosstalkSources_855434_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/530abfac-689c-44cd-8738-0205cc179bbb_isrForCrosstalkSources_855434_4.stderr parsl.dataflow.dflow INFO: Task 2471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3888d5be948d657550b3139963c6fae with result from task 2471 parsl.dataflow.dflow DEBUG: Task 2471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/befe6a48-45cc-4955-a883-c614c72ff450_isrForCrosstalkSources_855401_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/befe6a48-45cc-4955-a883-c614c72ff450_isrForCrosstalkSources_855401_26.stderr parsl.dataflow.dflow INFO: Task 2542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d7b0857b7cde9df03381673757ebb5 with result from task 2542 parsl.dataflow.dflow DEBUG: Task 2542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7443119e-463b-42d6-8d79-642463d6aa8e_isrForCrosstalkSources_855440_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/7443119e-463b-42d6-8d79-642463d6aa8e_isrForCrosstalkSources_855440_14.stderr parsl.dataflow.dflow INFO: Task 2292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acb60f0bb771421cf7bc0ca17892c125 with result from task 2292 parsl.dataflow.dflow DEBUG: Task 2292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5583e85c-de81-4fc2-bcad-28c701b8bcf0_isrForCrosstalkSources_855398_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/5583e85c-de81-4fc2-bcad-28c701b8bcf0_isrForCrosstalkSources_855398_60.stderr parsl.dataflow.dflow INFO: Task 2521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e05343ed6e2b5dfb9b2dd1c78ac1b10 with result from task 2521 parsl.dataflow.dflow DEBUG: Task 2521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8174c320-144d-47e2-bfe6-7e49c3690198_isrForCrosstalkSources_855428_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8174c320-144d-47e2-bfe6-7e49c3690198_isrForCrosstalkSources_855428_9.stderr parsl.dataflow.dflow INFO: Task 2219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9190bec2fffb87c166a25f6e1b957f39 with result from task 2219 parsl.dataflow.dflow DEBUG: Task 2219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d093dd3-3abd-44b4-b2cf-90bfbc49939d_isrForCrosstalkSources_855388_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0d093dd3-3abd-44b4-b2cf-90bfbc49939d_isrForCrosstalkSources_855388_43.stderr parsl.dataflow.dflow INFO: Task 2654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afc4719ec6eeb7fe99cec21744c91c4f with result from task 2654 parsl.dataflow.dflow DEBUG: Task 2654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d8451d6f-4596-45ee-9b38-abcc3dff143f_isrForCrosstalkSources_855428_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d8451d6f-4596-45ee-9b38-abcc3dff143f_isrForCrosstalkSources_855428_41.stderr parsl.dataflow.dflow INFO: Task 2709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d8fd22f55e7120c751e6690a58d02fc with result from task 2709 parsl.dataflow.dflow DEBUG: Task 2709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e730a62d-95d6-4995-b5f9-146160b2f83a_isrForCrosstalkSources_855396_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e730a62d-95d6-4995-b5f9-146160b2f83a_isrForCrosstalkSources_855396_28.stderr parsl.dataflow.dflow INFO: Task 2594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 096c195f3f38ec8081d900978dbd06b1 with result from task 2594 parsl.dataflow.dflow DEBUG: Task 2594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/856f2ec4-b563-49af-b452-92efa8e774f5_isrForCrosstalkSources_855430_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/856f2ec4-b563-49af-b452-92efa8e774f5_isrForCrosstalkSources_855430_8.stderr parsl.dataflow.dflow INFO: Task 2559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c60acd5eb9b49ee56a80f09033929e9 with result from task 2559 parsl.dataflow.dflow DEBUG: Task 2559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6ef4b051-0992-4368-8365-5db6630ba646_isrForCrosstalkSources_855445_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6ef4b051-0992-4368-8365-5db6630ba646_isrForCrosstalkSources_855445_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2692 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry efe168efbf7c1205c32e0bdb9996cd1d with result from task 2692 parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2692 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/96b92f8c-2bec-4eba-bc89-72b7d998ceaa_isrForCrosstalkSources_855447_19.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/96b92f8c-2bec-4eba-bc89-72b7d998ceaa_isrForCrosstalkSources_855447_19.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:53:59 proc_lsst.multi:146 [INFO] found job 16964190 in provider checkpoint proc_lsst.multi INFO: found job 16964190 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd1866de0d46af54577aef2b44c519e with result from task 2455 parsl.dataflow.dflow DEBUG: Task 2455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4ae4d2-b3d8-4426-b4df-f6d73f551d7a_isrForCrosstalkSources_855393_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4ae4d2-b3d8-4426-b4df-f6d73f551d7a_isrForCrosstalkSources_855393_38.stderr parsl.dataflow.dflow INFO: Task 2188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74b20abb8e89ccb2afb9349f11385fac with result from task 2188 parsl.dataflow.dflow DEBUG: Task 2188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a16b9cde-afa6-411f-88c2-b110e61b2f73_isrForCrosstalkSources_855380_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a16b9cde-afa6-411f-88c2-b110e61b2f73_isrForCrosstalkSources_855380_47.stderr parsl.dataflow.dflow INFO: Task 2214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba0522cfd2537064e80dc5ed90213070 with result from task 2214 parsl.dataflow.dflow DEBUG: Task 2214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/65fd9145-3a70-407f-907b-6a10d020b524_isrForCrosstalkSources_855425_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/65fd9145-3a70-407f-907b-6a10d020b524_isrForCrosstalkSources_855425_55.stderr parsl.dataflow.dflow INFO: Task 2321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c262439c90f4700fa87bf9d6b7cf9b80 with result from task 2321 parsl.dataflow.dflow DEBUG: Task 2321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/cabfc420-e525-490f-b444-b9f8788cf314_isrForCrosstalkSources_855439_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/cabfc420-e525-490f-b444-b9f8788cf314_isrForCrosstalkSources_855439_62.stderr parsl.dataflow.dflow INFO: Task 2465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71409be9f5fb20a0ecfdb622a62d3216 with result from task 2465 parsl.dataflow.dflow DEBUG: Task 2465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a76724ee-57e5-48b2-9577-613021527946_isrForCrosstalkSources_855405_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a76724ee-57e5-48b2-9577-613021527946_isrForCrosstalkSources_855405_24.stderr parsl.dataflow.dflow INFO: Task 2512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 104be2bfaa340ac97eee5bf66847bac0 with result from task 2512 parsl.dataflow.dflow DEBUG: Task 2512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1363d343-3add-405d-b5b3-810662c1bd93_isrForCrosstalkSources_855445_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/1363d343-3add-405d-b5b3-810662c1bd93_isrForCrosstalkSources_855445_15.stderr parsl.dataflow.dflow INFO: Task 2380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6436af1fb9ec214ccc642abd93e0d61 with result from task 2380 parsl.dataflow.dflow DEBUG: Task 2380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81c6ce37-a633-45b2-acac-17dc13168f74_isrForCrosstalkSources_855398_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/81c6ce37-a633-45b2-acac-17dc13168f74_isrForCrosstalkSources_855398_34.stderr parsl.dataflow.dflow INFO: Task 2704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eae43dd389791e680676154d6f295f9 with result from task 2704 parsl.dataflow.dflow DEBUG: Task 2704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a61819b-7fc2-44a7-8017-75682d1e1fd0_isrForCrosstalkSources_855396_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a61819b-7fc2-44a7-8017-75682d1e1fd0_isrForCrosstalkSources_855396_3.stderr parsl.dataflow.dflow INFO: Task 2437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a153a34090552587bea325ef680a72 with result from task 2437 parsl.dataflow.dflow DEBUG: Task 2437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8eaa6bf2-45a2-4f26-8920-b42ac3b34507_isrForCrosstalkSources_855402_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8eaa6bf2-45a2-4f26-8920-b42ac3b34507_isrForCrosstalkSources_855402_15.stderr parsl.dataflow.dflow INFO: Task 2693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 097ea7774f20dc7a25698d9dc5969ac1 with result from task 2693 parsl.dataflow.dflow DEBUG: Task 2693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7c57a126-009c-4f8d-9ebb-e1196b20c81c_isrForCrosstalkSources_855457_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/7c57a126-009c-4f8d-9ebb-e1196b20c81c_isrForCrosstalkSources_855457_21.stderr parsl.dataflow.dflow INFO: Task 2446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 375aab645326e1fb143b1afbfe30340e with result from task 2446 parsl.dataflow.dflow DEBUG: Task 2446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d075a05-6c23-40b1-b73a-42a243e23902_isrForCrosstalkSources_855459_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/2d075a05-6c23-40b1-b73a-42a243e23902_isrForCrosstalkSources_855459_49.stderr parsl.dataflow.dflow INFO: Task 2474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9ed632fd6b712051e76e4778920aab with result from task 2474 parsl.dataflow.dflow DEBUG: Task 2474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/91c156dd-9595-4234-b2d0-5ed66dff2d83_isrForCrosstalkSources_855439_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/91c156dd-9595-4234-b2d0-5ed66dff2d83_isrForCrosstalkSources_855439_16.stderr parsl.dataflow.dflow INFO: Task 2395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74fb933df6b23d7b9029fc351aa31219 with result from task 2395 parsl.dataflow.dflow DEBUG: Task 2395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/61d95868-715b-4782-8b5e-e4ac7edd0135_isrForCrosstalkSources_855445_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/61d95868-715b-4782-8b5e-e4ac7edd0135_isrForCrosstalkSources_855445_23.stderr parsl.dataflow.dflow INFO: Task 2573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b9696b311ebef69d132dc17197d3104 with result from task 2573 parsl.dataflow.dflow DEBUG: Task 2573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/adf10067-85b5-499a-8c0f-f684688d7502_isrForCrosstalkSources_855404_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/adf10067-85b5-499a-8c0f-f684688d7502_isrForCrosstalkSources_855404_53.stderr parsl.dataflow.dflow INFO: Task 2374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6fab2ef847498fb9b000e4c59634ae1 with result from task 2374 parsl.dataflow.dflow DEBUG: Task 2374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e801a099-9ff1-4044-b978-26dde178baae_isrForCrosstalkSources_855392_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e801a099-9ff1-4044-b978-26dde178baae_isrForCrosstalkSources_855392_15.stderr parsl.dataflow.dflow INFO: Task 2462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1dcd95cc77bf88876beb939fad31150 with result from task 2462 parsl.dataflow.dflow DEBUG: Task 2462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/70778215-8cae-4b16-9159-ad176360662a_isrForCrosstalkSources_855438_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/70778215-8cae-4b16-9159-ad176360662a_isrForCrosstalkSources_855438_6.stderr parsl.dataflow.dflow INFO: Task 2469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d004663c76018c595f7a4cc8832fb9a with result from task 2469 parsl.dataflow.dflow DEBUG: Task 2469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/9c5be387-9e0c-4997-88ef-7103470f7243_isrForCrosstalkSources_855407_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/9c5be387-9e0c-4997-88ef-7103470f7243_isrForCrosstalkSources_855407_21.stderr parsl.dataflow.dflow INFO: Task 2486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad5db8d5ed38f2c201c9a66fccfe9948 with result from task 2486 parsl.dataflow.dflow DEBUG: Task 2486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a2d6a55f-9069-4b67-801e-3bff01e8e43f_isrForCrosstalkSources_855427_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/a2d6a55f-9069-4b67-801e-3bff01e8e43f_isrForCrosstalkSources_855427_32.stderr parsl.dataflow.dflow INFO: Task 2631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 021d603c6b5f4dad2e5df2390e0c181e with result from task 2631 parsl.dataflow.dflow DEBUG: Task 2631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7c90a5a5-e500-4824-853a-e9862e67820b_isrForCrosstalkSources_855380_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7c90a5a5-e500-4824-853a-e9862e67820b_isrForCrosstalkSources_855380_39.stderr parsl.dataflow.dflow INFO: Task 2457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8903409b077290ea44ce841ac0620e6b with result from task 2457 parsl.dataflow.dflow DEBUG: Task 2457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/6248a671-7ce6-45bd-a3cb-a00cfbeb90d5_isrForCrosstalkSources_855405_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/6248a671-7ce6-45bd-a3cb-a00cfbeb90d5_isrForCrosstalkSources_855405_53.stderr parsl.dataflow.dflow INFO: Task 2627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b87db32c7e4dade102013e43e8a61458 with result from task 2627 parsl.dataflow.dflow DEBUG: Task 2627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/60f93199-c3ea-44ac-92de-63a6f50c1dca_isrForCrosstalkSources_855449_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/60f93199-c3ea-44ac-92de-63a6f50c1dca_isrForCrosstalkSources_855449_17.stderr parsl.dataflow.dflow INFO: Task 2490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c695f9b1bac22d9f9c138afb924ba18c with result from task 2490 parsl.dataflow.dflow DEBUG: Task 2490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/7f6eef0f-2c95-4cac-a911-0f14a1e35a35_isrForCrosstalkSources_855417_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/7f6eef0f-2c95-4cac-a911-0f14a1e35a35_isrForCrosstalkSources_855417_36.stderr parsl.dataflow.dflow INFO: Task 2626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76d12faf5641494c47560515c26f2dee with result from task 2626 parsl.dataflow.dflow DEBUG: Task 2626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6c92fba4-cb25-4f53-8b18-459a3b9d488c_isrForCrosstalkSources_855401_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6c92fba4-cb25-4f53-8b18-459a3b9d488c_isrForCrosstalkSources_855401_1.stderr parsl.dataflow.dflow INFO: Task 2638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe6c1ddd6a086ca26c3cee708fd49b9d with result from task 2638 parsl.dataflow.dflow DEBUG: Task 2638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/781f06e1-f676-451c-965b-7782a3b69a77_isrForCrosstalkSources_855458_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/781f06e1-f676-451c-965b-7782a3b69a77_isrForCrosstalkSources_855458_8.stderr parsl.dataflow.dflow INFO: Task 2583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6967effb988d2ea2dfca367918a1b860 with result from task 2583 parsl.dataflow.dflow DEBUG: Task 2583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f9c5d21a-e1fa-4212-9f3b-af41b5edb171_isrForCrosstalkSources_855452_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f9c5d21a-e1fa-4212-9f3b-af41b5edb171_isrForCrosstalkSources_855452_61.stderr parsl.dataflow.dflow INFO: Task 2677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 996d66aab58fbd5b8cd445ace6558990 with result from task 2677 parsl.dataflow.dflow DEBUG: Task 2677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2d818d0f-890a-447f-a2ff-fa732ceba02f_isrForCrosstalkSources_855452_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/2d818d0f-890a-447f-a2ff-fa732ceba02f_isrForCrosstalkSources_855452_52.stderr parsl.dataflow.dflow INFO: Task 2728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c48c800abf10709c7fe16de9a2997c with result from task 2728 parsl.dataflow.dflow DEBUG: Task 2728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/95a500eb-90da-489a-9c5e-ba71613e808d_isrForCrosstalkSources_855420_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/95a500eb-90da-489a-9c5e-ba71613e808d_isrForCrosstalkSources_855420_9.stderr parsl.dataflow.dflow INFO: Task 2401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00985165a459e06a5c5e477f8c4a1bcc with result from task 2401 parsl.dataflow.dflow DEBUG: Task 2401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6dad5528-a4df-4bd6-ba27-2ba2a6955eb1_isrForCrosstalkSources_855456_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6dad5528-a4df-4bd6-ba27-2ba2a6955eb1_isrForCrosstalkSources_855456_54.stderr parsl.dataflow.dflow INFO: Task 2444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3763a49c1bab56269c98cf706b33797 with result from task 2444 parsl.dataflow.dflow DEBUG: Task 2444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e2e1bac-1503-465c-b58d-c3ab0e660b81_isrForCrosstalkSources_855438_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e2e1bac-1503-465c-b58d-c3ab0e660b81_isrForCrosstalkSources_855438_3.stderr parsl.dataflow.dflow INFO: Task 2443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 611cfc2556cb3793e5e80af0fd2fe815 with result from task 2443 parsl.dataflow.dflow DEBUG: Task 2443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c79315c3-7485-4bd5-aeb4-0e6ca8a5dfa0_isrForCrosstalkSources_855457_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c79315c3-7485-4bd5-aeb4-0e6ca8a5dfa0_isrForCrosstalkSources_855457_34.stderr parsl.dataflow.dflow INFO: Task 2453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1df770d16f8e1b5b557e9c959350ba2 with result from task 2453 parsl.dataflow.dflow DEBUG: Task 2453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b5828a9-0d70-4314-ad78-9fbbe78ea600_isrForCrosstalkSources_855416_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5b5828a9-0d70-4314-ad78-9fbbe78ea600_isrForCrosstalkSources_855416_13.stderr parsl.dataflow.dflow INFO: Task 2563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 401977b09db2c1b7d34ab483ab59d049 with result from task 2563 parsl.dataflow.dflow DEBUG: Task 2563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a29424c1-76ff-4dc9-aec1-9eccfb0a5347_isrForCrosstalkSources_855404_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a29424c1-76ff-4dc9-aec1-9eccfb0a5347_isrForCrosstalkSources_855404_59.stderr parsl.dataflow.dflow INFO: Task 2311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c75b2818cca8e3e348463323d954df70 with result from task 2311 parsl.dataflow.dflow DEBUG: Task 2311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/034712d0-9eb0-4d20-9a98-0f638d04526e_isrForCrosstalkSources_855415_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/034712d0-9eb0-4d20-9a98-0f638d04526e_isrForCrosstalkSources_855415_38.stderr parsl.dataflow.dflow INFO: Task 2271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f87db6fbdc94f01fad78f203c8498dd with result from task 2271 parsl.dataflow.dflow DEBUG: Task 2271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/68e90117-fc46-4599-978e-8d7aec200db6_isrForCrosstalkSources_855430_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/68e90117-fc46-4599-978e-8d7aec200db6_isrForCrosstalkSources_855430_58.stderr parsl.dataflow.dflow INFO: Task 2517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085cff9b5d5053df1b6786ea0b59355c with result from task 2517 parsl.dataflow.dflow DEBUG: Task 2517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/35e1ae4c-1d11-4de9-a669-f55b5775d14d_isrForCrosstalkSources_855449_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/35e1ae4c-1d11-4de9-a669-f55b5775d14d_isrForCrosstalkSources_855449_39.stderr parsl.dataflow.dflow INFO: Task 2598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ed65a1f7ccf9a0a4464792e44aaf2a7 with result from task 2598 parsl.dataflow.dflow DEBUG: Task 2598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/63c36c85-7fdc-4b45-9b3a-b1822b62e865_isrForCrosstalkSources_855449_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/63c36c85-7fdc-4b45-9b3a-b1822b62e865_isrForCrosstalkSources_855449_10.stderr parsl.dataflow.dflow INFO: Task 2507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e921665ed0b7daa8b7f7310652a681 with result from task 2507 parsl.dataflow.dflow DEBUG: Task 2507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ca13b301-1b74-4412-ae1e-197453e79fa8_isrForCrosstalkSources_855447_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/ca13b301-1b74-4412-ae1e-197453e79fa8_isrForCrosstalkSources_855447_49.stderr parsl.dataflow.dflow INFO: Task 2580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1626a84d3226c72a7ec4af3c00cd13dc with result from task 2580 parsl.dataflow.dflow DEBUG: Task 2580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/65553bf8-cdb1-4b5f-884b-a30c40a8e71d_isrForCrosstalkSources_855400_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/65553bf8-cdb1-4b5f-884b-a30c40a8e71d_isrForCrosstalkSources_855400_17.stderr parsl.dataflow.dflow INFO: Task 2576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3722847c06d7007089baf3683fa27905 with result from task 2576 parsl.dataflow.dflow DEBUG: Task 2576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9f804cac-f320-4a04-8e5a-a606c05271c1_isrForCrosstalkSources_855394_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9f804cac-f320-4a04-8e5a-a606c05271c1_isrForCrosstalkSources_855394_2.stderr parsl.dataflow.dflow INFO: Task 2441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d78324891d4bc544d9c08039d2786f3f with result from task 2441 parsl.dataflow.dflow DEBUG: Task 2441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a28aec16-9dfb-43c5-9f08-108f0b42a020_isrForCrosstalkSources_855380_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a28aec16-9dfb-43c5-9f08-108f0b42a020_isrForCrosstalkSources_855380_53.stderr parsl.dataflow.dflow INFO: Task 2527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b29bfa713899a935a4b0ce8f724ab58f with result from task 2527 parsl.dataflow.dflow DEBUG: Task 2527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c6804db7-8254-4fdc-ac96-b56bdff6c1db_isrForCrosstalkSources_855448_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/c6804db7-8254-4fdc-ac96-b56bdff6c1db_isrForCrosstalkSources_855448_27.stderr parsl.dataflow.dflow INFO: Task 2489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d19c0db96330c6a31788e0c971aa48dc with result from task 2489 parsl.dataflow.dflow DEBUG: Task 2489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/26dfdd45-d715-4bef-88f7-4130d9bae02f_isrForCrosstalkSources_855383_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/26dfdd45-d715-4bef-88f7-4130d9bae02f_isrForCrosstalkSources_855383_16.stderr parsl.dataflow.dflow INFO: Task 2651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d3f54fbf1ec7720ed73a8c07ed4558d with result from task 2651 parsl.dataflow.dflow DEBUG: Task 2651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/df2f36ac-2e31-4bd6-937c-849cb61de4ef_isrForCrosstalkSources_855457_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/df2f36ac-2e31-4bd6-937c-849cb61de4ef_isrForCrosstalkSources_855457_40.stderr parsl.dataflow.dflow INFO: Task 2449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 703a82e3505d47e42bc093c6af17c028 with result from task 2449 parsl.dataflow.dflow DEBUG: Task 2449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/55da655a-a87c-487a-9675-979f97a07ab4_isrForCrosstalkSources_855438_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/55da655a-a87c-487a-9675-979f97a07ab4_isrForCrosstalkSources_855438_15.stderr parsl.dataflow.dflow INFO: Task 2614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de083c144b22f783acd0d33e6c5a32dd with result from task 2614 parsl.dataflow.dflow DEBUG: Task 2614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/626e8b04-dab2-43c3-8337-61a1bac424cb_isrForCrosstalkSources_855390_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/626e8b04-dab2-43c3-8337-61a1bac424cb_isrForCrosstalkSources_855390_5.stderr parsl.dataflow.dflow INFO: Task 2597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1c36ad87194cbc9f4d2d3a4e0f40b1d with result from task 2597 parsl.dataflow.dflow DEBUG: Task 2597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0a520dd5-ed2e-4f49-ac7c-53f30c14485a_isrForCrosstalkSources_855419_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0a520dd5-ed2e-4f49-ac7c-53f30c14485a_isrForCrosstalkSources_855419_57.stderr parsl.dataflow.dflow INFO: Task 2458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af6ff7320dd2c3836c3feeaae3322fd9 with result from task 2458 parsl.dataflow.dflow DEBUG: Task 2458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d17c8be4-bc84-419d-a641-c0efa38131ba_isrForCrosstalkSources_855397_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d17c8be4-bc84-419d-a641-c0efa38131ba_isrForCrosstalkSources_855397_9.stderr parsl.dataflow.dflow INFO: Task 2492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18d09e7f395e54f8c0da165e2f08a2e9 with result from task 2492 parsl.dataflow.dflow DEBUG: Task 2492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a22df2fd-1c69-4db8-bca2-d596a0d151e5_isrForCrosstalkSources_855445_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/a22df2fd-1c69-4db8-bca2-d596a0d151e5_isrForCrosstalkSources_855445_34.stderr parsl.dataflow.dflow INFO: Task 2600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8208c4534cb88fa67441d6f5910f9b09 with result from task 2600 parsl.dataflow.dflow DEBUG: Task 2600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/4346d26c-f654-492b-9569-5c48bfdbf942_isrForCrosstalkSources_855428_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/4346d26c-f654-492b-9569-5c48bfdbf942_isrForCrosstalkSources_855428_56.stderr parsl.dataflow.dflow INFO: Task 2606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2c3441bec9ddc612fb9c78df441806 with result from task 2606 parsl.dataflow.dflow DEBUG: Task 2606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/885242df-7aed-4a53-9d3e-f8a0ce72701d_isrForCrosstalkSources_855445_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/885242df-7aed-4a53-9d3e-f8a0ce72701d_isrForCrosstalkSources_855445_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:00 proc_lsst.multi:146 [INFO] found job 16964191 in provider checkpoint proc_lsst.multi INFO: found job 16964191 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7c06c0b55d8c169b29ee44dac64a814 with result from task 2647 parsl.dataflow.dflow DEBUG: Task 2647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b05b4eae-518b-4370-bc8b-06bd666fa819_isrForCrosstalkSources_855414_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b05b4eae-518b-4370-bc8b-06bd666fa819_isrForCrosstalkSources_855414_3.stderr parsl.dataflow.dflow INFO: Task 2617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 498e86be4099d6cf7257cff312a8f190 with result from task 2617 parsl.dataflow.dflow DEBUG: Task 2617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/144221a3-6af4-4e44-8ca2-66d863ab50e6_isrForCrosstalkSources_855432_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/144221a3-6af4-4e44-8ca2-66d863ab50e6_isrForCrosstalkSources_855432_2.stderr parsl.dataflow.dflow INFO: Task 2532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f68934e8cba640d2454cd048409e005d with result from task 2532 parsl.dataflow.dflow DEBUG: Task 2532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/5c841d25-0bbc-457f-976a-a5f8619794fb_isrForCrosstalkSources_855414_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/5c841d25-0bbc-457f-976a-a5f8619794fb_isrForCrosstalkSources_855414_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:00 proc_lsst.multi:146 [INFO] found job 16964192 in provider checkpoint proc_lsst.multi INFO: found job 16964192 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 387835c4b44159233e794f575675e7ac with result from task 2611 parsl.dataflow.dflow DEBUG: Task 2611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/873e0479-f344-403c-8375-3be2c1fdc0e9_isrForCrosstalkSources_855439_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/873e0479-f344-403c-8375-3be2c1fdc0e9_isrForCrosstalkSources_855439_37.stderr parsl.dataflow.dflow INFO: Task 2679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7b7508214b98e7c597ebdb05fe65e7 with result from task 2679 parsl.dataflow.dflow DEBUG: Task 2679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b1cb9fa3-b826-4dfb-814d-1abf50875437_isrForCrosstalkSources_855436_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b1cb9fa3-b826-4dfb-814d-1abf50875437_isrForCrosstalkSources_855436_59.stderr parsl.dataflow.dflow INFO: Task 2717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71bf552a2d0edb565e42be25344ca257 with result from task 2717 parsl.dataflow.dflow DEBUG: Task 2717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f74e9ded-9300-493a-9143-bb8b1d41fb2b_isrForCrosstalkSources_855458_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f74e9ded-9300-493a-9143-bb8b1d41fb2b_isrForCrosstalkSources_855458_38.stderr parsl.dataflow.dflow INFO: Task 2735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 271ba2d0a8d63fa71fe389eca52d47f5 with result from task 2735 parsl.dataflow.dflow DEBUG: Task 2735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1fc0a02b-cf46-404e-9968-b23c321ec330_isrForCrosstalkSources_855395_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/1fc0a02b-cf46-404e-9968-b23c321ec330_isrForCrosstalkSources_855395_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:00 proc_lsst.multi:146 [INFO] found job 16964193 in provider checkpoint proc_lsst.multi INFO: found job 16964193 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da27ea7db72d414cac76e9d24a2d06ad with result from task 2554 parsl.dataflow.dflow DEBUG: Task 2554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/484b8a4d-03ff-43f6-957e-89e7f8ae21c7_isrForCrosstalkSources_855447_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/484b8a4d-03ff-43f6-957e-89e7f8ae21c7_isrForCrosstalkSources_855447_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:01 proc_lsst.multi:146 [INFO] found job 16964194 in provider checkpoint proc_lsst.multi INFO: found job 16964194 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:01 proc_lsst.multi:146 [INFO] found job 16964195 in provider checkpoint proc_lsst.multi INFO: found job 16964195 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ecac04293d33b044799f72306a3a701 with result from task 2686 parsl.dataflow.dflow DEBUG: Task 2686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1fc3cca6-46e0-4310-82a6-f2ddc955abf2_isrForCrosstalkSources_855383_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/1fc3cca6-46e0-4310-82a6-f2ddc955abf2_isrForCrosstalkSources_855383_49.stderr parsl.dataflow.dflow INFO: Task 2689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39f68706e6c7331654a94daacaa005a with result from task 2689 parsl.dataflow.dflow DEBUG: Task 2689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/c5bd018c-0d80-452d-b2e0-8385c7cb9af2_isrForCrosstalkSources_855437_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/c5bd018c-0d80-452d-b2e0-8385c7cb9af2_isrForCrosstalkSources_855437_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:01 proc_lsst.multi:146 [INFO] found job 16964196 in provider checkpoint proc_lsst.multi INFO: found job 16964196 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f832c4f386bf04643f300eef813f2f7b with result from task 2641 parsl.dataflow.dflow DEBUG: Task 2641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/07db19af-e567-44f4-afff-712475211bf2_isrForCrosstalkSources_855454_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/07db19af-e567-44f4-afff-712475211bf2_isrForCrosstalkSources_855454_28.stderr parsl.dataflow.dflow INFO: Task 2642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83121c5dd6e5b06cc9ce2e35762f6f61 with result from task 2642 parsl.dataflow.dflow DEBUG: Task 2642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4dc85625-7a34-49b9-a6fd-107c196775bd_isrForCrosstalkSources_855430_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4dc85625-7a34-49b9-a6fd-107c196775bd_isrForCrosstalkSources_855430_4.stderr parsl.dataflow.dflow INFO: Task 2550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07d1cca0fe1596a5205d6738539d749f with result from task 2550 parsl.dataflow.dflow DEBUG: Task 2550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed2ebf13-8605-4a6a-8010-f9b1b9396dee_isrForCrosstalkSources_855453_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/ed2ebf13-8605-4a6a-8010-f9b1b9396dee_isrForCrosstalkSources_855453_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:01 proc_lsst.multi:146 [INFO] found job 16964197 in provider checkpoint proc_lsst.multi INFO: found job 16964197 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 093c19baf4ad29ab5827465c11e6c434 with result from task 2740 parsl.dataflow.dflow DEBUG: Task 2740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/42e2c45e-3ae2-4559-b090-31f306d91ae4_isrForCrosstalkSources_855425_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/42e2c45e-3ae2-4559-b090-31f306d91ae4_isrForCrosstalkSources_855425_32.stderr parsl.dataflow.dflow INFO: Task 2773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0fd8b826fa3b54be6e998cb80b95b43 with result from task 2773 parsl.dataflow.dflow DEBUG: Task 2773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cb18bdc9-bb6d-49da-b71b-cbce772577b3_isrForCrosstalkSources_855382_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cb18bdc9-bb6d-49da-b71b-cbce772577b3_isrForCrosstalkSources_855382_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:02 proc_lsst.multi:146 [INFO] found job 16964198 in provider checkpoint proc_lsst.multi INFO: found job 16964198 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0a29c778ca9e8f10b29a9373fcf5252 with result from task 2653 parsl.dataflow.dflow DEBUG: Task 2653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d96ed436-c274-429a-b949-3c48da0d2ed9_isrForCrosstalkSources_855421_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/d96ed436-c274-429a-b949-3c48da0d2ed9_isrForCrosstalkSources_855421_50.stderr parsl.dataflow.dflow INFO: Task 2783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e6200bd99d664d9b820ad7d6e5b7ff6 with result from task 2783 parsl.dataflow.dflow DEBUG: Task 2783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24c2bb7c-1931-4096-ae6b-0ddf818dc22b_isrForCrosstalkSources_855444_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/24c2bb7c-1931-4096-ae6b-0ddf818dc22b_isrForCrosstalkSources_855444_11.stderr parsl.dataflow.dflow INFO: Task 2566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c22ef2f822e5749b6783d49606400f9 with result from task 2566 parsl.dataflow.dflow DEBUG: Task 2566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ad5ef9c-8b2b-435a-9eac-ff8ea0e9a6e2_isrForCrosstalkSources_855418_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ad5ef9c-8b2b-435a-9eac-ff8ea0e9a6e2_isrForCrosstalkSources_855418_62.stderr parsl.dataflow.dflow INFO: Task 2794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86dcd28e7ff137035d7daf1032c27f7a with result from task 2794 parsl.dataflow.dflow DEBUG: Task 2794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/903add41-4790-4d6c-8f80-b31fbe6b5bb3_isrForCrosstalkSources_855431_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/903add41-4790-4d6c-8f80-b31fbe6b5bb3_isrForCrosstalkSources_855431_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:02 proc_lsst.multi:146 [INFO] found job 16964199 in provider checkpoint proc_lsst.multi INFO: found job 16964199 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f92637281284fea7f86fdbd361fddb08 with result from task 2733 parsl.dataflow.dflow DEBUG: Task 2733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a40d537b-3764-44f7-a85d-6461dece9db3_isrForCrosstalkSources_855386_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a40d537b-3764-44f7-a85d-6461dece9db3_isrForCrosstalkSources_855386_56.stderr parsl.dataflow.dflow INFO: Task 2623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 724d861ea71e2c4b3b5161d8f332df73 with result from task 2623 parsl.dataflow.dflow DEBUG: Task 2623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4c5223a6-60fa-4711-988a-570863e10a89_isrForCrosstalkSources_855415_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4c5223a6-60fa-4711-988a-570863e10a89_isrForCrosstalkSources_855415_21.stderr parsl.dataflow.dflow INFO: Task 2810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d16e6a95bf415f9aa843513f877d223c with result from task 2810 parsl.dataflow.dflow DEBUG: Task 2810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54de433c-1777-4f66-8007-2c667fcc4c50_isrForCrosstalkSources_855382_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/54de433c-1777-4f66-8007-2c667fcc4c50_isrForCrosstalkSources_855382_33.stderr parsl.dataflow.dflow INFO: Task 2824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e43fcb2a439d80d078bddd4bcfb036da with result from task 2824 parsl.dataflow.dflow DEBUG: Task 2824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ed6cc0ef-552f-4828-98fc-fd9c5f800667_isrForCrosstalkSources_855415_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/ed6cc0ef-552f-4828-98fc-fd9c5f800667_isrForCrosstalkSources_855415_45.stderr parsl.dataflow.dflow INFO: Task 2710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d08950c486653cb6bb9d7886f1808c93 with result from task 2710 parsl.dataflow.dflow DEBUG: Task 2710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c0e3d1e3-57dc-4a2f-908b-8b51efdaf377_isrForCrosstalkSources_855400_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c0e3d1e3-57dc-4a2f-908b-8b51efdaf377_isrForCrosstalkSources_855400_18.stderr parsl.dataflow.dflow INFO: Task 2788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d91bcd32b1daadd0f1f8276a1042f80c with result from task 2788 parsl.dataflow.dflow DEBUG: Task 2788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5d22c461-cee8-4ae6-874c-2948b0ceefc1_isrForCrosstalkSources_855433_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5d22c461-cee8-4ae6-874c-2948b0ceefc1_isrForCrosstalkSources_855433_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:03 proc_lsst.multi:146 [INFO] found job 16964200 in provider checkpoint proc_lsst.multi INFO: found job 16964200 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81cd4befef90416aa876e42a06cca8dd with result from task 2786 parsl.dataflow.dflow DEBUG: Task 2786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b8d879de-f1e5-4e38-a58a-38de10bad514_isrForCrosstalkSources_855389_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b8d879de-f1e5-4e38-a58a-38de10bad514_isrForCrosstalkSources_855389_27.stderr parsl.dataflow.dflow INFO: Task 2784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1db0d841389af03fe5ea136b4a942c0c with result from task 2784 parsl.dataflow.dflow DEBUG: Task 2784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e78ea31-b1bd-4e21-9efa-2a46a6ec55e7_isrForCrosstalkSources_855412_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0e78ea31-b1bd-4e21-9efa-2a46a6ec55e7_isrForCrosstalkSources_855412_18.stderr parsl.dataflow.dflow INFO: Task 2561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966b6802c6d231f25fa8a01253447c73 with result from task 2561 parsl.dataflow.dflow DEBUG: Task 2561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a622c46e-2f4b-4240-b3b4-2f2e9e052f82_isrForCrosstalkSources_855381_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a622c46e-2f4b-4240-b3b4-2f2e9e052f82_isrForCrosstalkSources_855381_61.stderr parsl.dataflow.dflow INFO: Task 2798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62af3486c32a4ae1c724be283b4518d8 with result from task 2798 parsl.dataflow.dflow DEBUG: Task 2798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/fb268a1a-9711-4d5e-bb2c-fe5002c2dc27_isrForCrosstalkSources_855426_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/fb268a1a-9711-4d5e-bb2c-fe5002c2dc27_isrForCrosstalkSources_855426_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:03 proc_lsst.multi:146 [INFO] found job 16964201 in provider checkpoint proc_lsst.multi INFO: found job 16964201 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640dd16bd6c6289bd335f0d846e590b5 with result from task 2806 parsl.dataflow.dflow DEBUG: Task 2806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a07f0796-bc6d-4662-8b67-1a7c795c144e_isrForCrosstalkSources_855450_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a07f0796-bc6d-4662-8b67-1a7c795c144e_isrForCrosstalkSources_855450_18.stderr parsl.dataflow.dflow INFO: Task 2762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f4132900b638b010c2e18e51826877b with result from task 2762 parsl.dataflow.dflow DEBUG: Task 2762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4a5b9e27-e098-42f0-853f-f808407caedf_isrForCrosstalkSources_855411_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4a5b9e27-e098-42f0-853f-f808407caedf_isrForCrosstalkSources_855411_10.stderr parsl.dataflow.dflow INFO: Task 2700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b914f615082f36175f1b4968fe842eb9 with result from task 2700 parsl.dataflow.dflow DEBUG: Task 2700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67d94c7a-ae78-45cc-96e6-c8574b419661_isrForCrosstalkSources_855393_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67d94c7a-ae78-45cc-96e6-c8574b419661_isrForCrosstalkSources_855393_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:04 proc_lsst.multi:146 [INFO] found job 16964202 in provider checkpoint proc_lsst.multi INFO: found job 16964202 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2af95c8399fc72c71fd9f5b36e97b9ea with result from task 2827 parsl.dataflow.dflow DEBUG: Task 2827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d0d900a8-0177-40fa-a06e-36310ad93fe5_isrForCrosstalkSources_855422_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d0d900a8-0177-40fa-a06e-36310ad93fe5_isrForCrosstalkSources_855422_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:04 proc_lsst.multi:146 [INFO] found job 16964203 in provider checkpoint proc_lsst.multi INFO: found job 16964203 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ec1903c69c55e0f42c6843240934d73 with result from task 2546 parsl.dataflow.dflow DEBUG: Task 2546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0040b903-8e4d-401e-bded-d12c0042c078_isrForCrosstalkSources_855412_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0040b903-8e4d-401e-bded-d12c0042c078_isrForCrosstalkSources_855412_34.stderr parsl.dataflow.dflow INFO: Task 2863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ea269b2ba72c9481f734e4ac66a5d8 with result from task 2863 parsl.dataflow.dflow DEBUG: Task 2863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ae236b45-ea49-4d79-94ba-2018c602dcd5_isrForCrosstalkSources_855406_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ae236b45-ea49-4d79-94ba-2018c602dcd5_isrForCrosstalkSources_855406_11.stderr parsl.dataflow.dflow INFO: Task 2625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b72fd69f019dcd96571477daf8c162a with result from task 2625 parsl.dataflow.dflow DEBUG: Task 2625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a7990f8a-1fb0-44e7-9059-b5a2db7c9bd0_isrForCrosstalkSources_855393_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a7990f8a-1fb0-44e7-9059-b5a2db7c9bd0_isrForCrosstalkSources_855393_43.stderr parsl.dataflow.dflow INFO: Task 2420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7ef4ae927bc1adb244d166fc65bcb85 with result from task 2420 parsl.dataflow.dflow DEBUG: Task 2420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d244876a-b9c8-4fc6-90a5-2319e9043f83_isrForCrosstalkSources_855443_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d244876a-b9c8-4fc6-90a5-2319e9043f83_isrForCrosstalkSources_855443_32.stderr parsl.dataflow.dflow INFO: Task 2756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1afd13ecf3f34a97a2e877d6756348b0 with result from task 2756 parsl.dataflow.dflow DEBUG: Task 2756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eb70a809-4ee0-4c51-9ac2-51257e766367_isrForCrosstalkSources_855396_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/eb70a809-4ee0-4c51-9ac2-51257e766367_isrForCrosstalkSources_855396_15.stderr parsl.dataflow.dflow INFO: Task 2760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faf4645d2fdafc41cb56598a98c41810 with result from task 2760 parsl.dataflow.dflow DEBUG: Task 2760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f80e0e83-6e58-4101-9258-309f0a7cbe25_isrForCrosstalkSources_855395_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f80e0e83-6e58-4101-9258-309f0a7cbe25_isrForCrosstalkSources_855395_22.stderr parsl.dataflow.dflow INFO: Task 2757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab07132edd8a8ba24cb07f792217bc5 with result from task 2757 parsl.dataflow.dflow DEBUG: Task 2757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ed6c90e-73af-4837-83f2-0919ba93046f_isrForCrosstalkSources_855401_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/5ed6c90e-73af-4837-83f2-0919ba93046f_isrForCrosstalkSources_855401_47.stderr parsl.dataflow.dflow INFO: Task 2831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e19099225bfb83ea508df39e4a6b3af with result from task 2831 parsl.dataflow.dflow DEBUG: Task 2831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d8dd7792-3120-402d-8c11-ff8fc7e44661_isrForCrosstalkSources_855385_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d8dd7792-3120-402d-8c11-ff8fc7e44661_isrForCrosstalkSources_855385_56.stderr parsl.dataflow.dflow INFO: Task 2725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dbc927d534fa28c752e2a8636ea0ffa with result from task 2725 parsl.dataflow.dflow DEBUG: Task 2725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00ef9d7c-d192-4b10-b2a3-b7376dec6cee_isrForCrosstalkSources_855390_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00ef9d7c-d192-4b10-b2a3-b7376dec6cee_isrForCrosstalkSources_855390_7.stderr parsl.dataflow.dflow INFO: Task 2809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e30b6dbbdc3fbc85d3b9c221e93da43 with result from task 2809 parsl.dataflow.dflow DEBUG: Task 2809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f932ec1a-1da7-4563-b7d7-148789509c3b_isrForCrosstalkSources_855426_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f932ec1a-1da7-4563-b7d7-148789509c3b_isrForCrosstalkSources_855426_15.stderr parsl.dataflow.dflow INFO: Task 2440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22e1cc2fca304e2bb8c10d87837bdffc with result from task 2440 parsl.dataflow.dflow DEBUG: Task 2440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1add1895-6d07-44cc-bea7-cdb5d46d1804_isrForCrosstalkSources_855409_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1add1895-6d07-44cc-bea7-cdb5d46d1804_isrForCrosstalkSources_855409_27.stderr parsl.dataflow.dflow INFO: Task 2785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 820b54d098cd4f7b5d685ad0ed3d7791 with result from task 2785 parsl.dataflow.dflow DEBUG: Task 2785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/923d5cfd-e281-4e40-87db-01cb4f7fd4d0_isrForCrosstalkSources_855417_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/923d5cfd-e281-4e40-87db-01cb4f7fd4d0_isrForCrosstalkSources_855417_39.stderr parsl.dataflow.dflow INFO: Task 2454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eceaa265d5e5c07ad9c8523eac85b9ef with result from task 2454 parsl.dataflow.dflow DEBUG: Task 2454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/d6ca3a75-3d6d-4b9c-8c3f-22db40d261ab_isrForCrosstalkSources_855427_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/d6ca3a75-3d6d-4b9c-8c3f-22db40d261ab_isrForCrosstalkSources_855427_57.stderr parsl.dataflow.dflow INFO: Task 2628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7a1b290c3466c21eb850e273cbe5b76 with result from task 2628 parsl.dataflow.dflow DEBUG: Task 2628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6b0dfa55-96ad-46ef-a513-38c882a20bb2_isrForCrosstalkSources_855431_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6b0dfa55-96ad-46ef-a513-38c882a20bb2_isrForCrosstalkSources_855431_56.stderr parsl.dataflow.dflow INFO: Task 2732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17438a9d0bd3f8a620af573116a5d1fc with result from task 2732 parsl.dataflow.dflow DEBUG: Task 2732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/4a8388b3-56ef-4971-85b6-7b3e979379ec_isrForCrosstalkSources_855458_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/4a8388b3-56ef-4971-85b6-7b3e979379ec_isrForCrosstalkSources_855458_14.stderr parsl.dataflow.dflow INFO: Task 2775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61dc9a6bc64816fc05802ab7c9cc1b3e with result from task 2775 parsl.dataflow.dflow DEBUG: Task 2775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/192380fe-a315-450f-a0dd-fe5465e69e42_isrForCrosstalkSources_855416_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/192380fe-a315-450f-a0dd-fe5465e69e42_isrForCrosstalkSources_855416_23.stderr parsl.dataflow.dflow INFO: Task 2691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b048e5a29f0183725335c1b36e184fe2 with result from task 2691 parsl.dataflow.dflow DEBUG: Task 2691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3d812fe9-28ea-45ff-a502-898cbbf59496_isrForCrosstalkSources_855397_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3d812fe9-28ea-45ff-a502-898cbbf59496_isrForCrosstalkSources_855397_41.stderr parsl.dataflow.dflow INFO: Task 2807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be477d5e64cea48e8f1d78a2fd23e3c8 with result from task 2807 parsl.dataflow.dflow DEBUG: Task 2807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/20674411-c058-4567-8c53-a4ea33ad1c50_isrForCrosstalkSources_855416_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/20674411-c058-4567-8c53-a4ea33ad1c50_isrForCrosstalkSources_855416_22.stderr parsl.dataflow.dflow INFO: Task 2822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af545b58de9a0072897080e3df0b5720 with result from task 2822 parsl.dataflow.dflow DEBUG: Task 2822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/960c354f-80c8-4f13-b3c0-3160f3f24f1b_isrForCrosstalkSources_855418_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/960c354f-80c8-4f13-b3c0-3160f3f24f1b_isrForCrosstalkSources_855418_39.stderr parsl.dataflow.dflow INFO: Task 2533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11baad6d41eb53005d5bd6492fc6da25 with result from task 2533 parsl.dataflow.dflow DEBUG: Task 2533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/cb669df7-0e85-475e-923a-3b9d4ecf05eb_isrForCrosstalkSources_855431_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/cb669df7-0e85-475e-923a-3b9d4ecf05eb_isrForCrosstalkSources_855431_2.stderr parsl.dataflow.dflow INFO: Task 2487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0378385c8a14977fa3134b94dea1ad7b with result from task 2487 parsl.dataflow.dflow DEBUG: Task 2487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/be97d3db-1c07-47c5-bc68-686b8768814f_isrForCrosstalkSources_855445_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/be97d3db-1c07-47c5-bc68-686b8768814f_isrForCrosstalkSources_855445_46.stderr parsl.dataflow.dflow INFO: Task 2706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 583acab1dbb628f663b5bb7cdb6df788 with result from task 2706 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:05 proc_lsst.multi:146 [INFO] found job 16964204 in provider checkpoint proc_lsst.multi INFO: found job 16964204 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cba82d1-d5dd-4b92-8841-78e3e6afe878_isrForCrosstalkSources_855403_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/3cba82d1-d5dd-4b92-8841-78e3e6afe878_isrForCrosstalkSources_855403_27.stderr parsl.dataflow.dflow INFO: Task 2494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fe620ceb8ae78f1f6a1af2832aead76 with result from task 2494 parsl.dataflow.dflow DEBUG: Task 2494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9e48b560-a799-4391-9e24-592141702347_isrForCrosstalkSources_855389_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9e48b560-a799-4391-9e24-592141702347_isrForCrosstalkSources_855389_36.stderr parsl.dataflow.dflow INFO: Task 2678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeb2730c159343b52a020ea9b1122066 with result from task 2678 parsl.dataflow.dflow DEBUG: Task 2678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c74a43a6-8c6e-4a26-9129-cb5ebc6af2d5_isrForCrosstalkSources_855427_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c74a43a6-8c6e-4a26-9129-cb5ebc6af2d5_isrForCrosstalkSources_855427_44.stderr parsl.dataflow.dflow INFO: Task 2442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88f76d9036e3c8505066713d4de5bb09 with result from task 2442 parsl.dataflow.dflow DEBUG: Task 2442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/4b66af39-a96e-4bf8-9b52-34f5f2ad5ea3_isrForCrosstalkSources_855416_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/4b66af39-a96e-4bf8-9b52-34f5f2ad5ea3_isrForCrosstalkSources_855416_55.stderr parsl.dataflow.dflow INFO: Task 2823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5edd9d83973567bc843b06833ae38506 with result from task 2823 parsl.dataflow.dflow DEBUG: Task 2823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/349614ff-a290-4379-8cbc-f8d4f49fac06_isrForCrosstalkSources_855447_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/349614ff-a290-4379-8cbc-f8d4f49fac06_isrForCrosstalkSources_855447_51.stderr parsl.dataflow.dflow INFO: Task 2640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b90bbe9b554377120fbfc461d5f8dcd5 with result from task 2640 parsl.dataflow.dflow DEBUG: Task 2640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3d852e45-d425-4058-b522-808e288040b9_isrForCrosstalkSources_855405_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3d852e45-d425-4058-b522-808e288040b9_isrForCrosstalkSources_855405_31.stderr parsl.dataflow.dflow INFO: Task 2371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 930ef0f5f5d1745f15de5bbd75f02901 with result from task 2371 parsl.dataflow.dflow DEBUG: Task 2371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81900804-cb1c-447a-842e-617db47098fd_isrForCrosstalkSources_855443_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/81900804-cb1c-447a-842e-617db47098fd_isrForCrosstalkSources_855443_48.stderr parsl.dataflow.dflow INFO: Task 2811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06684c92afd1092152cdfa26666c51f7 with result from task 2811 parsl.dataflow.dflow DEBUG: Task 2811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a840a83d-29a9-424f-a23a-72205b11b4ff_isrForCrosstalkSources_855444_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/a840a83d-29a9-424f-a23a-72205b11b4ff_isrForCrosstalkSources_855444_58.stderr parsl.dataflow.dflow INFO: Task 2719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f84c5981bc66b78d9b938bff37eada with result from task 2719 parsl.dataflow.dflow DEBUG: Task 2719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/775057d3-c7fd-46a7-9e87-585c93ec8ff1_isrForCrosstalkSources_855382_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/775057d3-c7fd-46a7-9e87-585c93ec8ff1_isrForCrosstalkSources_855382_42.stderr parsl.dataflow.dflow INFO: Task 2596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b571ad62187740a34eef84137ec40bd2 with result from task 2596 parsl.dataflow.dflow DEBUG: Task 2596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/79c0087e-a9f6-4191-974e-f153e60535f6_isrForCrosstalkSources_855400_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/79c0087e-a9f6-4191-974e-f153e60535f6_isrForCrosstalkSources_855400_4.stderr parsl.dataflow.dflow INFO: Task 2816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610df0c427627dc004f70e09c9ed0f70 with result from task 2816 parsl.dataflow.dflow DEBUG: Task 2816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f6b8b7d5-26ea-4c7f-8a05-0572b3d46381_isrForCrosstalkSources_855379_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f6b8b7d5-26ea-4c7f-8a05-0572b3d46381_isrForCrosstalkSources_855379_25.stderr parsl.dataflow.dflow INFO: Task 2804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cfef73152e98236c0b4987459b475c5 with result from task 2804 parsl.dataflow.dflow DEBUG: Task 2804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/37a3e70f-bf4a-4456-bbf6-516065f3a4c2_isrForCrosstalkSources_855387_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/37a3e70f-bf4a-4456-bbf6-516065f3a4c2_isrForCrosstalkSources_855387_26.stderr parsl.dataflow.dflow INFO: Task 2814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e0cb0abe3f393f456f5f392f9cc5cbc with result from task 2814 parsl.dataflow.dflow DEBUG: Task 2814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f56f7bd0-6536-4917-be64-4f8657927ebd_isrForCrosstalkSources_855452_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/f56f7bd0-6536-4917-be64-4f8657927ebd_isrForCrosstalkSources_855452_34.stderr parsl.dataflow.dflow INFO: Task 2820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 200264c9c8b558f394ac5c74daa73ada with result from task 2820 parsl.dataflow.dflow DEBUG: Task 2820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b102af4f-00f6-42cb-9134-75df1f7fb151_isrForCrosstalkSources_855448_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b102af4f-00f6-42cb-9134-75df1f7fb151_isrForCrosstalkSources_855448_51.stderr parsl.dataflow.dflow INFO: Task 2694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e84f66b552794b81642d8114e9162a81 with result from task 2694 parsl.dataflow.dflow DEBUG: Task 2694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0453c26-4031-4683-b84d-901c512e1077_isrForCrosstalkSources_855408_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0453c26-4031-4683-b84d-901c512e1077_isrForCrosstalkSources_855408_10.stderr parsl.dataflow.dflow INFO: Task 2602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22bfec2f5bdea4bec30705224055d3ff with result from task 2602 parsl.dataflow.dflow DEBUG: Task 2602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/4eca778f-afe9-4ac2-ab2d-d6a33f7a6c8c_isrForCrosstalkSources_855402_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/4eca778f-afe9-4ac2-ab2d-d6a33f7a6c8c_isrForCrosstalkSources_855402_48.stderr parsl.dataflow.dflow INFO: Task 2766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dcc6f15a4e73ea93b47066f7ab9da5b with result from task 2766 parsl.dataflow.dflow DEBUG: Task 2766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/dce3c548-412a-45ac-aac2-5187a75e18f4_isrForCrosstalkSources_855439_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/dce3c548-412a-45ac-aac2-5187a75e18f4_isrForCrosstalkSources_855439_26.stderr parsl.dataflow.dflow INFO: Task 2556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2106c126267a38d33129d890bf212b5e with result from task 2556 parsl.dataflow.dflow DEBUG: Task 2556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/56b1f32c-3c2e-408c-a08a-aeb9cbe13d21_isrForCrosstalkSources_855389_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/56b1f32c-3c2e-408c-a08a-aeb9cbe13d21_isrForCrosstalkSources_855389_31.stderr parsl.dataflow.dflow INFO: Task 2619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae10c294a62fac813af30e10e4d2a62 with result from task 2619 parsl.dataflow.dflow DEBUG: Task 2619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5f194826-ae73-43a8-8a06-caa1c38325be_isrForCrosstalkSources_855450_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/5f194826-ae73-43a8-8a06-caa1c38325be_isrForCrosstalkSources_855450_12.stderr parsl.dataflow.dflow INFO: Task 2635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d2b0100f3bcd585f5bf200f2388f115 with result from task 2635 parsl.dataflow.dflow DEBUG: Task 2635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/71ecedce-bbbe-48d5-acb1-44257e877c9c_isrForCrosstalkSources_855435_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/71ecedce-bbbe-48d5-acb1-44257e877c9c_isrForCrosstalkSources_855435_44.stderr parsl.dataflow.dflow INFO: Task 2737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e61d4cf199ea83f3588d53f48c0cfd4a with result from task 2737 parsl.dataflow.dflow DEBUG: Task 2737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c556e707-edd8-4ee7-af82-655bd4f8ce3f_isrForCrosstalkSources_855384_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c556e707-edd8-4ee7-af82-655bd4f8ce3f_isrForCrosstalkSources_855384_21.stderr parsl.dataflow.dflow INFO: Task 2701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db670316144c9b5b263f205a6c9e706d with result from task 2701 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:05 proc_lsst.multi:146 [INFO] found job 16964205 in provider checkpoint proc_lsst.multi INFO: found job 16964205 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/064e6457-75bf-4b1f-b109-06fd55366be8_isrForCrosstalkSources_855405_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/064e6457-75bf-4b1f-b109-06fd55366be8_isrForCrosstalkSources_855405_43.stderr parsl.dataflow.dflow INFO: Task 2751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c31420da076aa9353231a3c50f63284 with result from task 2751 parsl.dataflow.dflow DEBUG: Task 2751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ac1ad1e5-72b0-48a7-b5b6-e18fdfbd00b5_isrForCrosstalkSources_855414_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ac1ad1e5-72b0-48a7-b5b6-e18fdfbd00b5_isrForCrosstalkSources_855414_40.stderr parsl.dataflow.dflow INFO: Task 2761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72eac836997b6ba2dae904dc45ce2b7 with result from task 2761 parsl.dataflow.dflow DEBUG: Task 2761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/68e95841-b8e2-4797-b3a0-877c8a9a099a_isrForCrosstalkSources_855457_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/68e95841-b8e2-4797-b3a0-877c8a9a099a_isrForCrosstalkSources_855457_52.stderr parsl.dataflow.dflow INFO: Task 2750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa185bc5e87e7f78e9a548921b52bd9 with result from task 2750 parsl.dataflow.dflow DEBUG: Task 2750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/781e1ccc-62d7-45f3-8bf0-7d8bb7701661_isrForCrosstalkSources_855457_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/781e1ccc-62d7-45f3-8bf0-7d8bb7701661_isrForCrosstalkSources_855457_13.stderr parsl.dataflow.dflow INFO: Task 2730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 533bce797de5462a19ab607c86589766 with result from task 2730 parsl.dataflow.dflow DEBUG: Task 2730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2863dbf0-c968-4bb2-943e-30c69d4967eb_isrForCrosstalkSources_855453_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2863dbf0-c968-4bb2-943e-30c69d4967eb_isrForCrosstalkSources_855453_32.stderr parsl.dataflow.dflow INFO: Task 2739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8243406136e0898eecad0dacc15d40 with result from task 2739 parsl.dataflow.dflow DEBUG: Task 2739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/0bbeca93-5254-45b5-9bc9-d0ac8cd2980c_isrForCrosstalkSources_855440_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/0bbeca93-5254-45b5-9bc9-d0ac8cd2980c_isrForCrosstalkSources_855440_62.stderr parsl.dataflow.dflow INFO: Task 2744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fbe04009cc4599898617c3d82139fca with result from task 2744 parsl.dataflow.dflow DEBUG: Task 2744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/0141c2ec-7375-4b59-a711-6d8a87896b1a_isrForCrosstalkSources_855434_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/0141c2ec-7375-4b59-a711-6d8a87896b1a_isrForCrosstalkSources_855434_41.stderr parsl.dataflow.dflow INFO: Task 2758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ff235ed492fc1a6eba008d6f09c60cf with result from task 2758 parsl.dataflow.dflow DEBUG: Task 2758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3e08101f-ef19-489e-9a74-2ae1c7c2814c_isrForCrosstalkSources_855425_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3e08101f-ef19-489e-9a74-2ae1c7c2814c_isrForCrosstalkSources_855425_61.stderr parsl.dataflow.dflow INFO: Task 2741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f52d47d80cc25f8b4dd4af85e76c3893 with result from task 2741 parsl.dataflow.dflow DEBUG: Task 2741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3dd918a8-7a2b-4e66-bfca-08163dbcd28b_isrForCrosstalkSources_855401_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3dd918a8-7a2b-4e66-bfca-08163dbcd28b_isrForCrosstalkSources_855401_5.stderr parsl.dataflow.dflow INFO: Task 2742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36d76758f9c4621ef8c66962d3aafeb1 with result from task 2742 parsl.dataflow.dflow DEBUG: Task 2742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/eee5fb6d-ddf8-4eb9-a2c2-cdfec015cfcd_isrForCrosstalkSources_855406_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/eee5fb6d-ddf8-4eb9-a2c2-cdfec015cfcd_isrForCrosstalkSources_855406_6.stderr parsl.dataflow.dflow INFO: Task 2754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ffed7d6544c78ff259e0d0a54481722 with result from task 2754 parsl.dataflow.dflow DEBUG: Task 2754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2fa9445c-1cf4-44f1-9bf5-272f31d99e88_isrForCrosstalkSources_855409_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/2fa9445c-1cf4-44f1-9bf5-272f31d99e88_isrForCrosstalkSources_855409_35.stderr parsl.dataflow.dflow INFO: Task 2738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a39d3e6bb3c889c4cf0f7e0cc5ebcfe with result from task 2738 parsl.dataflow.dflow DEBUG: Task 2738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/3310c853-831c-47c5-aa72-258a3120877e_isrForCrosstalkSources_855451_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/3310c853-831c-47c5-aa72-258a3120877e_isrForCrosstalkSources_855451_20.stderr parsl.dataflow.dflow INFO: Task 2422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1f7f15107f5eb4e6f0837a561c3381 with result from task 2422 parsl.dataflow.dflow DEBUG: Task 2422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7edbbe18-fcab-4d38-b891-3dd1c48fd40d_isrForCrosstalkSources_855443_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7edbbe18-fcab-4d38-b891-3dd1c48fd40d_isrForCrosstalkSources_855443_59.stderr parsl.dataflow.dflow INFO: Task 2850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85dfbf3721ec19f4fdb360e816b611aa with result from task 2850 parsl.dataflow.dflow DEBUG: Task 2850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/36d8fd98-45d9-48f6-915c-254eee6c00a9_isrForCrosstalkSources_855384_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/36d8fd98-45d9-48f6-915c-254eee6c00a9_isrForCrosstalkSources_855384_2.stderr parsl.dataflow.dflow INFO: Task 2800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7adeccca22b0ed2af015b1fe29aa7a96 with result from task 2800 parsl.dataflow.dflow DEBUG: Task 2800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cfc1b682-f5a2-4cee-8c8e-1f9079e85fc0_isrForCrosstalkSources_855422_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cfc1b682-f5a2-4cee-8c8e-1f9079e85fc0_isrForCrosstalkSources_855422_48.stderr parsl.dataflow.dflow INFO: Task 2764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a969c7a8386fc3f3765f9c642ebe25e with result from task 2764 parsl.dataflow.dflow DEBUG: Task 2764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/943b9dcc-18fe-4097-9c45-0353f350c196_isrForCrosstalkSources_855379_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/943b9dcc-18fe-4097-9c45-0353f350c196_isrForCrosstalkSources_855379_56.stderr parsl.dataflow.dflow INFO: Task 2802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ae93aa02133e3e76f31836879af3a0b with result from task 2802 parsl.dataflow.dflow DEBUG: Task 2802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18330a6e-976a-4c76-b5b3-b78a5e2de3bc_isrForCrosstalkSources_855423_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/18330a6e-976a-4c76-b5b3-b78a5e2de3bc_isrForCrosstalkSources_855423_49.stderr parsl.dataflow.dflow INFO: Task 2841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b563fb60267afe29bedd96a74bc1564 with result from task 2841 parsl.dataflow.dflow DEBUG: Task 2841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9973acd3-24db-454a-aaad-090390849dbf_isrForCrosstalkSources_855394_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/9973acd3-24db-454a-aaad-090390849dbf_isrForCrosstalkSources_855394_16.stderr parsl.dataflow.dflow INFO: Task 2790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e429050cbd5734ab2fe72ba3308636cd with result from task 2790 parsl.dataflow.dflow DEBUG: Task 2790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a00629d0-6432-49d8-908d-d09f4dde7837_isrForCrosstalkSources_855431_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a00629d0-6432-49d8-908d-d09f4dde7837_isrForCrosstalkSources_855431_32.stderr parsl.dataflow.dflow INFO: Task 2433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a764c7c40dc8f519be1cc184dd9c41 with result from task 2433 parsl.dataflow.dflow DEBUG: Task 2433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f0a7803b-bb46-4d05-931f-b461b0e755e1_isrForCrosstalkSources_855384_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f0a7803b-bb46-4d05-931f-b461b0e755e1_isrForCrosstalkSources_855384_58.stderr parsl.dataflow.dflow INFO: Task 2871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbdae664948b786da6ba7dce62b6dc8e with result from task 2871 parsl.dataflow.dflow DEBUG: Task 2871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/21bc1823-f416-4d1c-ae26-212f97c2e007_isrForCrosstalkSources_855382_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/21bc1823-f416-4d1c-ae26-212f97c2e007_isrForCrosstalkSources_855382_16.stderr parsl.dataflow.dflow INFO: Task 2838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad37d40d8aa1d52761be95d126931b5 with result from task 2838 parsl.dataflow.dflow DEBUG: Task 2838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b74124e0-a748-4799-9482-f30d6f5c0b34_isrForCrosstalkSources_855381_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b74124e0-a748-4799-9482-f30d6f5c0b34_isrForCrosstalkSources_855381_58.stderr parsl.dataflow.dflow INFO: Task 2876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6fb7f0f386c09ee3f6cce595ed3c582 with result from task 2876 parsl.dataflow.dflow DEBUG: Task 2876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/eec39f82-2fd6-4bc3-8663-66ecc95aeaba_isrForCrosstalkSources_855393_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/eec39f82-2fd6-4bc3-8663-66ecc95aeaba_isrForCrosstalkSources_855393_26.stderr parsl.dataflow.dflow INFO: Task 2855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 915f05eee926de2687be6673071a7b36 with result from task 2855 parsl.dataflow.dflow DEBUG: Task 2855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1dff4c23-c6c6-4d99-8349-a3bc1bba5baf_isrForCrosstalkSources_855409_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1dff4c23-c6c6-4d99-8349-a3bc1bba5baf_isrForCrosstalkSources_855409_18.stderr parsl.dataflow.dflow INFO: Task 2819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99ba37d3d8a13ad4491f4f5913d0b9ba with result from task 2819 parsl.dataflow.dflow DEBUG: Task 2819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4def6a6e-7775-4aa9-abcf-f5f76b50726a_isrForCrosstalkSources_855384_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4def6a6e-7775-4aa9-abcf-f5f76b50726a_isrForCrosstalkSources_855384_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:06 proc_lsst.multi:146 [INFO] found job 16964206 in provider checkpoint proc_lsst.multi INFO: found job 16964206 in provider checkpoint parsl.dataflow.dflow INFO: Task 2826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a45e55cfcffb23441f62af3865f6564a with result from task 2826 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0081ccb0-10e4-4d0b-a892-52c8978f5f91_isrForCrosstalkSources_855428_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/0081ccb0-10e4-4d0b-a892-52c8978f5f91_isrForCrosstalkSources_855428_25.stderr parsl.dataflow.dflow INFO: Task 2774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41ee7477d7a31979dcd8c0b9b6ea8148 with result from task 2774 parsl.dataflow.dflow DEBUG: Task 2774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b659be9-f8f7-4ad2-b269-c8fc3dfef01a_isrForCrosstalkSources_855406_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b659be9-f8f7-4ad2-b269-c8fc3dfef01a_isrForCrosstalkSources_855406_60.stderr parsl.dataflow.dflow INFO: Task 2781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac477c52bc87ff672eac40ad67a89a65 with result from task 2781 parsl.dataflow.dflow DEBUG: Task 2781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/210b038e-d4ee-4c96-8d79-17dbd25c621f_isrForCrosstalkSources_855387_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/210b038e-d4ee-4c96-8d79-17dbd25c621f_isrForCrosstalkSources_855387_19.stderr parsl.dataflow.dflow INFO: Task 2779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478d47f274ebc9e854be2054d6980374 with result from task 2779 parsl.dataflow.dflow DEBUG: Task 2779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a235ac7e-0c43-4317-a518-d36a46172dee_isrForCrosstalkSources_855382_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a235ac7e-0c43-4317-a518-d36a46172dee_isrForCrosstalkSources_855382_3.stderr parsl.dataflow.dflow INFO: Task 2560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d327faf248231b4ad91e8f8bd8d138a with result from task 2560 parsl.dataflow.dflow DEBUG: Task 2560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20d9b9ec-30b4-4750-91ba-23da5b3e071d_isrForCrosstalkSources_855418_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20d9b9ec-30b4-4750-91ba-23da5b3e071d_isrForCrosstalkSources_855418_3.stderr parsl.dataflow.dflow INFO: Task 2634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e01c4de5eff3d4027fda93612a3b9c9b with result from task 2634 parsl.dataflow.dflow DEBUG: Task 2634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5536c7e0-bb6a-4fba-95c3-8d05ee534572_isrForCrosstalkSources_855454_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/5536c7e0-bb6a-4fba-95c3-8d05ee534572_isrForCrosstalkSources_855454_8.stderr parsl.dataflow.dflow INFO: Task 2633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a25877a7cbb4ce5fa6c396ee83ca2665 with result from task 2633 parsl.dataflow.dflow DEBUG: Task 2633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b8b642cd-03e8-4097-b588-ede7d72b3e69_isrForCrosstalkSources_855436_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b8b642cd-03e8-4097-b588-ede7d72b3e69_isrForCrosstalkSources_855436_3.stderr parsl.dataflow.dflow INFO: Task 2687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 395411c07b4d0b720116b10301eed47b with result from task 2687 parsl.dataflow.dflow DEBUG: Task 2687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a8992d08-a00b-4ef9-9c67-899168504c8c_isrForCrosstalkSources_855438_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a8992d08-a00b-4ef9-9c67-899168504c8c_isrForCrosstalkSources_855438_55.stderr parsl.dataflow.dflow INFO: Task 2818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea6eab2aef734387304ad25916df89a1 with result from task 2818 parsl.dataflow.dflow DEBUG: Task 2818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c2403b27-0c1d-4dd1-a7a9-878c6cff5903_isrForCrosstalkSources_855388_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c2403b27-0c1d-4dd1-a7a9-878c6cff5903_isrForCrosstalkSources_855388_1.stderr parsl.dataflow.dflow INFO: Task 2650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c0d763db62a64689c5777050e416ee with result from task 2650 parsl.dataflow.dflow DEBUG: Task 2650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e5563863-e8f4-46c5-b38e-4fcc737b8d75_isrForCrosstalkSources_855451_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e5563863-e8f4-46c5-b38e-4fcc737b8d75_isrForCrosstalkSources_855451_42.stderr parsl.dataflow.dflow INFO: Task 2520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98a76e78f043975ae14b1a484deac047 with result from task 2520 parsl.dataflow.dflow DEBUG: Task 2520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/afdf8c8a-6960-42f8-8b3f-304e072b28d3_isrForCrosstalkSources_855447_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/afdf8c8a-6960-42f8-8b3f-304e072b28d3_isrForCrosstalkSources_855447_9.stderr parsl.dataflow.dflow INFO: Task 2835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3697f074ab31043b8c771ce7627fee6 with result from task 2835 parsl.dataflow.dflow DEBUG: Task 2835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/de2a6eeb-8c52-4656-a557-04e89e5a952d_isrForCrosstalkSources_855405_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/de2a6eeb-8c52-4656-a557-04e89e5a952d_isrForCrosstalkSources_855405_11.stderr parsl.dataflow.dflow INFO: Task 2832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4251d799cf0df07be99e7a95f79b0d6 with result from task 2832 parsl.dataflow.dflow DEBUG: Task 2832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0029f1ea-259f-407e-8d1e-1e1dd25ac411_isrForCrosstalkSources_855425_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0029f1ea-259f-407e-8d1e-1e1dd25ac411_isrForCrosstalkSources_855425_42.stderr parsl.dataflow.dflow INFO: Task 2531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 227d484982bfde7210309aec14190c61 with result from task 2531 parsl.dataflow.dflow DEBUG: Task 2531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/705caf75-d18a-4575-ba7b-e09ae3b71545_isrForCrosstalkSources_855398_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/705caf75-d18a-4575-ba7b-e09ae3b71545_isrForCrosstalkSources_855398_10.stderr parsl.dataflow.dflow INFO: Task 2696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c65ce79fa2f787a2a0f776a295040ab with result from task 2696 parsl.dataflow.dflow DEBUG: Task 2696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf57d470-06e1-4617-b263-4f35542493e8_isrForCrosstalkSources_855432_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf57d470-06e1-4617-b263-4f35542493e8_isrForCrosstalkSources_855432_19.stderr parsl.dataflow.dflow INFO: Task 2858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b162c1e9aa5b9be3ee362dc5c80de0c with result from task 2858 parsl.dataflow.dflow DEBUG: Task 2858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9e1b533a-7e6a-449a-9674-5f7e5e0223ad_isrForCrosstalkSources_855380_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9e1b533a-7e6a-449a-9674-5f7e5e0223ad_isrForCrosstalkSources_855380_32.stderr parsl.dataflow.dflow INFO: Task 2746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 292f758590864fdd2fd9e508ce09abe6 with result from task 2746 parsl.dataflow.dflow DEBUG: Task 2746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a1efe05e-0039-476d-b7a5-6da6f1078a75_isrForCrosstalkSources_855441_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a1efe05e-0039-476d-b7a5-6da6f1078a75_isrForCrosstalkSources_855441_36.stderr parsl.dataflow.dflow INFO: Task 2731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55c98c28fd6c3ec1d223220e4f6be971 with result from task 2731 parsl.dataflow.dflow DEBUG: Task 2731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4dd33150-7f94-4b7e-9b1b-513567d8e982_isrForCrosstalkSources_855433_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4dd33150-7f94-4b7e-9b1b-513567d8e982_isrForCrosstalkSources_855433_45.stderr parsl.dataflow.dflow INFO: Task 2715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 550b47f0b070184325f769b9af3d71b6 with result from task 2715 parsl.dataflow.dflow DEBUG: Task 2715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/efbe6e01-125a-4090-bf13-27d72ec54073_isrForCrosstalkSources_855433_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/efbe6e01-125a-4090-bf13-27d72ec54073_isrForCrosstalkSources_855433_18.stderr parsl.dataflow.dflow INFO: Task 2780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 654775196b69d205475aa2c176651b27 with result from task 2780 parsl.dataflow.dflow DEBUG: Task 2780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/95d75a48-292d-4be9-b16e-448c5f88e768_isrForCrosstalkSources_855383_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/95d75a48-292d-4be9-b16e-448c5f88e768_isrForCrosstalkSources_855383_32.stderr parsl.dataflow.dflow INFO: Task 2844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22daf464ee21aa24ef1b803589eedbed with result from task 2844 parsl.dataflow.dflow DEBUG: Task 2844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0eb18e47-d636-4e38-9272-a9ff15e65682_isrForCrosstalkSources_855427_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0eb18e47-d636-4e38-9272-a9ff15e65682_isrForCrosstalkSources_855427_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:06 proc_lsst.multi:146 [INFO] found job 16964207 in provider checkpoint proc_lsst.multi INFO: found job 16964207 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef031e80590bb3a6b57aef03de158b55 with result from task 2840 parsl.dataflow.dflow DEBUG: Task 2840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65c89a88-e7f6-4f74-a0cf-d5017d17cc7c_isrForCrosstalkSources_855413_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/65c89a88-e7f6-4f74-a0cf-d5017d17cc7c_isrForCrosstalkSources_855413_28.stderr parsl.dataflow.dflow INFO: Task 2681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7c9b7e748a387c1f626e2c590ffb65c with result from task 2681 parsl.dataflow.dflow DEBUG: Task 2681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b356bd46-a268-4247-ae2f-f1cff0c531fd_isrForCrosstalkSources_855426_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/b356bd46-a268-4247-ae2f-f1cff0c531fd_isrForCrosstalkSources_855426_20.stderr parsl.dataflow.dflow INFO: Task 2637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b7f6b71ce06237192a41a799d0ee9b7 with result from task 2637 parsl.dataflow.dflow DEBUG: Task 2637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9e228086-0263-4f38-8589-724737c64982_isrForCrosstalkSources_855431_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/9e228086-0263-4f38-8589-724737c64982_isrForCrosstalkSources_855431_55.stderr parsl.dataflow.dflow INFO: Task 2872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd7dcf654eb80a60f48ed9f2184d20ff with result from task 2872 parsl.dataflow.dflow DEBUG: Task 2872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b148d249-40ea-4b7e-aa65-2dce07456cd4_isrForCrosstalkSources_855415_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b148d249-40ea-4b7e-aa65-2dce07456cd4_isrForCrosstalkSources_855415_23.stderr parsl.dataflow.dflow INFO: Task 2853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab506ba26088dabbd8d04e40ef8b8be4 with result from task 2853 parsl.dataflow.dflow DEBUG: Task 2853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/51c10c10-07e6-4a43-ab78-208bbeae764c_isrForCrosstalkSources_855442_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/51c10c10-07e6-4a43-ab78-208bbeae764c_isrForCrosstalkSources_855442_16.stderr parsl.dataflow.dflow INFO: Task 2574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 511aeb58ef20d34efef7c3512fb80e2c with result from task 2574 parsl.dataflow.dflow DEBUG: Task 2574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a1fafc8-852d-4729-9911-1f35a729888f_isrForCrosstalkSources_855447_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/9a1fafc8-852d-4729-9911-1f35a729888f_isrForCrosstalkSources_855447_29.stderr parsl.dataflow.dflow INFO: Task 2796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 359066d850f9575b2fdf10fabf9115e7 with result from task 2796 parsl.dataflow.dflow DEBUG: Task 2796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9f3085e7-574b-4e0a-bf5d-739cde62d6d0_isrForCrosstalkSources_855448_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9f3085e7-574b-4e0a-bf5d-739cde62d6d0_isrForCrosstalkSources_855448_43.stderr parsl.dataflow.dflow INFO: Task 2890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 731fc5682d5d04dbd11312f18ab8916c with result from task 2890 parsl.dataflow.dflow DEBUG: Task 2890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0e27612a-7b67-4021-bca6-c402ce8ac85a_isrForCrosstalkSources_855446_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0e27612a-7b67-4021-bca6-c402ce8ac85a_isrForCrosstalkSources_855446_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:07 proc_lsst.multi:146 [INFO] found job 16964208 in provider checkpoint proc_lsst.multi INFO: found job 16964208 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddf97cdde4595ed2d5739cf130b3e338 with result from task 2830 parsl.dataflow.dflow DEBUG: Task 2830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1c50c8f0-bcfc-40a2-9984-2ab8c03fc2e1_isrForCrosstalkSources_855403_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1c50c8f0-bcfc-40a2-9984-2ab8c03fc2e1_isrForCrosstalkSources_855403_29.stderr parsl.dataflow.dflow INFO: Task 2882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f34fc8a7754ad58432517901bc290f1 with result from task 2882 parsl.dataflow.dflow DEBUG: Task 2882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db48e072-fbe4-4516-a547-bdd46b28204d_isrForCrosstalkSources_855388_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db48e072-fbe4-4516-a547-bdd46b28204d_isrForCrosstalkSources_855388_44.stderr parsl.dataflow.dflow INFO: Task 2801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d96041f0d993e2eab0cf5c678f36c92 with result from task 2801 parsl.dataflow.dflow DEBUG: Task 2801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1c165b9c-962d-4c8d-9137-ff317465e2f7_isrForCrosstalkSources_855401_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1c165b9c-962d-4c8d-9137-ff317465e2f7_isrForCrosstalkSources_855401_61.stderr parsl.dataflow.dflow INFO: Task 2699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30603e0f9b9e1bcdb3d4c6b2d3e997d8 with result from task 2699 parsl.dataflow.dflow DEBUG: Task 2699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3109761e-6e0f-47f2-816c-04aa16e3853b_isrForCrosstalkSources_855397_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/3109761e-6e0f-47f2-816c-04aa16e3853b_isrForCrosstalkSources_855397_61.stderr parsl.dataflow.dflow INFO: Task 2888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31bee9a13cd130fe9a6025afa9e974ed with result from task 2888 parsl.dataflow.dflow DEBUG: Task 2888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8b066a2f-ea8f-4d47-a6f3-0d267ab6def8_isrForCrosstalkSources_855395_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8b066a2f-ea8f-4d47-a6f3-0d267ab6def8_isrForCrosstalkSources_855395_11.stderr parsl.dataflow.dflow INFO: Task 2869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e653d54556889ef8a652a53db5a2622c with result from task 2869 parsl.dataflow.dflow DEBUG: Task 2869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ba004dae-c62a-4692-976a-7f8b96ee9429_isrForCrosstalkSources_855442_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ba004dae-c62a-4692-976a-7f8b96ee9429_isrForCrosstalkSources_855442_4.stderr parsl.dataflow.dflow INFO: Task 2875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62e073283693ec78fbf5d4954d83816b with result from task 2875 parsl.dataflow.dflow DEBUG: Task 2875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/bad00a8f-ce24-4c96-98d0-237877bf641e_isrForCrosstalkSources_855423_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/bad00a8f-ce24-4c96-98d0-237877bf641e_isrForCrosstalkSources_855423_62.stderr parsl.dataflow.dflow INFO: Task 2793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 195c6f3a9287088ec57641209f7600b9 with result from task 2793 parsl.dataflow.dflow DEBUG: Task 2793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1979cfec-9135-4b23-b597-2d1977ddee08_isrForCrosstalkSources_855417_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1979cfec-9135-4b23-b597-2d1977ddee08_isrForCrosstalkSources_855417_29.stderr parsl.dataflow.dflow INFO: Task 2868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa779c3df9802dc656b2d8f39d4bef1 with result from task 2868 parsl.dataflow.dflow DEBUG: Task 2868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bea83dc5-67f0-4e35-b846-cf420a92570c_isrForCrosstalkSources_855410_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/bea83dc5-67f0-4e35-b846-cf420a92570c_isrForCrosstalkSources_855410_4.stderr parsl.dataflow.dflow INFO: Task 2851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7bf179d50101dd940251135137e4a5f with result from task 2851 parsl.dataflow.dflow DEBUG: Task 2851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d5ab5060-31b7-4f42-9031-43ed2b3dce22_isrForCrosstalkSources_855381_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d5ab5060-31b7-4f42-9031-43ed2b3dce22_isrForCrosstalkSources_855381_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:07 proc_lsst.multi:146 [INFO] found job 16964209 in provider checkpoint proc_lsst.multi INFO: found job 16964209 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf873635871b8139a267bdbd3b72e01 with result from task 2846 parsl.dataflow.dflow DEBUG: Task 2846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fc0b5ef9-fd1c-4d28-9643-729298565ecf_isrForCrosstalkSources_855449_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/fc0b5ef9-fd1c-4d28-9643-729298565ecf_isrForCrosstalkSources_855449_48.stderr parsl.dataflow.dflow INFO: Task 2842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cadf3d1208ab60e7e4954716d2452be with result from task 2842 parsl.dataflow.dflow DEBUG: Task 2842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe43df-d108-4b33-b260-30a56f2bee93_isrForCrosstalkSources_855423_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe43df-d108-4b33-b260-30a56f2bee93_isrForCrosstalkSources_855423_42.stderr parsl.dataflow.dflow INFO: Task 2873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec0cff7f336a42fb8ea3602349adea9 with result from task 2873 parsl.dataflow.dflow DEBUG: Task 2873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b27957e2-fe69-4234-b534-c065d42d3724_isrForCrosstalkSources_855381_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b27957e2-fe69-4234-b534-c065d42d3724_isrForCrosstalkSources_855381_59.stderr parsl.dataflow.dflow INFO: Task 2808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7036544b23551901d5cfb583665fd598 with result from task 2808 parsl.dataflow.dflow DEBUG: Task 2808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/10e859ba-dd62-4c79-9d2a-537b860063c3_isrForCrosstalkSources_855389_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/10e859ba-dd62-4c79-9d2a-537b860063c3_isrForCrosstalkSources_855389_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:08 proc_lsst.multi:146 [INFO] found job 16964210 in provider checkpoint proc_lsst.multi INFO: found job 16964210 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58b63ea1353cecf1f1a2d7bd7c3878d0 with result from task 2870 parsl.dataflow.dflow DEBUG: Task 2870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3e9d0e05-ad1a-4b88-88fa-84161be46c99_isrForCrosstalkSources_855381_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/3e9d0e05-ad1a-4b88-88fa-84161be46c99_isrForCrosstalkSources_855381_14.stderr parsl.dataflow.dflow INFO: Task 2789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c532a9b379d67928e3133c9598776d70 with result from task 2789 parsl.dataflow.dflow DEBUG: Task 2789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8466a2f4-fd86-472e-a019-850a689d0d84_isrForCrosstalkSources_855393_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8466a2f4-fd86-472e-a019-850a689d0d84_isrForCrosstalkSources_855393_59.stderr parsl.dataflow.dflow INFO: Task 2734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eccf0845d3fc5971e158010a65e0f54 with result from task 2734 parsl.dataflow.dflow DEBUG: Task 2734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d50066b5-a96c-4d69-844e-7b4eb707da23_isrForCrosstalkSources_855389_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d50066b5-a96c-4d69-844e-7b4eb707da23_isrForCrosstalkSources_855389_15.stderr parsl.dataflow.dflow INFO: Task 2889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9459ce03dd103a679499699d6f5cb3 with result from task 2889 parsl.dataflow.dflow DEBUG: Task 2889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d1acbe6c-7b74-4283-ab09-75a423a0b516_isrForCrosstalkSources_855410_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/d1acbe6c-7b74-4283-ab09-75a423a0b516_isrForCrosstalkSources_855410_47.stderr parsl.dataflow.dflow INFO: Task 2799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 751ff1a0f841ca4ce0387253c2be116b with result from task 2799 parsl.dataflow.dflow DEBUG: Task 2799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1d0addc1-3cb0-43d7-8a59-a5e8f29b289c_isrForCrosstalkSources_855384_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1d0addc1-3cb0-43d7-8a59-a5e8f29b289c_isrForCrosstalkSources_855384_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:08 proc_lsst.multi:146 [INFO] found job 16964211 in provider checkpoint proc_lsst.multi INFO: found job 16964211 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61aef485714da8bf36af2dcd5589a30c with result from task 2792 parsl.dataflow.dflow DEBUG: Task 2792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/459e7dbf-4966-405a-86ba-3a37cc874444_isrForCrosstalkSources_855408_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/459e7dbf-4966-405a-86ba-3a37cc874444_isrForCrosstalkSources_855408_50.stderr parsl.dataflow.dflow INFO: Task 2795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4415fd1cf9180f62d1c7c05b5ea6061c with result from task 2795 parsl.dataflow.dflow DEBUG: Task 2795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/71951d8c-1640-49ec-b7d3-a9353ad8f7c6_isrForCrosstalkSources_855454_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/71951d8c-1640-49ec-b7d3-a9353ad8f7c6_isrForCrosstalkSources_855454_37.stderr parsl.dataflow.dflow INFO: Task 2771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f206fa4c2c8b268a961aa29a2ad0da81 with result from task 2771 parsl.dataflow.dflow DEBUG: Task 2771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/398417d8-983c-4572-8812-0fddbc9a9b11_isrForCrosstalkSources_855454_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/398417d8-983c-4572-8812-0fddbc9a9b11_isrForCrosstalkSources_855454_10.stderr parsl.dataflow.dflow INFO: Task 2836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e840735b30a1f723db706e6c97a42e83 with result from task 2836 parsl.dataflow.dflow DEBUG: Task 2836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/cf926bef-e33b-4946-ade9-db3de9f17dba_isrForCrosstalkSources_855421_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/cf926bef-e33b-4946-ade9-db3de9f17dba_isrForCrosstalkSources_855421_19.stderr parsl.dataflow.dflow INFO: Task 2845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b8c21122b7dc06c73ca45c337a34a6 with result from task 2845 parsl.dataflow.dflow DEBUG: Task 2845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/46ff46b7-4095-4b81-a319-53abf2406c2d_isrForCrosstalkSources_855444_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/46ff46b7-4095-4b81-a319-53abf2406c2d_isrForCrosstalkSources_855444_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea7cdce98f0b9978bc72e5df33c0711 with result from task 2813 parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:08 proc_lsst.multi:146 [INFO] found job 16964212 in provider checkpoint proc_lsst.multi INFO: found job 16964212 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4c953f22-dbfb-49ca-ae9a-1515781ef94a_isrForCrosstalkSources_855392_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4c953f22-dbfb-49ca-ae9a-1515781ef94a_isrForCrosstalkSources_855392_11.stderr parsl.dataflow.dflow INFO: Task 2837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66279b0cb733e6a2f2f55eeea55901ee with result from task 2837 parsl.dataflow.dflow DEBUG: Task 2837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ed70679f-13c5-4711-ba7e-498a15398f61_isrForCrosstalkSources_855413_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ed70679f-13c5-4711-ba7e-498a15398f61_isrForCrosstalkSources_855413_29.stderr parsl.dataflow.dflow INFO: Task 2759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3828cef5aefff63b61bea8245fc7eb2 with result from task 2759 parsl.dataflow.dflow DEBUG: Task 2759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a2cae7f-8afb-41a3-9436-ba274dd2b91d_isrForCrosstalkSources_855442_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a2cae7f-8afb-41a3-9436-ba274dd2b91d_isrForCrosstalkSources_855442_44.stderr parsl.dataflow.dflow INFO: Task 2878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d147774503dd51a9bb737fdb03d98a7 with result from task 2878 parsl.dataflow.dflow DEBUG: Task 2878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c8db900d-95b7-46a3-a139-04a2281493cd_isrForCrosstalkSources_855385_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c8db900d-95b7-46a3-a139-04a2281493cd_isrForCrosstalkSources_855385_42.stderr parsl.dataflow.dflow INFO: Task 2770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379bc75b463905db08e7fa95c1fedac0 with result from task 2770 parsl.dataflow.dflow DEBUG: Task 2770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3b8dcfcd-b378-4c27-b462-93952187e55c_isrForCrosstalkSources_855423_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3b8dcfcd-b378-4c27-b462-93952187e55c_isrForCrosstalkSources_855423_47.stderr parsl.dataflow.dflow INFO: Task 2707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966215fd33b61e3b6995c5b951c6a9f2 with result from task 2707 parsl.dataflow.dflow DEBUG: Task 2707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/54bd31fa-093b-4e49-afb0-1001d4ba52b9_isrForCrosstalkSources_855401_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/54bd31fa-093b-4e49-afb0-1001d4ba52b9_isrForCrosstalkSources_855401_15.stderr parsl.dataflow.dflow INFO: Task 2797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f8be45f62b6ad45843d4fa6587a9c52 with result from task 2797 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:09 proc_lsst.multi:146 [INFO] found job 16964213 in provider checkpoint proc_lsst.multi INFO: found job 16964213 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/eebbf966-ea4f-4931-b53e-da4b0c43fd8a_isrForCrosstalkSources_855410_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/eebbf966-ea4f-4931-b53e-da4b0c43fd8a_isrForCrosstalkSources_855410_34.stderr parsl.dataflow.dflow INFO: Task 2848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f3c560c2527d76f43c476358c6e1f6 with result from task 2848 parsl.dataflow.dflow DEBUG: Task 2848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6a6fe69-ae55-4e46-b0c1-af79673dd5ef_isrForCrosstalkSources_855457_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6a6fe69-ae55-4e46-b0c1-af79673dd5ef_isrForCrosstalkSources_855457_48.stderr parsl.dataflow.dflow INFO: Task 2787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e9510228b68fa951e89b71c34e71d07 with result from task 2787 parsl.dataflow.dflow DEBUG: Task 2787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f2fa49f3-f380-40d4-a3d3-450b0455c027_isrForCrosstalkSources_855423_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f2fa49f3-f380-40d4-a3d3-450b0455c027_isrForCrosstalkSources_855423_58.stderr parsl.dataflow.dflow INFO: Task 2843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d547a978c470c533f7f774745274fb2 with result from task 2843 parsl.dataflow.dflow DEBUG: Task 2843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/267cf525-fc71-45d0-b4a2-4e30697e99b7_isrForCrosstalkSources_855448_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/267cf525-fc71-45d0-b4a2-4e30697e99b7_isrForCrosstalkSources_855448_12.stderr parsl.dataflow.dflow INFO: Task 2828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d576bdaeefdebcc0555a4ce488030e72 with result from task 2828 parsl.dataflow.dflow DEBUG: Task 2828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78a61bd8-0633-43ca-abc1-6c7264b12980_isrForCrosstalkSources_855387_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/78a61bd8-0633-43ca-abc1-6c7264b12980_isrForCrosstalkSources_855387_24.stderr parsl.dataflow.dflow INFO: Task 2829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5e224ad893a2a15816adeba97e4df9f with result from task 2829 parsl.dataflow.dflow DEBUG: Task 2829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2017614-add4-42cf-b273-2402dc5bb13e_isrForCrosstalkSources_855382_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2017614-add4-42cf-b273-2402dc5bb13e_isrForCrosstalkSources_855382_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.dataflow.dflow INFO: Task 2967 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry f04c80c9e92eae34bc7e5b8eb366fe05 with result from task 2967 parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:10 proc_lsst.multi:146 [INFO] found job 16964214 in provider checkpoint proc_lsst.multi INFO: found job 16964214 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 2967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5caf12e0-d2f9-4109-8837-aa02d946430e_isrForCrosstalkSources_855458_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5caf12e0-d2f9-4109-8837-aa02d946430e_isrForCrosstalkSources_855458_26.stderr parsl.dataflow.dflow INFO: Task 3067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 798eb9fdf0f1acdfa78b75325684faa7 with result from task 3067 parsl.dataflow.dflow DEBUG: Task 3067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/471a54e4-df06-4159-8a2f-b9ef665d4306_isrForCrosstalkSources_855437_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/471a54e4-df06-4159-8a2f-b9ef665d4306_isrForCrosstalkSources_855437_39.stderr parsl.dataflow.dflow INFO: Task 2791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33edb3cc3e1752beab021e9018fb5deb with result from task 2791 parsl.dataflow.dflow DEBUG: Task 2791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7729f04b-dff2-43f1-9a59-18047668277b_isrForCrosstalkSources_855456_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/7729f04b-dff2-43f1-9a59-18047668277b_isrForCrosstalkSources_855456_58.stderr parsl.dataflow.dflow INFO: Task 2849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae06f3d7be67c1b11dcb1bc989e7b83 with result from task 2849 parsl.dataflow.dflow DEBUG: Task 2849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5e2d4615-c189-466e-b537-b4f0e16614d5_isrForCrosstalkSources_855428_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5e2d4615-c189-466e-b537-b4f0e16614d5_isrForCrosstalkSources_855428_59.stderr parsl.dataflow.dflow INFO: Task 2856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b508afd838e0225d88cac92a0a89543c with result from task 2856 parsl.dataflow.dflow DEBUG: Task 2856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/be814532-e83a-4a9b-b4a1-2d48e20c7382_isrForCrosstalkSources_855405_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/be814532-e83a-4a9b-b4a1-2d48e20c7382_isrForCrosstalkSources_855405_19.stderr parsl.dataflow.dflow INFO: Task 2812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f97eb982190a67ff7cd03c754403dfe with result from task 2812 parsl.dataflow.dflow DEBUG: Task 2812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6fc45cd7-3f2d-48a3-9cce-ccb643d68279_isrForCrosstalkSources_855415_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6fc45cd7-3f2d-48a3-9cce-ccb643d68279_isrForCrosstalkSources_855415_60.stderr parsl.dataflow.dflow INFO: Task 2834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd4c7da40921fa8b41872cd06b3fdc0 with result from task 2834 parsl.dataflow.dflow DEBUG: Task 2834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c93d5bb2-e0cb-4621-8703-70a9939a16d1_isrForCrosstalkSources_855420_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c93d5bb2-e0cb-4621-8703-70a9939a16d1_isrForCrosstalkSources_855420_58.stderr parsl.dataflow.dflow INFO: Task 2852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9953a3e24946ab4faf2fa39a3875c17 with result from task 2852 parsl.dataflow.dflow DEBUG: Task 2852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/add8afd4-1065-47a5-b5e5-eafc50fb8b1f_isrForCrosstalkSources_855419_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/add8afd4-1065-47a5-b5e5-eafc50fb8b1f_isrForCrosstalkSources_855419_14.stderr parsl.dataflow.dflow INFO: Task 2860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ced147320ca8b4e5766e8d5faab9e5a with result from task 2860 parsl.dataflow.dflow DEBUG: Task 2860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2b819dd0-ad9f-456a-a33a-78afd4359d53_isrForCrosstalkSources_855419_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2b819dd0-ad9f-456a-a33a-78afd4359d53_isrForCrosstalkSources_855419_38.stderr parsl.dataflow.dflow INFO: Task 3024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 035b387e3133fb2cbc0860a09ef563cb with result from task 3024 parsl.dataflow.dflow DEBUG: Task 3024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c0f3b3f1-7ab4-4675-93ea-115cff4d1907_isrForCrosstalkSources_855410_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c0f3b3f1-7ab4-4675-93ea-115cff4d1907_isrForCrosstalkSources_855410_51.stderr parsl.dataflow.dflow INFO: Task 2815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 638cea57068d4a1ab7ed65b7391021b6 with result from task 2815 parsl.dataflow.dflow DEBUG: Task 2815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f4ee715c-12e4-4b48-9fce-8714659aed37_isrForCrosstalkSources_855444_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f4ee715c-12e4-4b48-9fce-8714659aed37_isrForCrosstalkSources_855444_37.stderr parsl.dataflow.dflow INFO: Task 2839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a5faca61ff6932bdf1166677db018c6 with result from task 2839 parsl.dataflow.dflow DEBUG: Task 2839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/dc4d5441-a444-4674-b634-d6b2f675145a_isrForCrosstalkSources_855457_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/dc4d5441-a444-4674-b634-d6b2f675145a_isrForCrosstalkSources_855457_2.stderr parsl.dataflow.dflow INFO: Task 2825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a112f1e67940b2128039877207e0934 with result from task 2825 parsl.dataflow.dflow DEBUG: Task 2825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/694edef1-f8ca-43ad-8143-a4ed1dba2a65_isrForCrosstalkSources_855401_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/694edef1-f8ca-43ad-8143-a4ed1dba2a65_isrForCrosstalkSources_855401_30.stderr parsl.dataflow.dflow INFO: Task 2885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bcc208c540cd6c7ad29aaf82bdf04e0 with result from task 2885 parsl.dataflow.dflow DEBUG: Task 2885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/eebd0392-76cf-4f72-a784-d94cf65ccd7b_isrForCrosstalkSources_855434_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/eebd0392-76cf-4f72-a784-d94cf65ccd7b_isrForCrosstalkSources_855434_22.stderr parsl.dataflow.dflow INFO: Task 2755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1749847f247ffb0f4cc31ff8a8b4d3b with result from task 2755 parsl.dataflow.dflow DEBUG: Task 2755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f2d4f20-a5af-4d59-8391-09b9be65db49_isrForCrosstalkSources_855456_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2f2d4f20-a5af-4d59-8391-09b9be65db49_isrForCrosstalkSources_855456_37.stderr parsl.dataflow.dflow INFO: Task 2866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a452ff72b94d0a3bbbd2fc66ea953cfc with result from task 2866 parsl.dataflow.dflow DEBUG: Task 2866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/90f59eb0-d3b1-4075-a17d-12c3c49dc50e_isrForCrosstalkSources_855418_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/90f59eb0-d3b1-4075-a17d-12c3c49dc50e_isrForCrosstalkSources_855418_48.stderr parsl.dataflow.dflow INFO: Task 2782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7ad13e8aab2cb750dc6b0ed05f7e0ea with result from task 2782 parsl.dataflow.dflow DEBUG: Task 2782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e9d614c7-0cc8-4858-8828-cf3374d103e1_isrForCrosstalkSources_855400_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e9d614c7-0cc8-4858-8828-cf3374d103e1_isrForCrosstalkSources_855400_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:10 proc_lsst.multi:146 [INFO] found job 16964215 in provider checkpoint proc_lsst.multi INFO: found job 16964215 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4188931e808cda5d9be15ef76f2cd0f with result from task 2896 parsl.dataflow.dflow DEBUG: Task 2896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ee4653b1-5ff1-4061-ac0a-cb1eb1f3c655_isrForCrosstalkSources_855450_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/ee4653b1-5ff1-4061-ac0a-cb1eb1f3c655_isrForCrosstalkSources_855450_8.stderr parsl.dataflow.dflow INFO: Task 2705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cb2cf64159638e5cf5bea1db70bd560 with result from task 2705 parsl.dataflow.dflow DEBUG: Task 2705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/78ec3619-04b1-41c9-8373-9ba64550b6a5_isrForCrosstalkSources_855406_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/78ec3619-04b1-41c9-8373-9ba64550b6a5_isrForCrosstalkSources_855406_29.stderr parsl.dataflow.dflow INFO: Task 2847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29036cf427b33f85e2f512f675271e0b with result from task 2847 parsl.dataflow.dflow DEBUG: Task 2847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cd09ea79-4a1f-41c7-a5bb-ae76ea5139e5_isrForCrosstalkSources_855407_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cd09ea79-4a1f-41c7-a5bb-ae76ea5139e5_isrForCrosstalkSources_855407_2.stderr parsl.dataflow.dflow INFO: Task 2877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d51a3095fa17bda92d0dae3206dc785 with result from task 2877 parsl.dataflow.dflow DEBUG: Task 2877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cae5085c-dc87-4090-8941-ed76bc64c463_isrForCrosstalkSources_855422_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/cae5085c-dc87-4090-8941-ed76bc64c463_isrForCrosstalkSources_855422_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:11 proc_lsst.multi:146 [INFO] found job 16964216 in provider checkpoint proc_lsst.multi INFO: found job 16964216 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a2381f3a7d4fafd93e6da3ab5c012a8 with result from task 2874 parsl.dataflow.dflow DEBUG: Task 2874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5853d7ed-8e7e-47e9-8d47-0cfa3abc0994_isrForCrosstalkSources_855433_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/5853d7ed-8e7e-47e9-8d47-0cfa3abc0994_isrForCrosstalkSources_855433_26.stderr parsl.dataflow.dflow INFO: Task 2769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e629c09d2737ef7c0016349798555b5 with result from task 2769 parsl.dataflow.dflow DEBUG: Task 2769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/25ba655c-d59c-4829-8895-895221c08550_isrForCrosstalkSources_855435_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/25ba655c-d59c-4829-8895-895221c08550_isrForCrosstalkSources_855435_36.stderr parsl.dataflow.dflow INFO: Task 2777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52534df7d3eb6df02d717c8dfb6dbe21 with result from task 2777 parsl.dataflow.dflow DEBUG: Task 2777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0588ecfa-0a9e-4eeb-83de-a4788059cf6e_isrForCrosstalkSources_855424_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0588ecfa-0a9e-4eeb-83de-a4788059cf6e_isrForCrosstalkSources_855424_61.stderr parsl.dataflow.dflow INFO: Task 2747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1593108cf60d50371157ecea2c4df33a with result from task 2747 parsl.dataflow.dflow DEBUG: Task 2747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6506b2f4-2629-454c-93c4-15bcd42d1b90_isrForCrosstalkSources_855443_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/6506b2f4-2629-454c-93c4-15bcd42d1b90_isrForCrosstalkSources_855443_38.stderr parsl.dataflow.dflow INFO: Task 2821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24c9bf436a0b24f02e86582d0433073 with result from task 2821 parsl.dataflow.dflow DEBUG: Task 2821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f6ab3741-dd04-48dd-a104-7c6654b480b1_isrForCrosstalkSources_855426_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f6ab3741-dd04-48dd-a104-7c6654b480b1_isrForCrosstalkSources_855426_36.stderr parsl.dataflow.dflow INFO: Task 2763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70603fbe6ba3181871287ca82aab626c with result from task 2763 parsl.dataflow.dflow DEBUG: Task 2763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/326775bb-8cb0-4e53-adad-ed2ce200d3cc_isrForCrosstalkSources_855391_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/326775bb-8cb0-4e53-adad-ed2ce200d3cc_isrForCrosstalkSources_855391_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:11 proc_lsst.multi:146 [INFO] found job 16964217 in provider checkpoint proc_lsst.multi INFO: found job 16964217 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32ec218f1bf6b136a6b01ddca1103f09 with result from task 2985 parsl.dataflow.dflow DEBUG: Task 2985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/54c490ab-91ea-4372-9903-f9dbc5f0b040_isrForCrosstalkSources_855413_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/54c490ab-91ea-4372-9903-f9dbc5f0b040_isrForCrosstalkSources_855413_40.stderr parsl.dataflow.dflow INFO: Task 2879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f772a80afc3c80e8a241c72c7ea1b3a with result from task 2879 parsl.dataflow.dflow DEBUG: Task 2879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/611f4f07-86ad-4548-b64b-7df7d7599ec5_isrForCrosstalkSources_855441_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/611f4f07-86ad-4548-b64b-7df7d7599ec5_isrForCrosstalkSources_855441_52.stderr parsl.dataflow.dflow INFO: Task 3009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3714875f82b4940488b971164b2a6dd5 with result from task 3009 parsl.dataflow.dflow DEBUG: Task 3009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a1ce4a50-3e33-4e73-b299-e1bd957858e1_isrForCrosstalkSources_855451_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a1ce4a50-3e33-4e73-b299-e1bd957858e1_isrForCrosstalkSources_855451_25.stderr parsl.dataflow.dflow INFO: Task 2752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f38090d86d20f1d2cec9cb1d748f05 with result from task 2752 parsl.dataflow.dflow DEBUG: Task 2752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/30afef6e-6c9b-4892-b24d-b2bf9e34017d_isrForCrosstalkSources_855397_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/30afef6e-6c9b-4892-b24d-b2bf9e34017d_isrForCrosstalkSources_855397_62.stderr parsl.dataflow.dflow INFO: Task 2994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a0fc5374d266f673edf1d38ca7d76fb with result from task 2994 parsl.dataflow.dflow DEBUG: Task 2994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d5bef3e7-218c-4904-8d04-ff77fd118408_isrForCrosstalkSources_855402_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/d5bef3e7-218c-4904-8d04-ff77fd118408_isrForCrosstalkSources_855402_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 2969 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry f55c49fda00ecc228452fa0160279246 with result from task 2969 parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2969 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a3e84d5a-8ee1-4d02-afc0-e794d86ce346_isrForCrosstalkSources_855393_20.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a3e84d5a-8ee1-4d02-afc0-e794d86ce346_isrForCrosstalkSources_855393_20.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:11 proc_lsst.multi:146 [INFO] found job 16964218 in provider checkpoint proc_lsst.multi INFO: found job 16964218 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19afaced834a5b0ae3d2b78807ddba0 with result from task 2859 parsl.dataflow.dflow DEBUG: Task 2859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918ec45-b20f-49b2-b1bd-6fc770db7ae2_isrForCrosstalkSources_855404_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918ec45-b20f-49b2-b1bd-6fc770db7ae2_isrForCrosstalkSources_855404_38.stderr parsl.dataflow.dflow INFO: Task 2865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a539c74529da61e2e5488922fb689aa with result from task 2865 parsl.dataflow.dflow DEBUG: Task 2865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/544d056e-9d5d-40b8-98f4-67660e544302_isrForCrosstalkSources_855419_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/544d056e-9d5d-40b8-98f4-67660e544302_isrForCrosstalkSources_855419_33.stderr parsl.dataflow.dflow INFO: Task 2880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80095bd666b15cfd33fd4237386b60f3 with result from task 2880 parsl.dataflow.dflow DEBUG: Task 2880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/32c6c38e-9725-48fa-91b5-007684518147_isrForCrosstalkSources_855411_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/32c6c38e-9725-48fa-91b5-007684518147_isrForCrosstalkSources_855411_40.stderr parsl.dataflow.dflow INFO: Task 2899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea0d88f7e40582a9f328622d29c9183c with result from task 2899 parsl.dataflow.dflow DEBUG: Task 2899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eec25275-1e5a-477d-b959-a004185d0d85_isrForCrosstalkSources_855446_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eec25275-1e5a-477d-b959-a004185d0d85_isrForCrosstalkSources_855446_17.stderr parsl.dataflow.dflow INFO: Task 2857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b103833e2af2dfd7c0ed8937fb2b5905 with result from task 2857 parsl.dataflow.dflow DEBUG: Task 2857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/9e441014-4a40-46a9-8503-cef7db0a1335_isrForCrosstalkSources_855405_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/9e441014-4a40-46a9-8503-cef7db0a1335_isrForCrosstalkSources_855405_8.stderr parsl.dataflow.dflow INFO: Task 3025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d420aad593d6125897d42366bae2403 with result from task 3025 parsl.dataflow.dflow DEBUG: Task 3025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/048db880-43d0-4a01-a4d2-0478f1937cdd_isrForCrosstalkSources_855411_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/048db880-43d0-4a01-a4d2-0478f1937cdd_isrForCrosstalkSources_855411_20.stderr parsl.dataflow.dflow INFO: Task 2861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8112de04f122a03301822847117ad9dc with result from task 2861 parsl.dataflow.dflow DEBUG: Task 2861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6553d294-b8e7-4869-9ec8-39c29660854d_isrForCrosstalkSources_855392_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6553d294-b8e7-4869-9ec8-39c29660854d_isrForCrosstalkSources_855392_54.stderr parsl.dataflow.dflow INFO: Task 2977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d9e4317a964f6ec4624ef829092466b with result from task 2977 parsl.dataflow.dflow DEBUG: Task 2977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/54622395-6534-4852-8456-fd1028d0890c_isrForCrosstalkSources_855429_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/54622395-6534-4852-8456-fd1028d0890c_isrForCrosstalkSources_855429_33.stderr parsl.dataflow.dflow INFO: Task 3005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478924f814f570344ea8f1385e9d1030 with result from task 3005 parsl.dataflow.dflow DEBUG: Task 3005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0db106a-e486-4b1a-8021-c69e5fcdf399_isrForCrosstalkSources_855388_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b0db106a-e486-4b1a-8021-c69e5fcdf399_isrForCrosstalkSources_855388_19.stderr parsl.dataflow.dflow INFO: Task 2817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc3f24d4a21c5ad377d1353fd58a15a with result from task 2817 parsl.dataflow.dflow DEBUG: Task 2817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/062a5d7f-d415-4eb7-94f6-40e68d5f5202_isrForCrosstalkSources_855414_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/062a5d7f-d415-4eb7-94f6-40e68d5f5202_isrForCrosstalkSources_855414_50.stderr parsl.dataflow.dflow INFO: Task 2907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4518ff85902d298c34376e189e0809b9 with result from task 2907 parsl.dataflow.dflow DEBUG: Task 2907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/dac43149-1333-4b22-b991-b36945f0f092_isrForCrosstalkSources_855411_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/dac43149-1333-4b22-b991-b36945f0f092_isrForCrosstalkSources_855411_35.stderr parsl.dataflow.dflow INFO: Task 2887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b97bdd2fa01a5a33a816cd8944613b with result from task 2887 parsl.dataflow.dflow DEBUG: Task 2887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ecada91b-e567-4404-976f-a9357ec728f4_isrForCrosstalkSources_855384_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/ecada91b-e567-4404-976f-a9357ec728f4_isrForCrosstalkSources_855384_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2718 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 5586320a4b9c206bcb446c4ed8055625 with result from task 2718 parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2718 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/011bc451-d838-491e-8a54-b370a0f6fc52_isrForCrosstalkSources_855410_19.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/011bc451-d838-491e-8a54-b370a0f6fc52_isrForCrosstalkSources_855410_19.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:11 proc_lsst.multi:146 [INFO] found job 16964220 in provider checkpoint proc_lsst.multi INFO: found job 16964220 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c4a75f618117a751e6bb281906a2176 with result from task 3029 parsl.dataflow.dflow DEBUG: Task 3029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4793ef6f-3ccd-4526-bc62-c74d17f6a3d3_isrForCrosstalkSources_855415_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4793ef6f-3ccd-4526-bc62-c74d17f6a3d3_isrForCrosstalkSources_855415_25.stderr parsl.dataflow.dflow INFO: Task 2988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 401308f2fd1c7e1f0e8070ea0c3b4269 with result from task 2988 parsl.dataflow.dflow DEBUG: Task 2988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f2474901-45d6-44f4-9ec8-816b27512454_isrForCrosstalkSources_855451_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f2474901-45d6-44f4-9ec8-816b27512454_isrForCrosstalkSources_855451_27.stderr parsl.dataflow.dflow INFO: Task 2989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43fe0159f3ff8e78a0b100e93d1d121 with result from task 2989 parsl.dataflow.dflow DEBUG: Task 2989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7d768068-8c89-4f8d-9bff-25bd6ac84fad_isrForCrosstalkSources_855406_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7d768068-8c89-4f8d-9bff-25bd6ac84fad_isrForCrosstalkSources_855406_33.stderr parsl.dataflow.dflow INFO: Task 3097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 120c9af6a90bbf939c119ce12e94e60e with result from task 3097 parsl.dataflow.dflow DEBUG: Task 3097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/aea7da20-708f-4cc1-8337-91a4ed06e834_isrForCrosstalkSources_855427_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/aea7da20-708f-4cc1-8337-91a4ed06e834_isrForCrosstalkSources_855427_45.stderr parsl.dataflow.dflow INFO: Task 2993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b7f188df7e1b444cc310875aa93c59 with result from task 2993 parsl.dataflow.dflow DEBUG: Task 2993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ba611f8-094b-4f3f-9755-b777c007d241_isrForCrosstalkSources_855427_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8ba611f8-094b-4f3f-9755-b777c007d241_isrForCrosstalkSources_855427_46.stderr parsl.dataflow.dflow INFO: Task 2995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c575abbf9a9b2b5d81c48c4269ef17 with result from task 2995 parsl.dataflow.dflow DEBUG: Task 2995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/36bbd2b6-63f7-4ea4-b126-5b756e45d733_isrForCrosstalkSources_855379_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/36bbd2b6-63f7-4ea4-b126-5b756e45d733_isrForCrosstalkSources_855379_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:12 proc_lsst.multi:146 [INFO] found job 16964221 in provider checkpoint proc_lsst.multi INFO: found job 16964221 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b5535291377c8634925a6226078bdd4 with result from task 3109 parsl.dataflow.dflow DEBUG: Task 3109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/fef3471e-c8c6-4fe0-a4fb-253be46bcf94_isrForCrosstalkSources_855456_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/fef3471e-c8c6-4fe0-a4fb-253be46bcf94_isrForCrosstalkSources_855456_13.stderr parsl.dataflow.dflow INFO: Task 2854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0d657d857d445e7b3f8e29e339f4dd7 with result from task 2854 parsl.dataflow.dflow DEBUG: Task 2854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/af5a5ac1-6b17-46a4-96e2-f75fe72b132f_isrForCrosstalkSources_855450_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/af5a5ac1-6b17-46a4-96e2-f75fe72b132f_isrForCrosstalkSources_855450_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:12 proc_lsst.multi:146 [INFO] found job 16964222 in provider checkpoint proc_lsst.multi INFO: found job 16964222 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:12 proc_lsst.multi:146 [INFO] found job 16964223 in provider checkpoint proc_lsst.multi INFO: found job 16964223 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:13 proc_lsst.multi:146 [INFO] found job 16964224 in provider checkpoint proc_lsst.multi INFO: found job 16964224 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6f156186e130308dfe8285e76cfe7b5 with result from task 3071 parsl.dataflow.dflow DEBUG: Task 3071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ed412f6c-d6eb-4d6f-9961-7de0b7b0e23a_isrForCrosstalkSources_855392_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ed412f6c-d6eb-4d6f-9961-7de0b7b0e23a_isrForCrosstalkSources_855392_27.stderr parsl.dataflow.dflow INFO: Task 2903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 928da7059e0c5464997f285a4feabb36 with result from task 2903 parsl.dataflow.dflow DEBUG: Task 2903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/fb880062-15e9-48e5-9133-0996e05e8258_isrForCrosstalkSources_855422_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/fb880062-15e9-48e5-9133-0996e05e8258_isrForCrosstalkSources_855422_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:13 proc_lsst.multi:146 [INFO] found job 16964225 in provider checkpoint proc_lsst.multi INFO: found job 16964225 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 134792514594d9b039b23d9450d9a2c8 with result from task 3008 parsl.dataflow.dflow DEBUG: Task 3008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6e1325de-38a4-453d-bce4-8093d76c2299_isrForCrosstalkSources_855429_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6e1325de-38a4-453d-bce4-8093d76c2299_isrForCrosstalkSources_855429_40.stderr parsl.dataflow.dflow INFO: Task 2884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e98cb39b008fcce5603438a58fa03ac with result from task 2884 parsl.dataflow.dflow DEBUG: Task 2884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/8489eddc-131e-4659-94b1-db07e6cd9656_isrForCrosstalkSources_855417_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/8489eddc-131e-4659-94b1-db07e6cd9656_isrForCrosstalkSources_855417_19.stderr parsl.dataflow.dflow INFO: Task 2941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8947771e8339584e38c7d18720e61177 with result from task 2941 parsl.dataflow.dflow DEBUG: Task 2941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/395a77d8-11f5-49eb-9fae-719d22e1fb41_isrForCrosstalkSources_855414_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/395a77d8-11f5-49eb-9fae-719d22e1fb41_isrForCrosstalkSources_855414_17.stderr parsl.dataflow.dflow INFO: Task 3043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c686e91e588339c7a23e06a65c7180 with result from task 3043 parsl.dataflow.dflow DEBUG: Task 3043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b93a3721-abc7-4a7a-a79e-d3338b9ff956_isrForCrosstalkSources_855454_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b93a3721-abc7-4a7a-a79e-d3338b9ff956_isrForCrosstalkSources_855454_25.stderr parsl.dataflow.dflow INFO: Task 2927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6da267f29b3507543b4e63790b2269cf with result from task 2927 parsl.dataflow.dflow DEBUG: Task 2927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/89b04792-8137-4e13-a249-01df559f5993_isrForCrosstalkSources_855383_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/89b04792-8137-4e13-a249-01df559f5993_isrForCrosstalkSources_855383_27.stderr parsl.dataflow.dflow INFO: Task 2893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 868d38a7912c146d6350d7c060fbb458 with result from task 2893 parsl.dataflow.dflow DEBUG: Task 2893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/6d15e703-ff5d-4399-a1f2-200fb6a0ac94_isrForCrosstalkSources_855388_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/6d15e703-ff5d-4399-a1f2-200fb6a0ac94_isrForCrosstalkSources_855388_11.stderr parsl.dataflow.dflow INFO: Task 2934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b0574cde876f5022faf13a4edd7525e with result from task 2934 parsl.dataflow.dflow DEBUG: Task 2934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1b1cea2d-a9f2-4151-a3e8-d5d264ccbff2_isrForCrosstalkSources_855459_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1b1cea2d-a9f2-4151-a3e8-d5d264ccbff2_isrForCrosstalkSources_855459_11.stderr parsl.dataflow.dflow INFO: Task 3039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b3cbd13f880d3100f77555e5375f005 with result from task 3039 parsl.dataflow.dflow DEBUG: Task 3039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5ffe6500-bf47-454b-9848-9ec1bc0ea2cc_isrForCrosstalkSources_855449_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5ffe6500-bf47-454b-9848-9ec1bc0ea2cc_isrForCrosstalkSources_855449_26.stderr parsl.dataflow.dflow INFO: Task 2901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e68560d0c88bc447ffad820d8074fe with result from task 2901 parsl.dataflow.dflow DEBUG: Task 2901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/258910da-2ca9-4fbe-9e34-b71bc747cbb6_isrForCrosstalkSources_855437_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/258910da-2ca9-4fbe-9e34-b71bc747cbb6_isrForCrosstalkSources_855437_43.stderr parsl.dataflow.dflow INFO: Task 2897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7bd4aa437e6331d3c1962d54555aca3 with result from task 2897 parsl.dataflow.dflow DEBUG: Task 2897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/137c4698-e6f6-4ae1-9f75-266db303f298_isrForCrosstalkSources_855420_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/137c4698-e6f6-4ae1-9f75-266db303f298_isrForCrosstalkSources_855420_6.stderr parsl.dataflow.dflow INFO: Task 2955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f57faeb51cbe0ac24928b9b462103e11 with result from task 2955 parsl.dataflow.dflow DEBUG: Task 2955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9b3b37f3-5bcb-49fc-b849-98679d4bfb02_isrForCrosstalkSources_855399_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9b3b37f3-5bcb-49fc-b849-98679d4bfb02_isrForCrosstalkSources_855399_10.stderr parsl.dataflow.dflow INFO: Task 2961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0e430829c303141dcaf80a9118c3e8b with result from task 2961 parsl.dataflow.dflow DEBUG: Task 2961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/7160b75d-6518-4fe2-9635-d558c3414c16_isrForCrosstalkSources_855447_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/7160b75d-6518-4fe2-9635-d558c3414c16_isrForCrosstalkSources_855447_56.stderr parsl.dataflow.dflow INFO: Task 3036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02f876e308e4d72330a1c0a92c2e1f82 with result from task 3036 parsl.dataflow.dflow DEBUG: Task 3036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5338b710-1b6f-4810-aabf-1f55a9960c97_isrForCrosstalkSources_855418_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5338b710-1b6f-4810-aabf-1f55a9960c97_isrForCrosstalkSources_855418_27.stderr parsl.dataflow.dflow INFO: Task 2973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b55afbc971876c71ecab64e9025ed34b with result from task 2973 parsl.dataflow.dflow DEBUG: Task 2973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7126994c-f289-4405-b3ac-49e5cedf2b54_isrForCrosstalkSources_855422_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/7126994c-f289-4405-b3ac-49e5cedf2b54_isrForCrosstalkSources_855422_33.stderr parsl.dataflow.dflow INFO: Task 2958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaafaaef37c6a82f153ff12996cfdfbc with result from task 2958 parsl.dataflow.dflow DEBUG: Task 2958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b760b8ca-9965-483d-aeaa-ff4252f4890e_isrForCrosstalkSources_855381_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/b760b8ca-9965-483d-aeaa-ff4252f4890e_isrForCrosstalkSources_855381_56.stderr parsl.dataflow.dflow INFO: Task 3066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f825062502028a64eec0aca6120e314 with result from task 3066 parsl.dataflow.dflow DEBUG: Task 3066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/adf30d3b-a151-4069-83cd-4f214929ca62_isrForCrosstalkSources_855381_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/adf30d3b-a151-4069-83cd-4f214929ca62_isrForCrosstalkSources_855381_32.stderr parsl.dataflow.dflow INFO: Task 2935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2fe2af1b367b7d9e94636c79f2a1ef1 with result from task 2935 parsl.dataflow.dflow DEBUG: Task 2935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/412509dc-3a20-42f0-8aa8-bf0e2983b2f5_isrForCrosstalkSources_855394_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/412509dc-3a20-42f0-8aa8-bf0e2983b2f5_isrForCrosstalkSources_855394_46.stderr parsl.dataflow.dflow INFO: Task 2957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c472ac20a3f160dce154a385824d2df with result from task 2957 parsl.dataflow.dflow DEBUG: Task 2957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7ed1e183-1cac-4d8d-9a60-85c04d8ade87_isrForCrosstalkSources_855403_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/7ed1e183-1cac-4d8d-9a60-85c04d8ade87_isrForCrosstalkSources_855403_14.stderr parsl.dataflow.dflow INFO: Task 2917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae1928f1ac5eba0d17f7ed7cfef00747 with result from task 2917 parsl.dataflow.dflow DEBUG: Task 2917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8a8ff100-a2ec-4214-bcbe-79820791394d_isrForCrosstalkSources_855407_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8a8ff100-a2ec-4214-bcbe-79820791394d_isrForCrosstalkSources_855407_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:13 proc_lsst.multi:146 [INFO] found job 16964226 in provider checkpoint proc_lsst.multi INFO: found job 16964226 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90348297cd05aa5cadcfb356257b99ad with result from task 2712 parsl.dataflow.dflow DEBUG: Task 2712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/19eb876b-de49-40da-a680-f00bb229be51_isrForCrosstalkSources_855430_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/19eb876b-de49-40da-a680-f00bb229be51_isrForCrosstalkSources_855430_35.stderr parsl.dataflow.dflow INFO: Task 3076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e84746c866af702b41e3eaed1cebb1 with result from task 3076 parsl.dataflow.dflow DEBUG: Task 3076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ffe49e53-180a-4468-a53d-b76023572a89_isrForCrosstalkSources_855394_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ffe49e53-180a-4468-a53d-b76023572a89_isrForCrosstalkSources_855394_56.stderr parsl.dataflow.dflow INFO: Task 2931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 301722810146c3ec994aa680bb49bb92 with result from task 2931 parsl.dataflow.dflow DEBUG: Task 2931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/84c2b62d-d718-40fa-ac38-1e51f073c27e_isrForCrosstalkSources_855443_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/84c2b62d-d718-40fa-ac38-1e51f073c27e_isrForCrosstalkSources_855443_13.stderr parsl.dataflow.dflow INFO: Task 2932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8890cc9593db6aeb04939e5f1a40a997 with result from task 2932 parsl.dataflow.dflow DEBUG: Task 2932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2cfe14d7-365e-43f1-a539-c2ecba14aadd_isrForCrosstalkSources_855434_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2cfe14d7-365e-43f1-a539-c2ecba14aadd_isrForCrosstalkSources_855434_46.stderr parsl.dataflow.dflow INFO: Task 2906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d3a44b2d537867e1d4f7d514bce4a99 with result from task 2906 parsl.dataflow.dflow DEBUG: Task 2906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccaddd3a-6150-4004-b5ff-71d5ad63497f_isrForCrosstalkSources_855433_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccaddd3a-6150-4004-b5ff-71d5ad63497f_isrForCrosstalkSources_855433_27.stderr parsl.dataflow.dflow INFO: Task 2895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5ceb6b479dfdf0963282b8f112eadda with result from task 2895 parsl.dataflow.dflow DEBUG: Task 2895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/647a6e96-ddf3-46e9-bfd6-404a82359225_isrForCrosstalkSources_855391_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/647a6e96-ddf3-46e9-bfd6-404a82359225_isrForCrosstalkSources_855391_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:14 proc_lsst.multi:146 [INFO] found job 16964227 in provider checkpoint proc_lsst.multi INFO: found job 16964227 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4883b72aa04fcb7e2d919ae316d5eeff with result from task 3047 parsl.dataflow.dflow DEBUG: Task 3047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f8af0d8d-d344-4654-afbd-d4ad47259a9e_isrForCrosstalkSources_855428_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/f8af0d8d-d344-4654-afbd-d4ad47259a9e_isrForCrosstalkSources_855428_27.stderr parsl.dataflow.dflow INFO: Task 2862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3963d77f9e48a13f844566b2c00fe2db with result from task 2862 parsl.dataflow.dflow DEBUG: Task 2862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9f18aa49-7710-4427-b661-d3e67b662762_isrForCrosstalkSources_855396_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9f18aa49-7710-4427-b661-d3e67b662762_isrForCrosstalkSources_855396_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2983 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry d11dfdbc42a944131865cfb3f295e525 with result from task 2983 parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2983 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/410d8ba8-b803-4dbb-8bfd-c2b78a0dddf1_isrForCrosstalkSources_855437_51.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/410d8ba8-b803-4dbb-8bfd-c2b78a0dddf1_isrForCrosstalkSources_855437_51.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2971 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry ba4340abe471fd451f033bdae7d98a69 with result from task 2971 parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2971 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1e1e089d-ab1a-4ce4-bc91-077b2cbfe45a_isrForCrosstalkSources_855388_23.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1e1e089d-ab1a-4ce4-bc91-077b2cbfe45a_isrForCrosstalkSources_855388_23.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:14 proc_lsst.multi:146 [INFO] found job 16964228 in provider checkpoint proc_lsst.multi INFO: found job 16964228 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08fe447f905058c7a22b71b699caea06 with result from task 3002 parsl.dataflow.dflow DEBUG: Task 3002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e95c16f4-2502-40d9-bcc3-1bdda882c587_isrForCrosstalkSources_855451_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e95c16f4-2502-40d9-bcc3-1bdda882c587_isrForCrosstalkSources_855451_53.stderr parsl.dataflow.dflow INFO: Task 2922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd17d01bf41166e1410c4a6b59667571 with result from task 2922 parsl.dataflow.dflow DEBUG: Task 2922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ca3b1ed9-3a0f-4536-8c9c-02fbeca6a046_isrForCrosstalkSources_855411_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ca3b1ed9-3a0f-4536-8c9c-02fbeca6a046_isrForCrosstalkSources_855411_33.stderr parsl.dataflow.dflow INFO: Task 3103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9fa3918f644ac4899aa58b38269b98 with result from task 3103 parsl.dataflow.dflow DEBUG: Task 3103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afb05fae-b3ed-4295-966e-e706e543b9d2_isrForCrosstalkSources_855452_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/afb05fae-b3ed-4295-966e-e706e543b9d2_isrForCrosstalkSources_855452_32.stderr parsl.dataflow.dflow INFO: Task 3089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8f86122861c11d8e0a51279474e2db with result from task 3089 parsl.dataflow.dflow DEBUG: Task 3089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/93ff4fb1-251c-4e37-92fb-efa06e745c59_isrForCrosstalkSources_855435_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/93ff4fb1-251c-4e37-92fb-efa06e745c59_isrForCrosstalkSources_855435_25.stderr parsl.dataflow.dflow INFO: Task 2943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e94b6111863732f2d400afa0888f4bc with result from task 2943 parsl.dataflow.dflow DEBUG: Task 2943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5e688c1c-eb8f-4c26-8506-890275866973_isrForCrosstalkSources_855394_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/5e688c1c-eb8f-4c26-8506-890275866973_isrForCrosstalkSources_855394_49.stderr parsl.dataflow.dflow INFO: Task 2891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f10f7045df775d326efeff26c3798c4 with result from task 2891 parsl.dataflow.dflow DEBUG: Task 2891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e21f2282-c2eb-4396-89b1-eedfee0aac81_isrForCrosstalkSources_855414_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e21f2282-c2eb-4396-89b1-eedfee0aac81_isrForCrosstalkSources_855414_42.stderr parsl.dataflow.dflow INFO: Task 3014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db5c5ebf83fb6f2ed33e0473d09228c6 with result from task 3014 parsl.dataflow.dflow DEBUG: Task 3014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/79bb7e94-f181-423a-8aac-89e2a4cebac4_isrForCrosstalkSources_855388_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/79bb7e94-f181-423a-8aac-89e2a4cebac4_isrForCrosstalkSources_855388_4.stderr parsl.dataflow.dflow INFO: Task 3022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72e0642068784724ac890e2a06c21296 with result from task 3022 parsl.dataflow.dflow DEBUG: Task 3022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5d40fde9-54d7-4d92-be5e-cfa567d0602f_isrForCrosstalkSources_855400_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5d40fde9-54d7-4d92-be5e-cfa567d0602f_isrForCrosstalkSources_855400_32.stderr parsl.dataflow.dflow INFO: Task 3040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97a3bd7d921b596ef169d12143721c3a with result from task 3040 parsl.dataflow.dflow DEBUG: Task 3040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/779162ff-1f82-45f4-8865-538f03369d81_isrForCrosstalkSources_855404_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/779162ff-1f82-45f4-8865-538f03369d81_isrForCrosstalkSources_855404_34.stderr parsl.dataflow.dflow INFO: Task 2966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a97c37f5b5b65ec79a25bfcf742fa249 with result from task 2966 parsl.dataflow.dflow DEBUG: Task 2966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8298ac3a-4b01-4fee-90e8-30bddd974bd6_isrForCrosstalkSources_855386_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8298ac3a-4b01-4fee-90e8-30bddd974bd6_isrForCrosstalkSources_855386_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:14 proc_lsst.multi:146 [INFO] found job 16964229 in provider checkpoint proc_lsst.multi INFO: found job 16964229 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7d5a6fecb1f22a07b80e7294d5446c0 with result from task 3001 parsl.dataflow.dflow DEBUG: Task 3001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e65b7292-de32-4a54-8f9b-2101861179a7_isrForCrosstalkSources_855418_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e65b7292-de32-4a54-8f9b-2101861179a7_isrForCrosstalkSources_855418_46.stderr parsl.dataflow.dflow INFO: Task 3079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df9cd2f47032b88bf41b35b2ab4ebdcc with result from task 3079 parsl.dataflow.dflow DEBUG: Task 3079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9e4f6810-a8ee-41d9-bcf0-5d2fdd213f2a_isrForCrosstalkSources_855423_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/9e4f6810-a8ee-41d9-bcf0-5d2fdd213f2a_isrForCrosstalkSources_855423_32.stderr parsl.dataflow.dflow INFO: Task 2928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f035e9d09ce01d5c9743c9a33a581a3 with result from task 2928 parsl.dataflow.dflow DEBUG: Task 2928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78cfe66b-30a8-43e2-9174-121f2f31ab27_isrForCrosstalkSources_855433_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78cfe66b-30a8-43e2-9174-121f2f31ab27_isrForCrosstalkSources_855433_52.stderr parsl.dataflow.dflow INFO: Task 2975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26f719029d66513d9cdc05338ffefeca with result from task 2975 parsl.dataflow.dflow DEBUG: Task 2975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9ecb1418-23cf-4132-bbce-ba8f77abc1f4_isrForCrosstalkSources_855456_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9ecb1418-23cf-4132-bbce-ba8f77abc1f4_isrForCrosstalkSources_855456_17.stderr parsl.dataflow.dflow INFO: Task 2933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc00911ea837a209e580059c602bf1c4 with result from task 2933 parsl.dataflow.dflow DEBUG: Task 2933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf0f8caa-c4e9-4790-956d-b1911e7c0dca_isrForCrosstalkSources_855402_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf0f8caa-c4e9-4790-956d-b1911e7c0dca_isrForCrosstalkSources_855402_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:14 proc_lsst.multi:146 [INFO] found job 16964230 in provider checkpoint proc_lsst.multi INFO: found job 16964230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e0485c231c73366cecc1a70c4c25f82 with result from task 3151 parsl.dataflow.dflow DEBUG: Task 3151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d18e6b20-398d-44ca-a0a6-2f314f852980_isrForCrosstalkSources_855385_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d18e6b20-398d-44ca-a0a6-2f314f852980_isrForCrosstalkSources_855385_46.stderr parsl.dataflow.dflow INFO: Task 2974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c262ba0631f1d30df9a237484de20f5 with result from task 2974 parsl.dataflow.dflow DEBUG: Task 2974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6504951e-ec01-48af-8716-11ecc9d20617_isrForCrosstalkSources_855394_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6504951e-ec01-48af-8716-11ecc9d20617_isrForCrosstalkSources_855394_15.stderr parsl.dataflow.dflow INFO: Task 3021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e9c59af61053a400c0f64e139b47935 with result from task 3021 parsl.dataflow.dflow DEBUG: Task 3021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dc05e594-40e8-4b53-a411-53b3274671f9_isrForCrosstalkSources_855434_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/dc05e594-40e8-4b53-a411-53b3274671f9_isrForCrosstalkSources_855434_47.stderr parsl.dataflow.dflow INFO: Task 3032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d8fa9e97fc0df771838b0507a07710d with result from task 3032 parsl.dataflow.dflow DEBUG: Task 3032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3ec1bdeb-e03e-46b0-bdc6-0a28ae329796_isrForCrosstalkSources_855440_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3ec1bdeb-e03e-46b0-bdc6-0a28ae329796_isrForCrosstalkSources_855440_5.stderr parsl.dataflow.dflow INFO: Task 3093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad274cd1edeff8d10a795886c379bccd with result from task 3093 parsl.dataflow.dflow DEBUG: Task 3093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7dea6153-4a94-470a-bb03-68a9696015e6_isrForCrosstalkSources_855414_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7dea6153-4a94-470a-bb03-68a9696015e6_isrForCrosstalkSources_855414_25.stderr parsl.dataflow.dflow INFO: Task 2714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1aecf38f9d4099c827b0b3bff3ee643 with result from task 2714 parsl.dataflow.dflow DEBUG: Task 2714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fc2b0fbe-24d8-4680-802c-af69d8bf90f0_isrForCrosstalkSources_855440_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fc2b0fbe-24d8-4680-802c-af69d8bf90f0_isrForCrosstalkSources_855440_27.stderr parsl.dataflow.dflow INFO: Task 3013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b6ca0ae327f85e2ae61c15cf33a9e4 with result from task 3013 parsl.dataflow.dflow DEBUG: Task 3013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/19bb55c9-d851-4454-9010-62bfda3b2a0c_isrForCrosstalkSources_855412_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/19bb55c9-d851-4454-9010-62bfda3b2a0c_isrForCrosstalkSources_855412_15.stderr parsl.dataflow.dflow INFO: Task 2930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2935a72c9bc1fee36a53f0a3a0d7f8f with result from task 2930 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2930 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/328f7bab-0af5-4403-9cf3-5da21f02aa35_isrForCrosstalkSources_855439_19.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/328f7bab-0af5-4403-9cf3-5da21f02aa35_isrForCrosstalkSources_855439_19.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:15 proc_lsst.multi:146 [INFO] found job 16964231 in provider checkpoint proc_lsst.multi INFO: found job 16964231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 470ac77564bfdfa03612515dc8dfcb19 with result from task 2980 parsl.dataflow.dflow DEBUG: Task 2980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/49b81e15-1149-4fe0-aba4-bb33824f4caf_isrForCrosstalkSources_855384_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/49b81e15-1149-4fe0-aba4-bb33824f4caf_isrForCrosstalkSources_855384_14.stderr parsl.dataflow.dflow INFO: Task 3048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c31349704d747ed29e29340857a1b55 with result from task 3048 parsl.dataflow.dflow DEBUG: Task 3048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf472d30-9774-4c55-a8c4-024494836aa0_isrForCrosstalkSources_855402_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/cf472d30-9774-4c55-a8c4-024494836aa0_isrForCrosstalkSources_855402_19.stderr parsl.dataflow.dflow INFO: Task 3035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b76c808f1383d8406332262573efc9ad with result from task 3035 parsl.dataflow.dflow DEBUG: Task 3035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/183ae7fc-dec9-44bb-972b-8f2bfffda74e_isrForCrosstalkSources_855452_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/183ae7fc-dec9-44bb-972b-8f2bfffda74e_isrForCrosstalkSources_855452_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2991 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry d8bfc583a9c6b8363268a74ebd019b04 with result from task 2991 parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2991 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f726129a-d77b-430b-b6e6-5756945fb3a9_isrForCrosstalkSources_855448_32.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/f726129a-d77b-430b-b6e6-5756945fb3a9_isrForCrosstalkSources_855448_32.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:15 proc_lsst.multi:146 [INFO] found job 16964232 in provider checkpoint proc_lsst.multi INFO: found job 16964232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4493b6b99b909d6ecaf7ad88d662c746 with result from task 3181 parsl.dataflow.dflow DEBUG: Task 3181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/930cd220-b14a-4b44-85f5-0de7569e764c_isrForCrosstalkSources_855438_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/930cd220-b14a-4b44-85f5-0de7569e764c_isrForCrosstalkSources_855438_26.stderr parsl.dataflow.dflow INFO: Task 3092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db3e9e60ced56ca3fbb605bd92528630 with result from task 3092 parsl.dataflow.dflow DEBUG: Task 3092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1653b9c5-0c45-4f66-b779-3c240442d317_isrForCrosstalkSources_855388_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/1653b9c5-0c45-4f66-b779-3c240442d317_isrForCrosstalkSources_855388_34.stderr parsl.dataflow.dflow INFO: Task 3074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36eca2d99b24aa2b15a8da2bbd5dc523 with result from task 3074 parsl.dataflow.dflow DEBUG: Task 3074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f8cad71-3217-45c5-80b2-d16a60439fbd_isrForCrosstalkSources_855393_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6f8cad71-3217-45c5-80b2-d16a60439fbd_isrForCrosstalkSources_855393_8.stderr parsl.dataflow.dflow INFO: Task 2982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d19a841fb1b889298ef056dd9ea373 with result from task 2982 parsl.dataflow.dflow DEBUG: Task 2982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b87205ec-f445-43ff-8096-cbe6c5dc2ba4_isrForCrosstalkSources_855411_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b87205ec-f445-43ff-8096-cbe6c5dc2ba4_isrForCrosstalkSources_855411_39.stderr parsl.dataflow.dflow INFO: Task 2905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20027197983cc0d79b6a437d8d9d4990 with result from task 2905 parsl.dataflow.dflow DEBUG: Task 2905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aea0dc2-3386-4e39-b9dc-d675adeed7f5_isrForCrosstalkSources_855411_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aea0dc2-3386-4e39-b9dc-d675adeed7f5_isrForCrosstalkSources_855411_11.stderr parsl.dataflow.dflow INFO: Task 2979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5171e8ddba465d8e5667ecf9547fc79a with result from task 2979 parsl.dataflow.dflow DEBUG: Task 2979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/75c96385-80d4-47b9-9b86-0f653e7e8ddc_isrForCrosstalkSources_855426_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/75c96385-80d4-47b9-9b86-0f653e7e8ddc_isrForCrosstalkSources_855426_3.stderr parsl.dataflow.dflow INFO: Task 3128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af966cd63b89aa6c91a0433440381f31 with result from task 3128 parsl.dataflow.dflow DEBUG: Task 3128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24be9d09-bb79-4de9-97d7-a1d5f6506014_isrForCrosstalkSources_855379_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24be9d09-bb79-4de9-97d7-a1d5f6506014_isrForCrosstalkSources_855379_15.stderr parsl.dataflow.dflow INFO: Task 3195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57614aca7a0cf1ad7e3200ec177a928c with result from task 3195 parsl.dataflow.dflow DEBUG: Task 3195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1d32bb10-c2c2-40e8-91e3-0d4457064999_isrForCrosstalkSources_855437_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1d32bb10-c2c2-40e8-91e3-0d4457064999_isrForCrosstalkSources_855437_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:15 proc_lsst.multi:146 [INFO] found job 16964233 in provider checkpoint proc_lsst.multi INFO: found job 16964233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d890b72675432f016eaa4abfff528e44 with result from task 3147 parsl.dataflow.dflow DEBUG: Task 3147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/580a8ac3-ade0-4a3b-829e-4a38a2e4fe07_isrForCrosstalkSources_855403_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/580a8ac3-ade0-4a3b-829e-4a38a2e4fe07_isrForCrosstalkSources_855403_13.stderr parsl.dataflow.dflow INFO: Task 3122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522b518f7a3294055c0cb442db119cf8 with result from task 3122 parsl.dataflow.dflow DEBUG: Task 3122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7e7606ea-df41-44bf-b54e-87ccd26c2926_isrForCrosstalkSources_855401_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/7e7606ea-df41-44bf-b54e-87ccd26c2926_isrForCrosstalkSources_855401_27.stderr parsl.dataflow.dflow INFO: Task 3099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c75e701924bbf47e78a83ce8cb46f326 with result from task 3099 parsl.dataflow.dflow DEBUG: Task 3099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/2c946e58-0575-40f8-8b5e-b97086b046b8_isrForCrosstalkSources_855428_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/2c946e58-0575-40f8-8b5e-b97086b046b8_isrForCrosstalkSources_855428_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:16 proc_lsst.multi:146 [INFO] found job 16964234 in provider checkpoint proc_lsst.multi INFO: found job 16964234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25449e27ac5b46f4dfc2d329bbec33a7 with result from task 3085 parsl.dataflow.dflow DEBUG: Task 3085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fca48b90-8f7f-46df-a507-8110f93c16f6_isrForCrosstalkSources_855415_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/fca48b90-8f7f-46df-a507-8110f93c16f6_isrForCrosstalkSources_855415_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:16 proc_lsst.multi:146 [INFO] found job 16964235 in provider checkpoint proc_lsst.multi INFO: found job 16964235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1a670d0973a8fef8e72348e1f43dd5 with result from task 3091 parsl.dataflow.dflow DEBUG: Task 3091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0c4a138a-7943-4e98-abe0-9ae4e814d987_isrForCrosstalkSources_855388_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/0c4a138a-7943-4e98-abe0-9ae4e814d987_isrForCrosstalkSources_855388_15.stderr parsl.dataflow.dflow INFO: Task 3019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13166cd6d493476bd14d7b200ab0ce67 with result from task 3019 parsl.dataflow.dflow DEBUG: Task 3019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e253d5fd-3fc2-4bac-8c77-37c4c21c6f2b_isrForCrosstalkSources_855396_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e253d5fd-3fc2-4bac-8c77-37c4c21c6f2b_isrForCrosstalkSources_855396_34.stderr parsl.dataflow.dflow INFO: Task 3175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aba43590b6d5ae79cbd4aeec7a0ffc5d with result from task 3175 parsl.dataflow.dflow DEBUG: Task 3175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/fba092a2-1304-4942-a29b-6b0800ec0625_isrForCrosstalkSources_855405_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/fba092a2-1304-4942-a29b-6b0800ec0625_isrForCrosstalkSources_855405_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:16 proc_lsst.multi:146 [INFO] found job 16964236 in provider checkpoint proc_lsst.multi INFO: found job 16964236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8273fe20f6c4efccc87430eb5e29ef0a with result from task 3138 parsl.dataflow.dflow DEBUG: Task 3138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/309ada1d-cb8a-4cbe-8ff6-6ea00ef569a2_isrForCrosstalkSources_855442_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/309ada1d-cb8a-4cbe-8ff6-6ea00ef569a2_isrForCrosstalkSources_855442_20.stderr parsl.dataflow.dflow INFO: Task 2768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 388997b10e11dadceb7a06eefa9151e5 with result from task 2768 parsl.dataflow.dflow DEBUG: Task 2768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/92ff179d-a04c-43a5-802d-d777c1cba94c_isrForCrosstalkSources_855402_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/92ff179d-a04c-43a5-802d-d777c1cba94c_isrForCrosstalkSources_855402_45.stderr parsl.dataflow.dflow INFO: Task 3117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59d3e93974910e6d39ed2a4f7e648d29 with result from task 3117 parsl.dataflow.dflow DEBUG: Task 3117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d2a5541b-809c-4f2f-9ae0-9c918070555e_isrForCrosstalkSources_855442_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d2a5541b-809c-4f2f-9ae0-9c918070555e_isrForCrosstalkSources_855442_14.stderr parsl.dataflow.dflow INFO: Task 3018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8b06653e8aa91552f11d0f59d06fd1c with result from task 3018 parsl.dataflow.dflow DEBUG: Task 3018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/35196dc1-5402-438d-bbbe-66324897304b_isrForCrosstalkSources_855425_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/35196dc1-5402-438d-bbbe-66324897304b_isrForCrosstalkSources_855425_19.stderr parsl.dataflow.dflow INFO: Task 3044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dec58745516050297b434317c1cfd4de with result from task 3044 parsl.dataflow.dflow DEBUG: Task 3044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/20cdfae8-1767-4971-b5f5-d36a449781cc_isrForCrosstalkSources_855397_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/20cdfae8-1767-4971-b5f5-d36a449781cc_isrForCrosstalkSources_855397_53.stderr parsl.dataflow.dflow INFO: Task 2939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 132ec103eca99007b02f58562959300f with result from task 2939 parsl.dataflow.dflow DEBUG: Task 2939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/af4e2486-9ed7-4f5e-abbb-43589408cf88_isrForCrosstalkSources_855395_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/af4e2486-9ed7-4f5e-abbb-43589408cf88_isrForCrosstalkSources_855395_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:16 proc_lsst.multi:146 [INFO] found job 16964237 in provider checkpoint proc_lsst.multi INFO: found job 16964237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e90c3448a01776665a3b442ff1ce47d with result from task 2716 parsl.dataflow.dflow DEBUG: Task 2716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/646a5aba-7123-4da3-9424-65317f26d806_isrForCrosstalkSources_855459_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/646a5aba-7123-4da3-9424-65317f26d806_isrForCrosstalkSources_855459_56.stderr parsl.dataflow.dflow INFO: Task 3006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c6931f18b7fd5d2e7823f0581429dd6 with result from task 3006 parsl.dataflow.dflow DEBUG: Task 3006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9f48430c-a8e3-43f1-860b-d96f56822cec_isrForCrosstalkSources_855403_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9f48430c-a8e3-43f1-860b-d96f56822cec_isrForCrosstalkSources_855403_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:17 proc_lsst.multi:146 [INFO] found job 16964238 in provider checkpoint proc_lsst.multi INFO: found job 16964238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddfeb91b0b38d5b26ee7677d2150b28f with result from task 2948 parsl.dataflow.dflow DEBUG: Task 2948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8834c3b7-707b-4c8b-9370-0e3692ede66b_isrForCrosstalkSources_855446_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8834c3b7-707b-4c8b-9370-0e3692ede66b_isrForCrosstalkSources_855446_31.stderr parsl.dataflow.dflow INFO: Task 2803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33bbbce85dbe21bf8a536b5bba62381d with result from task 2803 parsl.dataflow.dflow DEBUG: Task 2803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cc51daa7-0ffd-40fa-ba55-fccaca423f7b_isrForCrosstalkSources_855446_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cc51daa7-0ffd-40fa-ba55-fccaca423f7b_isrForCrosstalkSources_855446_43.stderr parsl.dataflow.dflow INFO: Task 3087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 951431e4a1480081b1e95245a47193ba with result from task 3087 parsl.dataflow.dflow DEBUG: Task 3087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6728434b-2f46-4b73-bb5b-0325cb56f27f_isrForCrosstalkSources_855420_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6728434b-2f46-4b73-bb5b-0325cb56f27f_isrForCrosstalkSources_855420_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:17 proc_lsst.multi:146 [INFO] found job 16964239 in provider checkpoint proc_lsst.multi INFO: found job 16964239 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:17 proc_lsst.multi:146 [INFO] found job 16964242 in provider checkpoint proc_lsst.multi INFO: found job 16964242 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894db393182c96af5510c4d4c945ee15 with result from task 3075 parsl.dataflow.dflow DEBUG: Task 3075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5c33899f-a189-4c88-b193-7c678aee69b1_isrForCrosstalkSources_855445_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5c33899f-a189-4c88-b193-7c678aee69b1_isrForCrosstalkSources_855445_56.stderr parsl.dataflow.dflow INFO: Task 3063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f454f0c30117585e7c5ca6109faf1c1 with result from task 3063 parsl.dataflow.dflow DEBUG: Task 3063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e55978a3-318d-40da-8ff1-00ea563446b8_isrForCrosstalkSources_855386_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/e55978a3-318d-40da-8ff1-00ea563446b8_isrForCrosstalkSources_855386_4.stderr parsl.dataflow.dflow INFO: Task 2867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d01ff256d845dd632ceb43833a148bea with result from task 2867 parsl.dataflow.dflow DEBUG: Task 2867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d16b6c0-526e-4257-9912-7a3cb5ec35ff_isrForCrosstalkSources_855410_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d16b6c0-526e-4257-9912-7a3cb5ec35ff_isrForCrosstalkSources_855410_43.stderr parsl.dataflow.dflow INFO: Task 2778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87e71021e298144ddd331031d4556ec6 with result from task 2778 parsl.dataflow.dflow DEBUG: Task 2778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f435e517-22f4-4c02-a2c0-c4f95f0ee67c_isrForCrosstalkSources_855390_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f435e517-22f4-4c02-a2c0-c4f95f0ee67c_isrForCrosstalkSources_855390_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:17 proc_lsst.multi:146 [INFO] found job 16964243 in provider checkpoint proc_lsst.multi INFO: found job 16964243 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9faa2d46a80dc4758c59045fd06373dc with result from task 2904 parsl.dataflow.dflow DEBUG: Task 2904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/6b11495c-6372-48e4-805f-a0364c108295_isrForCrosstalkSources_855442_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/6b11495c-6372-48e4-805f-a0364c108295_isrForCrosstalkSources_855442_39.stderr parsl.dataflow.dflow INFO: Task 3020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5984e63d1c791bacd7e185fd024c01d with result from task 3020 parsl.dataflow.dflow DEBUG: Task 3020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5ee77ab5-9f6b-4c30-bb51-2b96045a6b2f_isrForCrosstalkSources_855458_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5ee77ab5-9f6b-4c30-bb51-2b96045a6b2f_isrForCrosstalkSources_855458_33.stderr parsl.dataflow.dflow INFO: Task 3143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ebe8294319044c7c204e9d332936c61 with result from task 3143 parsl.dataflow.dflow DEBUG: Task 3143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a84d9ac7-2bdd-43e7-b30a-afc21972307f_isrForCrosstalkSources_855421_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a84d9ac7-2bdd-43e7-b30a-afc21972307f_isrForCrosstalkSources_855421_15.stderr parsl.dataflow.dflow INFO: Task 3055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecfeb9ee6a9bd4e604fc76be21733efc with result from task 3055 parsl.dataflow.dflow DEBUG: Task 3055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/44cf7bce-8780-4f5e-b40c-44659469529e_isrForCrosstalkSources_855451_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/44cf7bce-8780-4f5e-b40c-44659469529e_isrForCrosstalkSources_855451_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:18 proc_lsst.multi:146 [INFO] found job 16964244 in provider checkpoint proc_lsst.multi INFO: found job 16964244 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f7835e69b1e086b1070fb297702019e with result from task 3158 parsl.dataflow.dflow DEBUG: Task 3158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99b38168-9152-4bad-ba1f-35d2c6c6f297_isrForCrosstalkSources_855406_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99b38168-9152-4bad-ba1f-35d2c6c6f297_isrForCrosstalkSources_855406_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:18 proc_lsst.multi:146 [INFO] found job 16964245 in provider checkpoint proc_lsst.multi INFO: found job 16964245 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f959c0f3e41659a83a70a1f9045e2c2a with result from task 2767 parsl.dataflow.dflow DEBUG: Task 2767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/24080bc9-b19d-4df3-8a45-b6e073c956ed_isrForCrosstalkSources_855446_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/24080bc9-b19d-4df3-8a45-b6e073c956ed_isrForCrosstalkSources_855446_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:18 proc_lsst.multi:146 [INFO] found job 16964246 in provider checkpoint proc_lsst.multi INFO: found job 16964246 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5240e2478a29ee9104bda4059a93c531 with result from task 3000 parsl.dataflow.dflow DEBUG: Task 3000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/598a075d-6945-43cf-9c42-c366b12eadec_isrForCrosstalkSources_855429_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/598a075d-6945-43cf-9c42-c366b12eadec_isrForCrosstalkSources_855429_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:18 proc_lsst.multi:146 [INFO] found job 16964247 in provider checkpoint proc_lsst.multi INFO: found job 16964247 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d38a94df68f9c20877ec8d86ca7b42 with result from task 2990 parsl.dataflow.dflow DEBUG: Task 2990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/17c56097-a798-48d1-9d30-6aedff8a85af_isrForCrosstalkSources_855428_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/17c56097-a798-48d1-9d30-6aedff8a85af_isrForCrosstalkSources_855428_57.stderr parsl.dataflow.dflow INFO: Task 3168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55416ab362e63ff50c46524043b4ca78 with result from task 3168 parsl.dataflow.dflow DEBUG: Task 3168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ea38f6d2-3356-4fae-bd1e-b3db26edd2a8_isrForCrosstalkSources_855441_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/ea38f6d2-3356-4fae-bd1e-b3db26edd2a8_isrForCrosstalkSources_855441_21.stderr parsl.dataflow.dflow INFO: Task 3053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b482056d6c386403c12e6c8881caec3c with result from task 3053 parsl.dataflow.dflow DEBUG: Task 3053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c562df56-d135-41bd-bff1-0c046226bf7a_isrForCrosstalkSources_855450_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c562df56-d135-41bd-bff1-0c046226bf7a_isrForCrosstalkSources_855450_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:19 proc_lsst.multi:146 [INFO] found job 16964248 in provider checkpoint proc_lsst.multi INFO: found job 16964248 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32365d0cacfa6d0e0a9b13c755815626 with result from task 2883 parsl.dataflow.dflow DEBUG: Task 2883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e743353d-eca1-4960-af85-3ac62de17e02_isrForCrosstalkSources_855400_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e743353d-eca1-4960-af85-3ac62de17e02_isrForCrosstalkSources_855400_36.stderr parsl.dataflow.dflow INFO: Task 3098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f87b66d289f87120745e9e546280b330 with result from task 3098 parsl.dataflow.dflow DEBUG: Task 3098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e2a6f6a-3035-4551-879d-5c8eee9f23f2_isrForCrosstalkSources_855380_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/0e2a6f6a-3035-4551-879d-5c8eee9f23f2_isrForCrosstalkSources_855380_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:19 proc_lsst.multi:146 [INFO] found job 16964249 in provider checkpoint proc_lsst.multi INFO: found job 16964249 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3da364826d66f27a9c5c37151355a4b with result from task 3165 parsl.dataflow.dflow DEBUG: Task 3165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/17861c5a-cdf5-4817-98f6-4e48c0168f80_isrForCrosstalkSources_855445_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/17861c5a-cdf5-4817-98f6-4e48c0168f80_isrForCrosstalkSources_855445_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:19 proc_lsst.multi:146 [INFO] found job 16964250 in provider checkpoint proc_lsst.multi INFO: found job 16964250 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0169be1623424f1e8596c9cbc5e0103 with result from task 3015 parsl.dataflow.dflow DEBUG: Task 3015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/993efff5-3c71-44fd-931a-5ccc92c524a8_isrForCrosstalkSources_855451_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/993efff5-3c71-44fd-931a-5ccc92c524a8_isrForCrosstalkSources_855451_46.stderr parsl.dataflow.dflow INFO: Task 2726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b84845d42faf65c13c2a2aa11ad475b with result from task 2726 parsl.dataflow.dflow DEBUG: Task 2726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4f5adbc2-0fe3-46a1-ad5a-18b16ddc105b_isrForCrosstalkSources_855379_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/4f5adbc2-0fe3-46a1-ad5a-18b16ddc105b_isrForCrosstalkSources_855379_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:20 proc_lsst.multi:146 [INFO] found job 16964251 in provider checkpoint proc_lsst.multi INFO: found job 16964251 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4d6be4bb60054278d854ff2e4093fa with result from task 3232 parsl.dataflow.dflow DEBUG: Task 3232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f67af9af-f81e-4531-9e68-62a4bfdaa865_isrForCrosstalkSources_855384_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f67af9af-f81e-4531-9e68-62a4bfdaa865_isrForCrosstalkSources_855384_10.stderr parsl.dataflow.dflow INFO: Task 3012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f74cf77aa2d8f89f56369a6e6c4b4b6 with result from task 3012 parsl.dataflow.dflow DEBUG: Task 3012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/341e1c34-b787-4806-ad65-e070cc908c73_isrForCrosstalkSources_855459_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/341e1c34-b787-4806-ad65-e070cc908c73_isrForCrosstalkSources_855459_59.stderr parsl.dataflow.dflow INFO: Task 2881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 309b44b4a2c0dce3b53f0d7ab0bd992f with result from task 2881 parsl.dataflow.dflow DEBUG: Task 2881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dc4c5788-1d84-4304-875e-9bffe02430da_isrForCrosstalkSources_855446_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/dc4c5788-1d84-4304-875e-9bffe02430da_isrForCrosstalkSources_855446_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:20 proc_lsst.multi:146 [INFO] found job 16964252 in provider checkpoint proc_lsst.multi INFO: found job 16964252 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c23fee7ca3368abad57a31c7b2d07b with result from task 2921 parsl.dataflow.dflow DEBUG: Task 2921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e55cfff1-1657-402a-9a19-007feeb4489c_isrForCrosstalkSources_855445_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e55cfff1-1657-402a-9a19-007feeb4489c_isrForCrosstalkSources_855445_41.stderr parsl.dataflow.dflow INFO: Task 2949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e64f9a121654a2d02a7696692132df5 with result from task 2949 parsl.dataflow.dflow DEBUG: Task 2949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ebb3823-6de7-483c-ab49-1af9ba3488ec_isrForCrosstalkSources_855442_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ebb3823-6de7-483c-ab49-1af9ba3488ec_isrForCrosstalkSources_855442_9.stderr parsl.dataflow.dflow INFO: Task 3106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bac0f428644d3a2f41e564438d01c4b6 with result from task 3106 parsl.dataflow.dflow DEBUG: Task 3106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/97fb034e-fd53-4cca-b5cf-cb277c04e854_isrForCrosstalkSources_855397_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/97fb034e-fd53-4cca-b5cf-cb277c04e854_isrForCrosstalkSources_855397_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:20 proc_lsst.multi:146 [INFO] found job 16964253 in provider checkpoint proc_lsst.multi INFO: found job 16964253 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfcc5d8ffd75e8de553f80140e614fcc with result from task 2898 parsl.dataflow.dflow DEBUG: Task 2898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/380aa4aa-b58b-4a27-b910-4db0d80e7860_isrForCrosstalkSources_855424_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/380aa4aa-b58b-4a27-b910-4db0d80e7860_isrForCrosstalkSources_855424_62.stderr parsl.dataflow.dflow INFO: Task 2765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57543f5f195b3711b112f59271aa38ca with result from task 2765 parsl.dataflow.dflow DEBUG: Task 2765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/86ebcf46-0d78-47e0-88fc-ca12ba2d8755_isrForCrosstalkSources_855449_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/86ebcf46-0d78-47e0-88fc-ca12ba2d8755_isrForCrosstalkSources_855449_1.stderr parsl.dataflow.dflow INFO: Task 2916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad474e0d8747f5e330dde9a3235a7bcf with result from task 2916 parsl.dataflow.dflow DEBUG: Task 2916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fd3e1c6a-5489-4d23-a7ec-8a907b8d60b8_isrForCrosstalkSources_855458_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fd3e1c6a-5489-4d23-a7ec-8a907b8d60b8_isrForCrosstalkSources_855458_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:20 proc_lsst.multi:146 [INFO] found job 16964254 in provider checkpoint proc_lsst.multi INFO: found job 16964254 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1dc8f4ef89710777be63421da6d0291 with result from task 3237 parsl.dataflow.dflow DEBUG: Task 3237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/05f2a6e2-c175-4e89-a8bd-4153c69fddfb_isrForCrosstalkSources_855435_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/05f2a6e2-c175-4e89-a8bd-4153c69fddfb_isrForCrosstalkSources_855435_24.stderr parsl.dataflow.dflow INFO: Task 3152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa826b04e48d847442b55870a9ee07f5 with result from task 3152 parsl.dataflow.dflow DEBUG: Task 3152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b6575efc-4324-4b1b-b3c9-9f1d98f5f705_isrForCrosstalkSources_855439_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b6575efc-4324-4b1b-b3c9-9f1d98f5f705_isrForCrosstalkSources_855439_11.stderr parsl.dataflow.dflow INFO: Task 2776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112932699dac0823ba07bff21793c0cb with result from task 2776 parsl.dataflow.dflow DEBUG: Task 2776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/86a6564a-1e32-4f00-80bc-6453d4654ea0_isrForCrosstalkSources_855390_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/86a6564a-1e32-4f00-80bc-6453d4654ea0_isrForCrosstalkSources_855390_2.stderr parsl.dataflow.dflow INFO: Task 3176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a57da48e7ebda9131e6b116c880d2b29 with result from task 3176 parsl.dataflow.dflow DEBUG: Task 3176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed53f06a-d00d-490a-9d85-f9a23f7c87e9_isrForCrosstalkSources_855455_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed53f06a-d00d-490a-9d85-f9a23f7c87e9_isrForCrosstalkSources_855455_62.stderr parsl.dataflow.dflow INFO: Task 2902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62cf501432cc95b3c0a56eba67a4af9d with result from task 2902 parsl.dataflow.dflow DEBUG: Task 2902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d272b08f-60cb-4fec-b92c-f8d289134a1e_isrForCrosstalkSources_855428_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/d272b08f-60cb-4fec-b92c-f8d289134a1e_isrForCrosstalkSources_855428_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:21 proc_lsst.multi:146 [INFO] found job 16964255 in provider checkpoint proc_lsst.multi INFO: found job 16964255 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 949084c8456dfa990a2bd36b46732ac5 with result from task 3111 parsl.dataflow.dflow DEBUG: Task 3111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/151441f9-1dbc-4861-bf9c-d03334088bce_isrForCrosstalkSources_855446_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/151441f9-1dbc-4861-bf9c-d03334088bce_isrForCrosstalkSources_855446_2.stderr parsl.dataflow.dflow INFO: Task 2987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb37dc46ce7e1ff4691046dada28f68 with result from task 2987 parsl.dataflow.dflow DEBUG: Task 2987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/88082731-6430-4387-85af-8dc36b53afbd_isrForCrosstalkSources_855432_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/88082731-6430-4387-85af-8dc36b53afbd_isrForCrosstalkSources_855432_54.stderr parsl.dataflow.dflow INFO: Task 2996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db9ba6fdb6d1d174edcb3cbd30d6e95 with result from task 2996 parsl.dataflow.dflow DEBUG: Task 2996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a1aa9702-1190-4398-8263-c680d5adaf7b_isrForCrosstalkSources_855404_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a1aa9702-1190-4398-8263-c680d5adaf7b_isrForCrosstalkSources_855404_37.stderr parsl.dataflow.dflow INFO: Task 3130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28d165db7dfb92797753d17bfe07541f with result from task 3130 parsl.dataflow.dflow DEBUG: Task 3130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a249783e-da8f-4c02-b5a6-bc8f7ee3ef31_isrForCrosstalkSources_855395_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a249783e-da8f-4c02-b5a6-bc8f7ee3ef31_isrForCrosstalkSources_855395_61.stderr parsl.dataflow.dflow INFO: Task 2926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4335e4775fdfcf29622be9fd468d67bc with result from task 2926 parsl.dataflow.dflow DEBUG: Task 2926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c5273657-e651-4517-8c76-6735290b2160_isrForCrosstalkSources_855427_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c5273657-e651-4517-8c76-6735290b2160_isrForCrosstalkSources_855427_59.stderr parsl.dataflow.dflow INFO: Task 3068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b61d80b3cc27513588f3447dff54b380 with result from task 3068 parsl.dataflow.dflow DEBUG: Task 3068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/84d17b0a-5198-4c5e-8b78-48d3f6946ad4_isrForCrosstalkSources_855387_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/84d17b0a-5198-4c5e-8b78-48d3f6946ad4_isrForCrosstalkSources_855387_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:21 proc_lsst.multi:146 [INFO] found job 16964256 in provider checkpoint proc_lsst.multi INFO: found job 16964256 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e47eae6c2cd72b45415429fc6ee2c48 with result from task 2963 parsl.dataflow.dflow DEBUG: Task 2963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/c99ba92a-ab84-4634-94f0-4368472f9975_isrForCrosstalkSources_855453_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/c99ba92a-ab84-4634-94f0-4368472f9975_isrForCrosstalkSources_855453_10.stderr parsl.dataflow.dflow INFO: Task 2892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e465211302cafe8d6b3fcbdb2ede36 with result from task 2892 parsl.dataflow.dflow DEBUG: Task 2892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/d9727d62-4663-409c-81a5-e9069608445c_isrForCrosstalkSources_855424_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/d9727d62-4663-409c-81a5-e9069608445c_isrForCrosstalkSources_855424_43.stderr parsl.dataflow.dflow INFO: Task 2805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee92425b7ee2a342fe7c8ec49c70b13d with result from task 2805 parsl.dataflow.dflow DEBUG: Task 2805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/37a635bb-86d7-449e-8ffa-04523940a286_isrForCrosstalkSources_855413_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/37a635bb-86d7-449e-8ffa-04523940a286_isrForCrosstalkSources_855413_48.stderr parsl.dataflow.dflow INFO: Task 2951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14512675b6ef6f1b75b154a3cde2fd65 with result from task 2951 parsl.dataflow.dflow DEBUG: Task 2951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0a6841e4-e8c2-426d-9694-4a5f4b7e378c_isrForCrosstalkSources_855425_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/0a6841e4-e8c2-426d-9694-4a5f4b7e378c_isrForCrosstalkSources_855425_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:21 proc_lsst.multi:146 [INFO] found job 16964257 in provider checkpoint proc_lsst.multi INFO: found job 16964257 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a238c164c41fecd1408c57ee25f9353 with result from task 2772 parsl.dataflow.dflow DEBUG: Task 2772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/92b38801-16fd-4dcc-b238-2a6647f0d82a_isrForCrosstalkSources_855388_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/92b38801-16fd-4dcc-b238-2a6647f0d82a_isrForCrosstalkSources_855388_2.stderr parsl.dataflow.dflow INFO: Task 2952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry decc5c9fc4024026dc74922084c658b8 with result from task 2952 parsl.dataflow.dflow DEBUG: Task 2952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/779cfd83-f8f1-40b2-8632-3ef3813ec54f_isrForCrosstalkSources_855391_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/779cfd83-f8f1-40b2-8632-3ef3813ec54f_isrForCrosstalkSources_855391_12.stderr parsl.dataflow.dflow INFO: Task 2729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68e1a9f0a5955c5a813089198b625258 with result from task 2729 parsl.dataflow.dflow DEBUG: Task 2729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/07750463-170b-415e-bf48-119061fe3a91_isrForCrosstalkSources_855444_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/07750463-170b-415e-bf48-119061fe3a91_isrForCrosstalkSources_855444_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 3129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea9f77e4232844ffc094eb23115c167d with result from task 3129 parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:21 proc_lsst.multi:146 [INFO] found job 16964258 in provider checkpoint proc_lsst.multi INFO: found job 16964258 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 3129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a66678d3-2bc9-4c19-9d94-ccd592876e75_isrForCrosstalkSources_855449_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/a66678d3-2bc9-4c19-9d94-ccd592876e75_isrForCrosstalkSources_855449_42.stderr parsl.dataflow.dflow INFO: Task 3124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8debda573090c56d5df1ff422b38102a with result from task 3124 parsl.dataflow.dflow DEBUG: Task 3124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a80334c6-4aba-4587-b0eb-7312d09322f8_isrForCrosstalkSources_855406_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a80334c6-4aba-4587-b0eb-7312d09322f8_isrForCrosstalkSources_855406_4.stderr parsl.dataflow.dflow INFO: Task 3155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 504e5e158f31094f0b5e9db73daa3249 with result from task 3155 parsl.dataflow.dflow DEBUG: Task 3155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f198a668-8569-441d-80d9-cc9b2edfaf00_isrForCrosstalkSources_855423_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f198a668-8569-441d-80d9-cc9b2edfaf00_isrForCrosstalkSources_855423_33.stderr parsl.dataflow.dflow INFO: Task 3119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70e8c4507dabe6b1e207e8087c0321ff with result from task 3119 parsl.dataflow.dflow DEBUG: Task 3119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/830555ed-471b-4190-8288-7469b58c4281_isrForCrosstalkSources_855419_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/830555ed-471b-4190-8288-7469b58c4281_isrForCrosstalkSources_855419_53.stderr parsl.dataflow.dflow INFO: Task 3011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56098997cadf41d4b08780d6fedc4b15 with result from task 3011 parsl.dataflow.dflow DEBUG: Task 3011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/15dfe727-2d57-47fd-b18a-d3f3421c8cc1_isrForCrosstalkSources_855397_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/15dfe727-2d57-47fd-b18a-d3f3421c8cc1_isrForCrosstalkSources_855397_60.stderr parsl.dataflow.dflow INFO: Task 3094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4d97aec9a5a989c37d6f24d5e70a6d with result from task 3094 parsl.dataflow.dflow DEBUG: Task 3094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ee233356-fabe-4df0-8587-63fbfc8d5e9e_isrForCrosstalkSources_855405_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ee233356-fabe-4df0-8587-63fbfc8d5e9e_isrForCrosstalkSources_855405_6.stderr parsl.dataflow.dflow INFO: Task 2937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bafd1fa5e4a95ff998db3234a085d45c with result from task 2937 parsl.dataflow.dflow DEBUG: Task 2937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6ec7b5fa-29ae-4eca-a098-3d4e7ab05433_isrForCrosstalkSources_855402_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/6ec7b5fa-29ae-4eca-a098-3d4e7ab05433_isrForCrosstalkSources_855402_41.stderr parsl.dataflow.dflow INFO: Task 3046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1434c3cfb5eeab0482ed58e7c62bb04 with result from task 3046 parsl.dataflow.dflow DEBUG: Task 3046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/9d1cf79e-cb91-4a8b-9f55-7f42a0a278e6_isrForCrosstalkSources_855443_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/9d1cf79e-cb91-4a8b-9f55-7f42a0a278e6_isrForCrosstalkSources_855443_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:22 proc_lsst.multi:146 [INFO] found job 16964259 in provider checkpoint proc_lsst.multi INFO: found job 16964259 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a366ed1a0fe02ac68668f72aa3f16be4 with result from task 2724 parsl.dataflow.dflow DEBUG: Task 2724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ddfe1634-c447-48f3-ad25-fe9b17a6fd9f_isrForCrosstalkSources_855398_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ddfe1634-c447-48f3-ad25-fe9b17a6fd9f_isrForCrosstalkSources_855398_40.stderr parsl.dataflow.dflow INFO: Task 2970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b193b2c7300e18df7622bdec009fd1 with result from task 2970 parsl.dataflow.dflow DEBUG: Task 2970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4770957c-70f2-459e-8279-20b97fc09e81_isrForCrosstalkSources_855409_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4770957c-70f2-459e-8279-20b97fc09e81_isrForCrosstalkSources_855409_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:22 proc_lsst.multi:146 [INFO] found job 16964260 in provider checkpoint proc_lsst.multi INFO: found job 16964260 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3594bebfd26e08c3089c5411779d6e8 with result from task 3247 parsl.dataflow.dflow DEBUG: Task 3247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d09f6af0-e994-446f-8c06-4969ed03af12_isrForCrosstalkSources_855417_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d09f6af0-e994-446f-8c06-4969ed03af12_isrForCrosstalkSources_855417_21.stderr parsl.dataflow.dflow INFO: Task 3193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ae88617a948725cb5838519405fa8c with result from task 3193 parsl.dataflow.dflow DEBUG: Task 3193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9487032c-d180-4295-8afb-6656cedfbc0d_isrForCrosstalkSources_855456_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9487032c-d180-4295-8afb-6656cedfbc0d_isrForCrosstalkSources_855456_22.stderr parsl.dataflow.dflow INFO: Task 3052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd6a707cac2f394d0a496a314c8aacc5 with result from task 3052 parsl.dataflow.dflow DEBUG: Task 3052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/757d2580-ddbf-424d-b5ae-58398da9bb07_isrForCrosstalkSources_855440_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/757d2580-ddbf-424d-b5ae-58398da9bb07_isrForCrosstalkSources_855440_36.stderr parsl.dataflow.dflow INFO: Task 3010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42fbbe474386b2c0bcfb147b3ed2b4db with result from task 3010 parsl.dataflow.dflow DEBUG: Task 3010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/342966e1-c1b1-4f44-8976-c0468ed9d280_isrForCrosstalkSources_855457_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/342966e1-c1b1-4f44-8976-c0468ed9d280_isrForCrosstalkSources_855457_6.stderr parsl.dataflow.dflow INFO: Task 3045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f22558f71b186273d766509a8c5ab51 with result from task 3045 parsl.dataflow.dflow DEBUG: Task 3045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a47e2378-1c64-4e59-a553-ad57df705a72_isrForCrosstalkSources_855432_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/a47e2378-1c64-4e59-a553-ad57df705a72_isrForCrosstalkSources_855432_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:22 proc_lsst.multi:146 [INFO] found job 16964261 in provider checkpoint proc_lsst.multi INFO: found job 16964261 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 785f181f8bacda485d34aec1fa56fba9 with result from task 2915 parsl.dataflow.dflow DEBUG: Task 2915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6a38255e-8ac8-456c-a75f-a3d1bbc41e92_isrForCrosstalkSources_855412_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6a38255e-8ac8-456c-a75f-a3d1bbc41e92_isrForCrosstalkSources_855412_12.stderr parsl.dataflow.dflow INFO: Task 2960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce179678849ad4f462f1d341f1089b5b with result from task 2960 parsl.dataflow.dflow DEBUG: Task 2960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc62d57c-94ca-44ba-8a75-274d2be52533_isrForCrosstalkSources_855421_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/dc62d57c-94ca-44ba-8a75-274d2be52533_isrForCrosstalkSources_855421_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:22 proc_lsst.multi:146 [INFO] found job 16964262 in provider checkpoint proc_lsst.multi INFO: found job 16964262 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1682c115e558b1e42a6000309a286eb4 with result from task 3031 parsl.dataflow.dflow DEBUG: Task 3031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f1f80d01-9acd-4169-bf75-58fcc53fc19a_isrForCrosstalkSources_855429_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f1f80d01-9acd-4169-bf75-58fcc53fc19a_isrForCrosstalkSources_855429_9.stderr parsl.dataflow.dflow INFO: Task 3224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a9e516118077d17e21dd7af82a289b7 with result from task 3224 parsl.dataflow.dflow DEBUG: Task 3224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b199ca80-65f3-4dfa-986d-f4fb4f9061ea_isrForCrosstalkSources_855424_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b199ca80-65f3-4dfa-986d-f4fb4f9061ea_isrForCrosstalkSources_855424_53.stderr parsl.dataflow.dflow INFO: Task 3136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aad565b692a07f23cfc6841f94c5c455 with result from task 3136 parsl.dataflow.dflow DEBUG: Task 3136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5ecaad0c-6375-421e-9391-23678addbe86_isrForCrosstalkSources_855430_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5ecaad0c-6375-421e-9391-23678addbe86_isrForCrosstalkSources_855430_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:23 proc_lsst.multi:146 [INFO] found job 16964263 in provider checkpoint proc_lsst.multi INFO: found job 16964263 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2cb116fec73d0bba2efbf6c9217e4af with result from task 2720 parsl.dataflow.dflow DEBUG: Task 2720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab94639c-3e70-4999-b338-a635a4807858_isrForCrosstalkSources_855420_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ab94639c-3e70-4999-b338-a635a4807858_isrForCrosstalkSources_855420_37.stderr parsl.dataflow.dflow INFO: Task 3038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a1b94e2dc9d198cbb7fe1d2dfb99842 with result from task 3038 parsl.dataflow.dflow DEBUG: Task 3038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9b5291b9-dcd0-4016-befd-b99d76434c5e_isrForCrosstalkSources_855395_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/9b5291b9-dcd0-4016-befd-b99d76434c5e_isrForCrosstalkSources_855395_2.stderr parsl.dataflow.dflow INFO: Task 3126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba02b653a81837985badff31956fc3e0 with result from task 3126 parsl.dataflow.dflow DEBUG: Task 3126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8c400940-51f8-48be-98b8-5576111a894f_isrForCrosstalkSources_855456_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8c400940-51f8-48be-98b8-5576111a894f_isrForCrosstalkSources_855456_62.stderr parsl.dataflow.dflow INFO: Task 3219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e044e95019ab39a3ac845039da7ba93e with result from task 3219 parsl.dataflow.dflow DEBUG: Task 3219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/b2a6fa28-2ea7-4397-b2b9-74be16c7b4e3_isrForCrosstalkSources_855418_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/b2a6fa28-2ea7-4397-b2b9-74be16c7b4e3_isrForCrosstalkSources_855418_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:23 proc_lsst.multi:146 [INFO] found job 16964264 in provider checkpoint proc_lsst.multi INFO: found job 16964264 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad328c45d33726a542d0deb99e7071dd with result from task 2981 parsl.dataflow.dflow DEBUG: Task 2981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/31249e03-035f-4469-a977-6d1b7579a871_isrForCrosstalkSources_855435_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/31249e03-035f-4469-a977-6d1b7579a871_isrForCrosstalkSources_855435_23.stderr parsl.dataflow.dflow INFO: Task 3120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8606877ae7d9540231f3d20281ac751 with result from task 3120 parsl.dataflow.dflow DEBUG: Task 3120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/778ea8ca-3f76-4c4c-b8d6-f13151176ca6_isrForCrosstalkSources_855418_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/778ea8ca-3f76-4c4c-b8d6-f13151176ca6_isrForCrosstalkSources_855418_12.stderr parsl.dataflow.dflow INFO: Task 2886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 765f80ab6fa032f3367849e19788a1ec with result from task 2886 parsl.dataflow.dflow DEBUG: Task 2886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/daf3188a-915d-4dca-b572-f924e47d5ac3_isrForCrosstalkSources_855449_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/daf3188a-915d-4dca-b572-f924e47d5ac3_isrForCrosstalkSources_855449_50.stderr parsl.dataflow.dflow INFO: Task 3174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a502cf0244c97b3a3d2ae65e3c407b3 with result from task 3174 parsl.dataflow.dflow DEBUG: Task 3174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/54add0ca-0db0-49fb-b3d3-e346bef50085_isrForCrosstalkSources_855391_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/54add0ca-0db0-49fb-b3d3-e346bef50085_isrForCrosstalkSources_855391_60.stderr parsl.dataflow.dflow INFO: Task 3221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c213045efc5684dbe39ddaa3314533d with result from task 3221 parsl.dataflow.dflow DEBUG: Task 3221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bc32e6f3-66a7-49de-bcbe-ec0532ef82b5_isrForCrosstalkSources_855405_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bc32e6f3-66a7-49de-bcbe-ec0532ef82b5_isrForCrosstalkSources_855405_41.stderr parsl.dataflow.dflow INFO: Task 3080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51e2487cc4889253f2e32fe8bb7691a4 with result from task 3080 parsl.dataflow.dflow DEBUG: Task 3080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/81c2782a-3b54-4a14-94ea-73235adf7d5f_isrForCrosstalkSources_855412_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/81c2782a-3b54-4a14-94ea-73235adf7d5f_isrForCrosstalkSources_855412_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:23 proc_lsst.multi:146 [INFO] found job 16964265 in provider checkpoint proc_lsst.multi INFO: found job 16964265 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f895e32be9c0782b0510fb1f7f802c with result from task 3188 parsl.dataflow.dflow DEBUG: Task 3188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/63ea6ae9-ed73-4d08-b28e-49a8dd0a685a_isrForCrosstalkSources_855433_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/63ea6ae9-ed73-4d08-b28e-49a8dd0a685a_isrForCrosstalkSources_855433_44.stderr parsl.dataflow.dflow INFO: Task 3157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d196f0eb2ffb6b41c32343abc297d85e with result from task 3157 parsl.dataflow.dflow DEBUG: Task 3157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/445dcbf1-1a04-4ee0-88a4-bb09d6ee5916_isrForCrosstalkSources_855382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/445dcbf1-1a04-4ee0-88a4-bb09d6ee5916_isrForCrosstalkSources_855382_2.stderr parsl.dataflow.dflow INFO: Task 3156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d8a3f79a77cf18d0669a491b857f74 with result from task 3156 parsl.dataflow.dflow DEBUG: Task 3156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e05006cf-1977-4752-975a-88e0cbc2155d_isrForCrosstalkSources_855382_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e05006cf-1977-4752-975a-88e0cbc2155d_isrForCrosstalkSources_855382_53.stderr parsl.dataflow.dflow INFO: Task 3149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5dafa97722c03990debd95c73a576cb with result from task 3149 parsl.dataflow.dflow DEBUG: Task 3149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/79b347b6-066b-4170-97ad-9006d697445d_isrForCrosstalkSources_855455_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/79b347b6-066b-4170-97ad-9006d697445d_isrForCrosstalkSources_855455_60.stderr parsl.dataflow.dflow INFO: Task 3226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aea7c1b8902f3ddd72af8a3f746a97e0 with result from task 3226 parsl.dataflow.dflow DEBUG: Task 3226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d95f3e71-5664-453b-820c-7e5b4543a7e6_isrForCrosstalkSources_855386_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d95f3e71-5664-453b-820c-7e5b4543a7e6_isrForCrosstalkSources_855386_55.stderr parsl.dataflow.dflow INFO: Task 2721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80da686ebda16c44ef34b0da0cf9c222 with result from task 2721 parsl.dataflow.dflow DEBUG: Task 2721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/04dab376-e987-4c37-8387-78703602e39a_isrForCrosstalkSources_855390_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/04dab376-e987-4c37-8387-78703602e39a_isrForCrosstalkSources_855390_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:24 proc_lsst.multi:146 [INFO] found job 16964266 in provider checkpoint proc_lsst.multi INFO: found job 16964266 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8adf1b68cd72316d7d896a61210e9e00 with result from task 2942 parsl.dataflow.dflow DEBUG: Task 2942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/854677dc-76bf-46a1-86fb-74aad2d9958a_isrForCrosstalkSources_855381_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/854677dc-76bf-46a1-86fb-74aad2d9958a_isrForCrosstalkSources_855381_9.stderr parsl.dataflow.dflow INFO: Task 2947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d89b54c641bc83aea430db92a9cc0d6e with result from task 2947 parsl.dataflow.dflow DEBUG: Task 2947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9dbabf65-ed80-4c6c-9851-0ea70618673e_isrForCrosstalkSources_855387_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/9dbabf65-ed80-4c6c-9851-0ea70618673e_isrForCrosstalkSources_855387_9.stderr parsl.dataflow.dflow INFO: Task 3203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a23111972eb19b33a20032120aeef1fd with result from task 3203 parsl.dataflow.dflow DEBUG: Task 3203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/225d352b-6098-4f09-90e1-cd7f79a9d8cd_isrForCrosstalkSources_855400_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/225d352b-6098-4f09-90e1-cd7f79a9d8cd_isrForCrosstalkSources_855400_55.stderr parsl.dataflow.dflow INFO: Task 2984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9021bc3745ff772e0adf581c9bf68dcd with result from task 2984 parsl.dataflow.dflow DEBUG: Task 2984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87284f1d-6afc-47c2-9c50-43cdc0d08dad_isrForCrosstalkSources_855391_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87284f1d-6afc-47c2-9c50-43cdc0d08dad_isrForCrosstalkSources_855391_61.stderr parsl.dataflow.dflow INFO: Task 2912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0a4889eb3c8590dd12e4ea44733ec71 with result from task 2912 parsl.dataflow.dflow DEBUG: Task 2912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a88d2115-8738-46a7-9d44-59627189149a_isrForCrosstalkSources_855383_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a88d2115-8738-46a7-9d44-59627189149a_isrForCrosstalkSources_855383_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:24 proc_lsst.multi:146 [INFO] found job 16964267 in provider checkpoint proc_lsst.multi INFO: found job 16964267 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2929 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry d9f880c265237715f7cea9209a6f481f with result from task 2929 parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2929 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/15be9895-4cfc-40c4-8b05-adb7a8f64434_isrForCrosstalkSources_855394_7.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/15be9895-4cfc-40c4-8b05-adb7a8f64434_isrForCrosstalkSources_855394_7.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:24 proc_lsst.multi:146 [INFO] found job 16964268 in provider checkpoint proc_lsst.multi INFO: found job 16964268 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb089537aabb7644c44b4f762d247cc1 with result from task 2864 parsl.dataflow.dflow DEBUG: Task 2864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/30cd0134-2e33-4dd9-96a7-fcaff743d9f6_isrForCrosstalkSources_855440_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/30cd0134-2e33-4dd9-96a7-fcaff743d9f6_isrForCrosstalkSources_855440_59.stderr parsl.dataflow.dflow INFO: Task 3102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a19620a10e28d5179b56bd7c7d48c624 with result from task 3102 parsl.dataflow.dflow DEBUG: Task 3102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2543b678-93d6-4e0d-b84e-f8bb8abdde2b_isrForCrosstalkSources_855421_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/2543b678-93d6-4e0d-b84e-f8bb8abdde2b_isrForCrosstalkSources_855421_1.stderr parsl.dataflow.dflow INFO: Task 2964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b69e7bae784c602fb659a2cb049b259 with result from task 2964 parsl.dataflow.dflow DEBUG: Task 2964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e73f48bf-78b4-4ebc-a4f0-acf180985535_isrForCrosstalkSources_855450_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e73f48bf-78b4-4ebc-a4f0-acf180985535_isrForCrosstalkSources_855450_61.stderr parsl.dataflow.dflow INFO: Task 2919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b406b198ed72584112d64cd68e951e9 with result from task 2919 parsl.dataflow.dflow DEBUG: Task 2919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2c982f0f-6d82-4f2e-a589-3ec56c55d0a7_isrForCrosstalkSources_855447_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/2c982f0f-6d82-4f2e-a589-3ec56c55d0a7_isrForCrosstalkSources_855447_47.stderr parsl.dataflow.dflow INFO: Task 2894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51b8226a43c66684ffc7e83ea2e87a75 with result from task 2894 parsl.dataflow.dflow DEBUG: Task 2894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7d2a2699-0883-480f-a326-cdce309b08ea_isrForCrosstalkSources_855424_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7d2a2699-0883-480f-a326-cdce309b08ea_isrForCrosstalkSources_855424_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6addb64a03cf265331b391690c0d6dd with result from task 3207 parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 3207 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dee6f79c-fb92-469f-919f-2e8ca5b39bfd_isrForCrosstalkSources_855418_22.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dee6f79c-fb92-469f-919f-2e8ca5b39bfd_isrForCrosstalkSources_855418_22.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:24 proc_lsst.multi:146 [INFO] found job 16964269 in provider checkpoint proc_lsst.multi INFO: found job 16964269 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cbf27ef306c69786bfc3c799b3e9548 with result from task 3061 parsl.dataflow.dflow DEBUG: Task 3061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f7f152dd-d75c-48da-9aca-be59759c86ad_isrForCrosstalkSources_855418_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/f7f152dd-d75c-48da-9aca-be59759c86ad_isrForCrosstalkSources_855418_7.stderr parsl.dataflow.dflow INFO: Task 3057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ff61c0c2a1f7de2baf57bf301dd4d64 with result from task 3057 parsl.dataflow.dflow DEBUG: Task 3057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/085d2b09-5d4f-4d03-ba6c-2ede309fe595_isrForCrosstalkSources_855381_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/085d2b09-5d4f-4d03-ba6c-2ede309fe595_isrForCrosstalkSources_855381_20.stderr parsl.dataflow.dflow INFO: Task 2946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420669f63c3c48ee48040f593587897d with result from task 2946 parsl.dataflow.dflow DEBUG: Task 2946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/157f80ef-9795-49a6-b14d-e8179dc7fd8e_isrForCrosstalkSources_855402_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/157f80ef-9795-49a6-b14d-e8179dc7fd8e_isrForCrosstalkSources_855402_49.stderr parsl.dataflow.dflow INFO: Task 2945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 398915a9622cc054cde096153f394731 with result from task 2945 parsl.dataflow.dflow DEBUG: Task 2945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/768d5dd4-bd31-4491-afca-d7c9d619261d_isrForCrosstalkSources_855417_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/768d5dd4-bd31-4491-afca-d7c9d619261d_isrForCrosstalkSources_855417_22.stderr parsl.dataflow.dflow INFO: Task 3054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6143a2fcc8ddca9dfcb872b3d15a523 with result from task 3054 parsl.dataflow.dflow DEBUG: Task 3054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/1621226e-76d6-4021-b227-e953bf774c88_isrForCrosstalkSources_855418_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/1621226e-76d6-4021-b227-e953bf774c88_isrForCrosstalkSources_855418_43.stderr parsl.dataflow.dflow INFO: Task 3177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ac723187b510a4c216962e44192e89d with result from task 3177 parsl.dataflow.dflow DEBUG: Task 3177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/108f9026-1909-4bbc-aa78-361b8e337298_isrForCrosstalkSources_855432_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/108f9026-1909-4bbc-aa78-361b8e337298_isrForCrosstalkSources_855432_58.stderr parsl.dataflow.dflow INFO: Task 2944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7bf679287721b167f895586b39b1c2c with result from task 2944 parsl.dataflow.dflow DEBUG: Task 2944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/919c6312-7078-4649-aa86-ebe5ab689852_isrForCrosstalkSources_855422_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/919c6312-7078-4649-aa86-ebe5ab689852_isrForCrosstalkSources_855422_5.stderr parsl.dataflow.dflow INFO: Task 3170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e3b907dce88f3c39ab279ea76823975 with result from task 3170 parsl.dataflow.dflow DEBUG: Task 3170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/777f044c-989a-420d-a866-9856ab75bfd4_isrForCrosstalkSources_855385_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/777f044c-989a-420d-a866-9856ab75bfd4_isrForCrosstalkSources_855385_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:25 proc_lsst.multi:146 [INFO] found job 16964270 in provider checkpoint proc_lsst.multi INFO: found job 16964270 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19024fca1923a417ed642922c479c0b with result from task 2972 parsl.dataflow.dflow DEBUG: Task 2972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/5ad4ca0d-90e6-4f43-b837-de4d493ad15e_isrForCrosstalkSources_855409_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/5ad4ca0d-90e6-4f43-b837-de4d493ad15e_isrForCrosstalkSources_855409_2.stderr parsl.dataflow.dflow INFO: Task 3059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316353f69c8160959bc958f4044d74f8 with result from task 3059 parsl.dataflow.dflow DEBUG: Task 3059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/31bcb7cb-a532-428e-a0ac-38140ee6b933_isrForCrosstalkSources_855413_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/31bcb7cb-a532-428e-a0ac-38140ee6b933_isrForCrosstalkSources_855413_60.stderr parsl.dataflow.dflow INFO: Task 2962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebe8f93c8b7610d9a2d81c8161c1b85b with result from task 2962 parsl.dataflow.dflow DEBUG: Task 2962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/2d92fe30-906e-4d00-9e8f-9f3c81f2e33a_isrForCrosstalkSources_855381_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/2d92fe30-906e-4d00-9e8f-9f3c81f2e33a_isrForCrosstalkSources_855381_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:25 proc_lsst.multi:146 [INFO] found job 16964271 in provider checkpoint proc_lsst.multi INFO: found job 16964271 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b649140c364bb20e5e4d29c620d59603 with result from task 3197 parsl.dataflow.dflow DEBUG: Task 3197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/edd5a820-1c68-47be-b7d8-263e07ae92bf_isrForCrosstalkSources_855436_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/edd5a820-1c68-47be-b7d8-263e07ae92bf_isrForCrosstalkSources_855436_48.stderr parsl.dataflow.dflow INFO: Task 3196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df6562bda0520a5383dd6a04fa5ba3e4 with result from task 3196 parsl.dataflow.dflow DEBUG: Task 3196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/738a3ac8-47c8-41e1-93ee-a51e6dc241c4_isrForCrosstalkSources_855394_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/738a3ac8-47c8-41e1-93ee-a51e6dc241c4_isrForCrosstalkSources_855394_11.stderr parsl.dataflow.dflow INFO: Task 2914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c2d121892002538a255f395e7e45c62 with result from task 2914 parsl.dataflow.dflow DEBUG: Task 2914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1d75fac3-806d-42ea-b2f2-e92c949b32b4_isrForCrosstalkSources_855427_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1d75fac3-806d-42ea-b2f2-e92c949b32b4_isrForCrosstalkSources_855427_43.stderr parsl.dataflow.dflow INFO: Task 3056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccd5e261abefcc579b67306073c3e713 with result from task 3056 parsl.dataflow.dflow DEBUG: Task 3056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/88aeb0b9-275b-490d-9ff0-f7d9cd8e0518_isrForCrosstalkSources_855443_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/88aeb0b9-275b-490d-9ff0-f7d9cd8e0518_isrForCrosstalkSources_855443_12.stderr parsl.dataflow.dflow INFO: Task 3070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b929df667f65b82a825dc4f92e0d6bb with result from task 3070 parsl.dataflow.dflow DEBUG: Task 3070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/48423b36-9c01-42e5-853c-09c9d2bc36b3_isrForCrosstalkSources_855446_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/48423b36-9c01-42e5-853c-09c9d2bc36b3_isrForCrosstalkSources_855446_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:25 proc_lsst.multi:146 [INFO] found job 16964272 in provider checkpoint proc_lsst.multi INFO: found job 16964272 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aafc460dbfee19168a5537c7d9181b5c with result from task 3078 parsl.dataflow.dflow DEBUG: Task 3078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5fb5ab6e-c927-42e8-aa3f-9d941c2cb6b4_isrForCrosstalkSources_855403_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5fb5ab6e-c927-42e8-aa3f-9d941c2cb6b4_isrForCrosstalkSources_855403_23.stderr parsl.dataflow.dflow INFO: Task 2910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ed55a8448d00b21fb280e34caee793 with result from task 2910 parsl.dataflow.dflow DEBUG: Task 2910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/33354eb1-a860-446f-9adf-b8e4209ea673_isrForCrosstalkSources_855395_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/33354eb1-a860-446f-9adf-b8e4209ea673_isrForCrosstalkSources_855395_52.stderr parsl.dataflow.dflow INFO: Task 3064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7127f9304e368d798ec036895fa22aa3 with result from task 3064 parsl.dataflow.dflow DEBUG: Task 3064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/06a5168d-3530-4df8-8b80-2caf2619de0d_isrForCrosstalkSources_855391_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/06a5168d-3530-4df8-8b80-2caf2619de0d_isrForCrosstalkSources_855391_9.stderr parsl.dataflow.dflow INFO: Task 2940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da7dbe9c0ff3a89102afffc3c40131c3 with result from task 2940 parsl.dataflow.dflow DEBUG: Task 2940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0d6a8ec2-c008-4447-94fb-8388dc06a86b_isrForCrosstalkSources_855395_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/0d6a8ec2-c008-4447-94fb-8388dc06a86b_isrForCrosstalkSources_855395_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:25 proc_lsst.multi:146 [INFO] found job 16964273 in provider checkpoint proc_lsst.multi INFO: found job 16964273 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 721c9ab2a2f0c1d3f5e77b2d321dcc57 with result from task 3065 parsl.dataflow.dflow DEBUG: Task 3065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/87ad0db7-28d1-4fa1-aecb-19d94ae4d0a5_isrForCrosstalkSources_855392_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/87ad0db7-28d1-4fa1-aecb-19d94ae4d0a5_isrForCrosstalkSources_855392_31.stderr parsl.dataflow.dflow INFO: Task 3115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7338b06a94c0544f3d7da273444fedb with result from task 3115 parsl.dataflow.dflow DEBUG: Task 3115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/fe561538-8534-4e04-ba4d-009612695ea1_isrForCrosstalkSources_855392_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/fe561538-8534-4e04-ba4d-009612695ea1_isrForCrosstalkSources_855392_6.stderr parsl.dataflow.dflow INFO: Task 2938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ceb17d6ddfc59fcc600302f33586e59 with result from task 2938 parsl.dataflow.dflow DEBUG: Task 2938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e7522906-dbba-480a-9be0-579d55840a1b_isrForCrosstalkSources_855380_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e7522906-dbba-480a-9be0-579d55840a1b_isrForCrosstalkSources_855380_58.stderr parsl.dataflow.dflow INFO: Task 2925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcb537105d89085645a428447d962dca with result from task 2925 parsl.dataflow.dflow DEBUG: Task 2925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d1d3fba8-bbca-4b78-a1e7-22282812ee4c_isrForCrosstalkSources_855420_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d1d3fba8-bbca-4b78-a1e7-22282812ee4c_isrForCrosstalkSources_855420_36.stderr parsl.dataflow.dflow INFO: Task 2976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3adf83b8b4ed109ad1e3123464a0516 with result from task 2976 parsl.dataflow.dflow DEBUG: Task 2976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c4dff29b-9be6-4783-8ac7-798a25c20f64_isrForCrosstalkSources_855450_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/c4dff29b-9be6-4783-8ac7-798a25c20f64_isrForCrosstalkSources_855450_35.stderr parsl.dataflow.dflow INFO: Task 2997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540e7e2ddde767f6274ecdb983eb25ab with result from task 2997 parsl.dataflow.dflow DEBUG: Task 2997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e473d6b0-e22e-4dd1-a0a8-f5b275175da0_isrForCrosstalkSources_855423_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e473d6b0-e22e-4dd1-a0a8-f5b275175da0_isrForCrosstalkSources_855423_13.stderr parsl.dataflow.dflow INFO: Task 3037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baf8e453e233ffee0a3ea4bcd9c67283 with result from task 3037 parsl.dataflow.dflow DEBUG: Task 3037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/2816fc43-1de6-4532-bb22-5a3cd6fda399_isrForCrosstalkSources_855387_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/2816fc43-1de6-4532-bb22-5a3cd6fda399_isrForCrosstalkSources_855387_1.stderr parsl.dataflow.dflow INFO: Task 3134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09a53edf9436f62002bcddfc54f19abe with result from task 3134 parsl.dataflow.dflow DEBUG: Task 3134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/faa91b9a-4c37-4a3a-84a8-0bc885237d6f_isrForCrosstalkSources_855456_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/faa91b9a-4c37-4a3a-84a8-0bc885237d6f_isrForCrosstalkSources_855456_47.stderr parsl.dataflow.dflow INFO: Task 3148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0122201afe6c4519ecd6aac2347806 with result from task 3148 parsl.dataflow.dflow DEBUG: Task 3148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8372b7e-0188-47f6-8a08-af8fe5369012_isrForCrosstalkSources_855450_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d8372b7e-0188-47f6-8a08-af8fe5369012_isrForCrosstalkSources_855450_43.stderr parsl.dataflow.dflow INFO: Task 2968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 366a10ba2701153538cd355d3a289769 with result from task 2968 parsl.dataflow.dflow DEBUG: Task 2968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c55c0281-c57a-4d8d-b831-0f4c6478dee0_isrForCrosstalkSources_855419_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c55c0281-c57a-4d8d-b831-0f4c6478dee0_isrForCrosstalkSources_855419_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:26 proc_lsst.multi:146 [INFO] found job 16964274 in provider checkpoint proc_lsst.multi INFO: found job 16964274 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef6fe297c088d85779baa59eef57090 with result from task 3144 parsl.dataflow.dflow DEBUG: Task 3144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0fe2b54c-4a4a-4965-8f21-fad9d49704c2_isrForCrosstalkSources_855450_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0fe2b54c-4a4a-4965-8f21-fad9d49704c2_isrForCrosstalkSources_855450_54.stderr parsl.dataflow.dflow INFO: Task 3004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb0bbd936f6b1eed8ec78ea4705b0541 with result from task 3004 parsl.dataflow.dflow DEBUG: Task 3004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ad78ceb9-ebed-4d73-b2b1-d81a0219e0d0_isrForCrosstalkSources_855392_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/ad78ceb9-ebed-4d73-b2b1-d81a0219e0d0_isrForCrosstalkSources_855392_43.stderr parsl.dataflow.dflow INFO: Task 3159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f92f3af8422fa0832f360d13c0b77f8c with result from task 3159 parsl.dataflow.dflow DEBUG: Task 3159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/30f11702-0533-4d06-8d00-7216c6f8e7f4_isrForCrosstalkSources_855402_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/30f11702-0533-4d06-8d00-7216c6f8e7f4_isrForCrosstalkSources_855402_5.stderr parsl.dataflow.dflow INFO: Task 3084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80f33f171471d93889b24affea5839d1 with result from task 3084 parsl.dataflow.dflow DEBUG: Task 3084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8224a843-7255-43dc-b340-4d9ff7832f2a_isrForCrosstalkSources_855441_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8224a843-7255-43dc-b340-4d9ff7832f2a_isrForCrosstalkSources_855441_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:27 proc_lsst.multi:146 [INFO] found job 16964275 in provider checkpoint proc_lsst.multi INFO: found job 16964275 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 2908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d73b8b7fb9565ab07a092fd2071f3d28 with result from task 2908 parsl.dataflow.dflow DEBUG: Task 2908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3e57a946-3749-470f-a889-34fab3eb9158_isrForCrosstalkSources_855380_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3e57a946-3749-470f-a889-34fab3eb9158_isrForCrosstalkSources_855380_36.stderr parsl.dataflow.dflow INFO: Task 3034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1da30869b5f1aa2d27c2da2d901a251 with result from task 3034 parsl.dataflow.dflow DEBUG: Task 3034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ec44760d-ab13-4d8d-a5b8-9cd2a70ae087_isrForCrosstalkSources_855424_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ec44760d-ab13-4d8d-a5b8-9cd2a70ae087_isrForCrosstalkSources_855424_30.stderr parsl.dataflow.dflow INFO: Task 3137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a721eed13529e2101a8b3dfbbfef79c with result from task 3137 parsl.dataflow.dflow DEBUG: Task 3137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ba184d63-eae0-41e2-90f1-42a10e8e405a_isrForCrosstalkSources_855429_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ba184d63-eae0-41e2-90f1-42a10e8e405a_isrForCrosstalkSources_855429_47.stderr parsl.dataflow.dflow INFO: Task 3172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b59088f247794f0031b3c914433f5ae9 with result from task 3172 parsl.dataflow.dflow DEBUG: Task 3172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f746a478-aed0-4ec1-8252-dfc12870da28_isrForCrosstalkSources_855398_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f746a478-aed0-4ec1-8252-dfc12870da28_isrForCrosstalkSources_855398_57.stderr parsl.dataflow.dflow INFO: Task 3167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17aaca43568e9454589040f8235464cb with result from task 3167 parsl.dataflow.dflow DEBUG: Task 3167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/f4ba1914-4649-4104-9a49-c64a4fe547ea_isrForCrosstalkSources_855442_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/f4ba1914-4649-4104-9a49-c64a4fe547ea_isrForCrosstalkSources_855442_57.stderr parsl.dataflow.dflow INFO: Task 3007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 493f769e5452351a831a1dc4c064c6d1 with result from task 3007 parsl.dataflow.dflow DEBUG: Task 3007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/19731c85-63eb-4f80-86fa-fa50552bd097_isrForCrosstalkSources_855397_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/19731c85-63eb-4f80-86fa-fa50552bd097_isrForCrosstalkSources_855397_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:27 proc_lsst.multi:146 [INFO] found job 16964276 in provider checkpoint proc_lsst.multi INFO: found job 16964276 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e507bcef407de3a88a150e9082874f4 with result from task 3086 parsl.dataflow.dflow DEBUG: Task 3086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/941643e5-5a88-408d-b8bc-0768e327c9a3_isrForCrosstalkSources_855404_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/941643e5-5a88-408d-b8bc-0768e327c9a3_isrForCrosstalkSources_855404_2.stderr parsl.dataflow.dflow INFO: Task 3169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd1af5fc59bee3f3ae70a3806709dd78 with result from task 3169 parsl.dataflow.dflow DEBUG: Task 3169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1a538d35-3571-4fd8-a5d3-1d2e6021518f_isrForCrosstalkSources_855384_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/1a538d35-3571-4fd8-a5d3-1d2e6021518f_isrForCrosstalkSources_855384_53.stderr parsl.dataflow.dflow INFO: Task 2920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e947bfa8dd5ff9bccaa273b3a9cf0924 with result from task 2920 parsl.dataflow.dflow DEBUG: Task 2920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aa8fff54-1be5-430e-8ce0-ff95d520b18b_isrForCrosstalkSources_855426_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aa8fff54-1be5-430e-8ce0-ff95d520b18b_isrForCrosstalkSources_855426_62.stderr parsl.dataflow.dflow INFO: Task 2923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cae145e8d794415b7f71139f82daccac with result from task 2923 parsl.dataflow.dflow DEBUG: Task 2923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ee5ca1a4-085f-455f-82bb-3e596ad01efa_isrForCrosstalkSources_855433_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ee5ca1a4-085f-455f-82bb-3e596ad01efa_isrForCrosstalkSources_855433_57.stderr parsl.dataflow.dflow INFO: Task 2918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e379fb19e32d13519721543f7f7c3da1 with result from task 2918 parsl.dataflow.dflow DEBUG: Task 2918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/10b52193-a551-420b-b5c9-0c5076af39c0_isrForCrosstalkSources_855400_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/10b52193-a551-420b-b5c9-0c5076af39c0_isrForCrosstalkSources_855400_47.stderr parsl.dataflow.dflow INFO: Task 2986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f486f6b3456ef79142ad7d64f2bc706 with result from task 2986 parsl.dataflow.dflow DEBUG: Task 2986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/5e87eeaf-fed3-446f-8593-78ee602f2856_isrForCrosstalkSources_855399_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/5e87eeaf-fed3-446f-8593-78ee602f2856_isrForCrosstalkSources_855399_48.stderr parsl.dataflow.dflow INFO: Task 3201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d3bb7e02231c12344d0174df7b77f2 with result from task 3201 parsl.dataflow.dflow DEBUG: Task 3201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3688b4-6e1f-4d11-bb41-c1a35f91e865_isrForCrosstalkSources_855445_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5d3688b4-6e1f-4d11-bb41-c1a35f91e865_isrForCrosstalkSources_855445_48.stderr parsl.dataflow.dflow INFO: Task 3072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d72daa29c16a8e29cc5c2574543b6c with result from task 3072 parsl.dataflow.dflow DEBUG: Task 3072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ca322b92-a0df-4caa-9719-1ed2a6a6e86d_isrForCrosstalkSources_855459_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ca322b92-a0df-4caa-9719-1ed2a6a6e86d_isrForCrosstalkSources_855459_37.stderr parsl.dataflow.dflow INFO: Task 2913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b187dc14b7e98d809235feebc91caed with result from task 2913 parsl.dataflow.dflow DEBUG: Task 2913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/be3f152b-722e-4cec-bf81-a29cb251e274_isrForCrosstalkSources_855400_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/be3f152b-722e-4cec-bf81-a29cb251e274_isrForCrosstalkSources_855400_62.stderr parsl.dataflow.dflow INFO: Task 3257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd08704b63cdfdbb2adcd6f58eb72bb with result from task 3257 parsl.dataflow.dflow DEBUG: Task 3257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2d10b200-29db-4e8b-abd3-fdbf00f15e7f_isrForCrosstalkSources_855432_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2d10b200-29db-4e8b-abd3-fdbf00f15e7f_isrForCrosstalkSources_855432_59.stderr parsl.dataflow.dflow INFO: Task 3150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ea4fb47735fe25eced59a6056c81764 with result from task 3150 parsl.dataflow.dflow DEBUG: Task 3150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1a50e18a-f8e6-43ea-be0e-8d7e432730ab_isrForCrosstalkSources_855382_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1a50e18a-f8e6-43ea-be0e-8d7e432730ab_isrForCrosstalkSources_855382_38.stderr parsl.dataflow.dflow INFO: Task 2950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 185483fa71579cad8536a8ef44a08101 with result from task 2950 parsl.dataflow.dflow DEBUG: Task 2950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ecdd6c6e-6c2d-483e-8c6e-36ac6dac08cc_isrForCrosstalkSources_855449_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ecdd6c6e-6c2d-483e-8c6e-36ac6dac08cc_isrForCrosstalkSources_855449_36.stderr parsl.dataflow.dflow INFO: Task 3114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d059fb5372d0646391b35a5ced4e0f4c with result from task 3114 parsl.dataflow.dflow DEBUG: Task 3114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a38bec8a-0cb8-4d55-be38-958991c26429_isrForCrosstalkSources_855403_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a38bec8a-0cb8-4d55-be38-958991c26429_isrForCrosstalkSources_855403_1.stderr parsl.dataflow.dflow INFO: Task 2956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b585222d6432239e7e4fe9429e0ee3ef with result from task 2956 parsl.dataflow.dflow DEBUG: Task 2956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d1b73dc2-16c7-4537-9aba-450cb1eca11e_isrForCrosstalkSources_855407_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d1b73dc2-16c7-4537-9aba-450cb1eca11e_isrForCrosstalkSources_855407_41.stderr parsl.dataflow.dflow INFO: Task 3049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19732185f3d05d45754b936e7207c94b with result from task 3049 parsl.dataflow.dflow DEBUG: Task 3049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5202251c-026f-4cd7-8327-541bdc12e6b1_isrForCrosstalkSources_855434_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5202251c-026f-4cd7-8327-541bdc12e6b1_isrForCrosstalkSources_855434_31.stderr parsl.dataflow.dflow INFO: Task 3003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeff74643f4c5172d8e5d15dac88add5 with result from task 3003 parsl.dataflow.dflow DEBUG: Task 3003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0539a862-ffc1-4f8b-b9f8-1e628a685037_isrForCrosstalkSources_855386_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/0539a862-ffc1-4f8b-b9f8-1e628a685037_isrForCrosstalkSources_855386_61.stderr parsl.dataflow.dflow INFO: Task 3050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43e07ea208b3a553e1f4095b391d7b5 with result from task 3050 parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:28 proc_lsst.multi:146 [INFO] found job 16964277 in provider checkpoint proc_lsst.multi INFO: found job 16964277 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow DEBUG: Task 3050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/4d8dcda9-088a-4bf9-aa3c-6648e07a6da7_isrForCrosstalkSources_855395_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/4d8dcda9-088a-4bf9-aa3c-6648e07a6da7_isrForCrosstalkSources_855395_55.stderr parsl.dataflow.dflow INFO: Task 3112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e83a7def455d3349861a8f8e0bb4f20 with result from task 3112 parsl.dataflow.dflow DEBUG: Task 3112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/cfc40796-2557-4782-9a95-65155da3929d_isrForCrosstalkSources_855450_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/cfc40796-2557-4782-9a95-65155da3929d_isrForCrosstalkSources_855450_19.stderr parsl.dataflow.dflow INFO: Task 3192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15f6da44a3f5faf4f9f336adcc238f1d with result from task 3192 parsl.dataflow.dflow DEBUG: Task 3192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/84e52086-02ea-4e49-8f6d-16bbdfaf5e68_isrForCrosstalkSources_855448_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/84e52086-02ea-4e49-8f6d-16bbdfaf5e68_isrForCrosstalkSources_855448_33.stderr parsl.dataflow.dflow INFO: Task 3199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45e4219bf0e4f0f6949f8f27302d8d2c with result from task 3199 parsl.dataflow.dflow DEBUG: Task 3199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a8562c53-a77b-4732-8000-e02778069740_isrForCrosstalkSources_855408_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a8562c53-a77b-4732-8000-e02778069740_isrForCrosstalkSources_855408_19.stderr parsl.dataflow.dflow INFO: Task 3146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9d79df8224b14042a4181636f830761 with result from task 3146 parsl.dataflow.dflow DEBUG: Task 3146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/2a4801a7-5262-49ab-b1d7-a38a14500290_isrForCrosstalkSources_855413_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/2a4801a7-5262-49ab-b1d7-a38a14500290_isrForCrosstalkSources_855413_57.stderr parsl.dataflow.dflow INFO: Task 3383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37307451f58886031901d55e3f8aaacc with result from task 3383 parsl.dataflow.dflow DEBUG: Task 3383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/4b3eda03-6f67-4424-a91d-3255c78c3494_isrForCrosstalkSources_855427_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/4b3eda03-6f67-4424-a91d-3255c78c3494_isrForCrosstalkSources_855427_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964153 PD 16964154 PD 16964155 PD 16964156 PD 16964157 PD 16964158 PD 16964159 PD 16964160 PD 16964161 PD 16964162 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:54:28 proc_lsst.multi:146 [INFO] found job 16964278 in provider checkpoint proc_lsst.multi INFO: found job 16964278 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278' parsl.dataflow.dflow INFO: Task 3410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f222fcdac88959fb44022c2969c56b57 with result from task 3410 parsl.dataflow.dflow DEBUG: Task 3410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/181886ad-03ce-4f54-9e5f-c7370d34dbd7_isrForCrosstalkSources_855405_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/181886ad-03ce-4f54-9e5f-c7370d34dbd7_isrForCrosstalkSources_855405_20.stderr parsl.dataflow.dflow INFO: Task 2909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1374e220450430f7787ee2f1467a1bcc with result from task 2909 parsl.dataflow.dflow DEBUG: Task 2909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/74c707f4-011e-4362-9e31-1b996f252ebb_isrForCrosstalkSources_855451_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/74c707f4-011e-4362-9e31-1b996f252ebb_isrForCrosstalkSources_855451_55.stderr parsl.dataflow.dflow INFO: Task 3027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70fb28a48ea0a657c86f38380e36325 with result from task 3027 parsl.dataflow.dflow DEBUG: Task 3027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/32fe50a5-8f99-444a-bcba-67b0044daa99_isrForCrosstalkSources_855426_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/32fe50a5-8f99-444a-bcba-67b0044daa99_isrForCrosstalkSources_855426_30.stderr parsl.dataflow.dflow INFO: Task 2936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3887bf7be31b646c4ddebc68b3c87fee with result from task 2936 parsl.dataflow.dflow DEBUG: Task 2936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5445774a-ebc0-4762-86b5-bbd58a9c8f6b_isrForCrosstalkSources_855395_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5445774a-ebc0-4762-86b5-bbd58a9c8f6b_isrForCrosstalkSources_855395_54.stderr parsl.dataflow.dflow INFO: Task 3287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33d81bbb14227f3f6e724cfc14b43a0a with result from task 3287 parsl.dataflow.dflow DEBUG: Task 3287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dad2ffd7-0bab-4de6-9679-cfd51bdee422_isrForCrosstalkSources_855418_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/dad2ffd7-0bab-4de6-9679-cfd51bdee422_isrForCrosstalkSources_855418_26.stderr parsl.dataflow.dflow INFO: Task 3104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60f2ed895eaa0e2e6d19a01a5bd8aa43 with result from task 3104 parsl.dataflow.dflow DEBUG: Task 3104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db45e412-231f-4950-9ad2-18ac46b203fa_isrForCrosstalkSources_855388_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/db45e412-231f-4950-9ad2-18ac46b203fa_isrForCrosstalkSources_855388_56.stderr parsl.dataflow.dflow INFO: Task 3180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac9839140289e0a15d37f01593bcf76 with result from task 3180 parsl.dataflow.dflow DEBUG: Task 3180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de53612c-9c40-40c6-9c2f-af72fdc74a27_isrForCrosstalkSources_855459_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de53612c-9c40-40c6-9c2f-af72fdc74a27_isrForCrosstalkSources_855459_3.stderr parsl.dataflow.dflow INFO: Task 3107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b32de1ee90295a4a1bead09910e38c42 with result from task 3107 parsl.dataflow.dflow DEBUG: Task 3107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0648fbad-c73b-406b-b6fd-9414a588cb71_isrForCrosstalkSources_855412_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/0648fbad-c73b-406b-b6fd-9414a588cb71_isrForCrosstalkSources_855412_9.stderr parsl.dataflow.dflow INFO: Task 3204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408369da1d08c2420c572c499da4c436 with result from task 3204 parsl.dataflow.dflow DEBUG: Task 3204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/aacad426-95cb-4cc5-a93d-7aa6adbd6ad0_isrForCrosstalkSources_855382_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/aacad426-95cb-4cc5-a93d-7aa6adbd6ad0_isrForCrosstalkSources_855382_55.stderr parsl.dataflow.dflow INFO: Task 3028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 687c3f522a2aec79a4f29f0223c5e9db with result from task 3028 parsl.dataflow.dflow DEBUG: Task 3028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5d6117fe-a165-4372-8c7c-3870b78eb169_isrForCrosstalkSources_855428_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/5d6117fe-a165-4372-8c7c-3870b78eb169_isrForCrosstalkSources_855428_44.stderr parsl.dataflow.dflow INFO: Task 3178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cb8606087be03e37127a269ab1162ed with result from task 3178 parsl.dataflow.dflow DEBUG: Task 3178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d1a2981f-b157-4c07-b374-cc7bd304701e_isrForCrosstalkSources_855443_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/d1a2981f-b157-4c07-b374-cc7bd304701e_isrForCrosstalkSources_855443_60.stderr parsl.dataflow.dflow INFO: Task 3095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98d3635c70d5234da7abdbddca2ebce9 with result from task 3095 parsl.dataflow.dflow DEBUG: Task 3095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1089286f-1ac3-4cd9-a4fa-9f6fe455678f_isrForCrosstalkSources_855421_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1089286f-1ac3-4cd9-a4fa-9f6fe455678f_isrForCrosstalkSources_855421_42.stderr parsl.dataflow.dflow INFO: Task 2999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d583aa3698416dd369072ee843931a13 with result from task 2999 parsl.dataflow.dflow DEBUG: Task 2999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e51ca827-eeaf-4b4c-92a3-057d5190f1e0_isrForCrosstalkSources_855441_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/e51ca827-eeaf-4b4c-92a3-057d5190f1e0_isrForCrosstalkSources_855441_47.stderr parsl.dataflow.dflow INFO: Task 2998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f7daa23812b239b13b93c0137da651 with result from task 2998 parsl.dataflow.dflow DEBUG: Task 2998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/560a825b-1099-4b20-8164-9e8424f27361_isrForCrosstalkSources_855385_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/560a825b-1099-4b20-8164-9e8424f27361_isrForCrosstalkSources_855385_44.stderr parsl.dataflow.dflow INFO: Task 3033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4888494a8f1fe0beb04eece98cd452cc with result from task 3033 parsl.dataflow.dflow DEBUG: Task 3033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/51358ccd-b027-4dca-8094-e8aa54ea5ec4_isrForCrosstalkSources_855427_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/51358ccd-b027-4dca-8094-e8aa54ea5ec4_isrForCrosstalkSources_855427_49.stderr parsl.dataflow.dflow INFO: Task 3101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d053cb6de15ec4c8278fe4b6fc5e1e1b with result from task 3101 parsl.dataflow.dflow DEBUG: Task 3101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e03f4126-52f8-4778-af1f-646b828612a5_isrForCrosstalkSources_855404_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e03f4126-52f8-4778-af1f-646b828612a5_isrForCrosstalkSources_855404_29.stderr parsl.dataflow.dflow INFO: Task 3110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e194981bdd800fd635710ebdc27bc00 with result from task 3110 parsl.dataflow.dflow DEBUG: Task 3110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/16670da2-e09a-46f7-b274-dd9f711d171b_isrForCrosstalkSources_855398_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/16670da2-e09a-46f7-b274-dd9f711d171b_isrForCrosstalkSources_855398_9.stderr parsl.dataflow.dflow INFO: Task 3256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94b169295e47af1f13ac85a49f668a3f with result from task 3256 parsl.dataflow.dflow DEBUG: Task 3256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/91fba636-dcac-4a09-bd97-068bd1350272_isrForCrosstalkSources_855391_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/91fba636-dcac-4a09-bd97-068bd1350272_isrForCrosstalkSources_855391_51.stderr parsl.dataflow.dflow INFO: Task 3069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c59658688345181fb58f281d8b505ce with result from task 3069 parsl.dataflow.dflow DEBUG: Task 3069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6b63290a-2bf4-446e-9897-37fbe00aaa5a_isrForCrosstalkSources_855383_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/6b63290a-2bf4-446e-9897-37fbe00aaa5a_isrForCrosstalkSources_855383_23.stderr parsl.dataflow.dflow INFO: Task 3374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0541556ecf661ae5a6104a062153726a with result from task 3374 parsl.dataflow.dflow DEBUG: Task 3374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2c132c63-330e-4301-afed-6d3ef4653d78_isrForCrosstalkSources_855382_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/2c132c63-330e-4301-afed-6d3ef4653d78_isrForCrosstalkSources_855382_30.stderr parsl.dataflow.dflow INFO: Task 3105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20eebde199d37c2fa33afe11cdc2560b with result from task 3105 parsl.dataflow.dflow DEBUG: Task 3105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/080ad1ce-3d12-4426-b000-c6de40212d88_isrForCrosstalkSources_855395_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/080ad1ce-3d12-4426-b000-c6de40212d88_isrForCrosstalkSources_855395_7.stderr parsl.dataflow.dflow INFO: Task 3121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24a96c9911d3e8956ced19f91b5bab12 with result from task 3121 parsl.dataflow.dflow DEBUG: Task 3121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18f0e5fe-a34b-4102-a32e-5a6b71034718_isrForCrosstalkSources_855443_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18f0e5fe-a34b-4102-a32e-5a6b71034718_isrForCrosstalkSources_855443_30.stderr parsl.dataflow.dflow INFO: Task 3345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abac1aa8cc88776d8cb9dd74762b5455 with result from task 3345 parsl.dataflow.dflow DEBUG: Task 3345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d3e643c1-4532-48c7-a9c7-137ad03b8e45_isrForCrosstalkSources_855397_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/d3e643c1-4532-48c7-a9c7-137ad03b8e45_isrForCrosstalkSources_855397_24.stderr parsl.dataflow.dflow INFO: Task 3051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b18feb79c3407d9699ad223948d81520 with result from task 3051 parsl.dataflow.dflow DEBUG: Task 3051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/20e16015-8539-462d-b148-acda03081c1d_isrForCrosstalkSources_855410_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/20e16015-8539-462d-b148-acda03081c1d_isrForCrosstalkSources_855410_55.stderr parsl.dataflow.dflow INFO: Task 2978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93faebb3426f12b3fbfdbcf0c8e6fc6f with result from task 2978 parsl.dataflow.dflow DEBUG: Task 2978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/1767a961-3366-4aff-af8c-d0d0409573be_isrForCrosstalkSources_855405_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/1767a961-3366-4aff-af8c-d0d0409573be_isrForCrosstalkSources_855405_62.stderr parsl.dataflow.dflow INFO: Task 2965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b829077de0b285965cc4cc1c48c284 with result from task 2965 parsl.dataflow.dflow DEBUG: Task 2965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/72d6aec5-9905-4c70-b318-9a588c74e7c5_isrForCrosstalkSources_855413_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/72d6aec5-9905-4c70-b318-9a588c74e7c5_isrForCrosstalkSources_855413_42.stderr parsl.dataflow.dflow INFO: Task 3215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d6341c7e43f08253d59a43a8e4c6c6 with result from task 3215 parsl.dataflow.dflow DEBUG: Task 3215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e75f4c15-9c56-4af2-9c74-27c4648ff9ab_isrForCrosstalkSources_855423_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/e75f4c15-9c56-4af2-9c74-27c4648ff9ab_isrForCrosstalkSources_855423_12.stderr parsl.dataflow.dflow INFO: Task 3023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8314a4fb94aac76b54ee9e49b5a7b0ef with result from task 3023 parsl.dataflow.dflow DEBUG: Task 3023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4988558f-8e83-42a9-8cd4-501210093783_isrForCrosstalkSources_855451_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4988558f-8e83-42a9-8cd4-501210093783_isrForCrosstalkSources_855451_12.stderr parsl.dataflow.dflow INFO: Task 3030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bed110c612109faf78d14f3a7008bc79 with result from task 3030 parsl.dataflow.dflow DEBUG: Task 3030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/236ee426-c51d-499b-8097-ee1523ea8038_isrForCrosstalkSources_855389_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/236ee426-c51d-499b-8097-ee1523ea8038_isrForCrosstalkSources_855389_41.stderr parsl.dataflow.dflow INFO: Task 3082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5e8977e391198cd102fd512fe359666 with result from task 3082 parsl.dataflow.dflow DEBUG: Task 3082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0177d9ca-3c8c-4d9b-90d7-55c8acf0e14e_isrForCrosstalkSources_855411_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0177d9ca-3c8c-4d9b-90d7-55c8acf0e14e_isrForCrosstalkSources_855411_44.stderr parsl.dataflow.dflow INFO: Task 3026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76882c1a92b68a7ff46cf8c579fa6c86 with result from task 3026 parsl.dataflow.dflow DEBUG: Task 3026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1b849368-eb3f-4781-8f19-867f251810d8_isrForCrosstalkSources_855417_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/1b849368-eb3f-4781-8f19-867f251810d8_isrForCrosstalkSources_855417_41.stderr parsl.dataflow.dflow INFO: Task 3058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c80c1e759e8c78b7297ab47598195c9 with result from task 3058 parsl.dataflow.dflow DEBUG: Task 3058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4a6ff96c-f4e2-42d5-9917-58a94eb9b6f6_isrForCrosstalkSources_855451_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4a6ff96c-f4e2-42d5-9917-58a94eb9b6f6_isrForCrosstalkSources_855451_31.stderr parsl.dataflow.dflow INFO: Task 3017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b629752f76951a064445ac0f2f6b252 with result from task 3017 parsl.dataflow.dflow DEBUG: Task 3017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5ca4db1c-7c84-4d4c-b307-e149f6456243_isrForCrosstalkSources_855453_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/5ca4db1c-7c84-4d4c-b307-e149f6456243_isrForCrosstalkSources_855453_43.stderr parsl.dataflow.dflow INFO: Task 3083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce5d597799335eac34fe14a6c705a9e8 with result from task 3083 parsl.dataflow.dflow DEBUG: Task 3083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b7d8d590-f1ef-4696-a605-a3c05165194c_isrForCrosstalkSources_855389_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b7d8d590-f1ef-4696-a605-a3c05165194c_isrForCrosstalkSources_855389_50.stderr parsl.dataflow.dflow INFO: Task 3425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11325940b38f2a0f419e52b3f065e5ea with result from task 3425 parsl.dataflow.dflow DEBUG: Task 3425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c97ac78d-099f-4672-b7e5-c0d0c5ac231b_isrForCrosstalkSources_855410_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/c97ac78d-099f-4672-b7e5-c0d0c5ac231b_isrForCrosstalkSources_855410_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964125 PD 16964126 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964162 PD 16964161 PD 16964160 PD 16964159 PD 16964158 PD 16964157 PD 16964156 PD 16964155 PD 16964154 PD 16964153 PD 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964141 R 16964142 R 16964143 R 16964144 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 250, active_tasks = 2062 parsl.dataflow.strategy DEBUG: Executor multi has 2062 active tasks, 127/123 running/pending blocks, and 625 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 250 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 62 more blocks parsl.executors.status_handling INFO: Scaling out by 62 blocks parsl.executors.status_handling INFO: Allocated block ID 250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-250 2024-03-09 14:54:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 2024-03-09 14:54:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 194 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 194 2024-03-09 14:54:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e09ccc93a526d368a8041d347ec4858 with result from task 3131 parsl.dataflow.dflow DEBUG: Task 3131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/7f429e02-bae3-47b9-81d7-3fa12bbf1184_isrForCrosstalkSources_855398_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/7f429e02-bae3-47b9-81d7-3fa12bbf1184_isrForCrosstalkSources_855398_25.stderr parsl.dataflow.dflow INFO: Task 3321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 910cca861ea88a4292b16c764db7fade with result from task 3321 parsl.dataflow.dflow DEBUG: Task 3321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f262d202-cb53-4bd9-8df5-9c475ab1f501_isrForCrosstalkSources_855453_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f262d202-cb53-4bd9-8df5-9c475ab1f501_isrForCrosstalkSources_855453_6.stderr parsl.dataflow.dflow INFO: Task 3096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f15bf538e3e49f7b857ce179f9c12ab with result from task 3096 parsl.dataflow.dflow DEBUG: Task 3096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8a98c94f-9fcb-4c23-b597-7ac875066d23_isrForCrosstalkSources_855425_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8a98c94f-9fcb-4c23-b597-7ac875066d23_isrForCrosstalkSources_855425_35.stderr parsl.dataflow.dflow INFO: Task 2900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3b1c6b296c316bdbb33cfa8eed85d1e with result from task 2900 parsl.dataflow.dflow DEBUG: Task 2900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b738176c-b4de-4090-b55b-90d7120944fd_isrForCrosstalkSources_855420_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/b738176c-b4de-4090-b55b-90d7120944fd_isrForCrosstalkSources_855420_47.stderr parsl.dataflow.dflow INFO: Task 3140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c7cc708ba50580a328cc217303d72e with result from task 3140 parsl.dataflow.dflow DEBUG: Task 3140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/fd569ef2-153d-43ff-ae8c-c1cd76f62f72_isrForCrosstalkSources_855407_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/fd569ef2-153d-43ff-ae8c-c1cd76f62f72_isrForCrosstalkSources_855407_38.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d31388402bac38babf80364a8af6e436 with result from task 3273 parsl.dataflow.dflow DEBUG: Task 3273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/7a09041c-52db-4ec9-9d80-5aba81b5dcba_isrForCrosstalkSources_855420_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/7a09041c-52db-4ec9-9d80-5aba81b5dcba_isrForCrosstalkSources_855420_28.stderr parsl.dataflow.dflow INFO: Task 3141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eb4c914742af45f08c6b1ef829361f8 with result from task 3141 parsl.dataflow.dflow DEBUG: Task 3141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/64fb6471-a692-414f-8edf-a47b21f64520_isrForCrosstalkSources_855435_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/64fb6471-a692-414f-8edf-a47b21f64520_isrForCrosstalkSources_855435_33.stderr parsl.dataflow.dflow INFO: Task 2924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b780f2aef3a188c3e8754efcc6c61002 with result from task 2924 parsl.dataflow.dflow DEBUG: Task 2924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/230a11b2-2718-4e49-8e05-4209600d57ec_isrForCrosstalkSources_855379_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/230a11b2-2718-4e49-8e05-4209600d57ec_isrForCrosstalkSources_855379_41.stderr parsl.dataflow.dflow INFO: Task 3240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e50f5ae985b03230e33ed90dc320990b with result from task 3240 parsl.dataflow.dflow DEBUG: Task 3240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d86ef9a2-be1d-4639-a97d-0034c8b6207e_isrForCrosstalkSources_855429_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d86ef9a2-be1d-4639-a97d-0034c8b6207e_isrForCrosstalkSources_855429_32.stderr parsl.dataflow.dflow INFO: Task 3062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af0c02a89b4a8d56a766af7d3541a4ce with result from task 3062 parsl.dataflow.dflow DEBUG: Task 3062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/25759866-53ba-431e-95b2-fcab8520b4c4_isrForCrosstalkSources_855396_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/25759866-53ba-431e-95b2-fcab8520b4c4_isrForCrosstalkSources_855396_29.stderr parsl.dataflow.dflow INFO: Task 3430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7f57a405a9813dc9a51c874381e1c12 with result from task 3430 parsl.dataflow.dflow DEBUG: Task 3430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a86efdff-4641-4ca5-9ea8-e50afcd0f1b5_isrForCrosstalkSources_855456_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a86efdff-4641-4ca5-9ea8-e50afcd0f1b5_isrForCrosstalkSources_855456_36.stderr parsl.dataflow.dflow INFO: Task 2911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f39749e69fd6381bba8744f9ff15b779 with result from task 2911 parsl.dataflow.dflow DEBUG: Task 2911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/4bd27203-1214-4651-b0ee-0d614c4d175f_isrForCrosstalkSources_855420_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/4bd27203-1214-4651-b0ee-0d614c4d175f_isrForCrosstalkSources_855420_53.stderr parsl.dataflow.dflow INFO: Task 3073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1fdea2e7a497d77356e3893d559dd63 with result from task 3073 parsl.dataflow.dflow DEBUG: Task 3073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/30e8f903-1ab0-4a10-bc71-636420bbfd65_isrForCrosstalkSources_855411_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/30e8f903-1ab0-4a10-bc71-636420bbfd65_isrForCrosstalkSources_855411_7.stderr parsl.dataflow.dflow INFO: Task 3116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d2c04ac068cf81d00cae9b9f7667af with result from task 3116 parsl.dataflow.dflow DEBUG: Task 3116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0ca423fa-18e0-4bcf-9b03-9b468161708b_isrForCrosstalkSources_855459_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/0ca423fa-18e0-4bcf-9b03-9b468161708b_isrForCrosstalkSources_855459_13.stderr parsl.dataflow.dflow INFO: Task 3320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 598d6148dc5bba084f06156698eb592c with result from task 3320 parsl.dataflow.dflow DEBUG: Task 3320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/cbb83a5a-3046-4cd8-ace9-eefcd9016d10_isrForCrosstalkSources_855396_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/cbb83a5a-3046-4cd8-ace9-eefcd9016d10_isrForCrosstalkSources_855396_62.stderr parsl.dataflow.dflow INFO: Task 3088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5a34036dc0aca3dd2f8b622915df025 with result from task 3088 parsl.dataflow.dflow DEBUG: Task 3088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6dec1bc1-2edc-4447-9a0a-48d6c8452bde_isrForCrosstalkSources_855431_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6dec1bc1-2edc-4447-9a0a-48d6c8452bde_isrForCrosstalkSources_855431_47.stderr parsl.dataflow.dflow INFO: Task 2722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18604c66e5be25f30c418f0f956fecd4 with result from task 2722 parsl.dataflow.dflow DEBUG: Task 2722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/494105fb-73fc-42c8-bcfc-ac62ad07af0c_isrForCrosstalkSources_855430_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/494105fb-73fc-42c8-bcfc-ac62ad07af0c_isrForCrosstalkSources_855430_50.stderr parsl.dataflow.dflow INFO: Task 3341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3647c15a5afd51cec07d9131a9bfeb6f with result from task 3341 parsl.dataflow.dflow DEBUG: Task 3341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/5700b2d9-e122-428c-98f5-558ba04ec00d_isrForCrosstalkSources_855423_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/5700b2d9-e122-428c-98f5-558ba04ec00d_isrForCrosstalkSources_855423_48.stderr parsl.dataflow.dflow INFO: Task 3258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e548a5bfaae82081eb5a49088cc5154e with result from task 3258 parsl.dataflow.dflow DEBUG: Task 3258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/453bfdbf-375e-4186-9163-c1744be546a5_isrForCrosstalkSources_855420_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/453bfdbf-375e-4186-9163-c1744be546a5_isrForCrosstalkSources_855420_14.stderr parsl.dataflow.dflow INFO: Task 3042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 077b75bd9b891cdaebc35fccc5317e0f with result from task 3042 parsl.dataflow.dflow DEBUG: Task 3042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9a2d515a-36a4-428c-9ff8-1916cf2738c8_isrForCrosstalkSources_855389_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9a2d515a-36a4-428c-9ff8-1916cf2738c8_isrForCrosstalkSources_855389_55.stderr parsl.dataflow.dflow INFO: Task 3227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adc14b354130b267513f82ef39f1e532 with result from task 3227 parsl.dataflow.dflow DEBUG: Task 3227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/48ebe5a6-102a-40a9-b4a8-75f5bd224caa_isrForCrosstalkSources_855397_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/48ebe5a6-102a-40a9-b4a8-75f5bd224caa_isrForCrosstalkSources_855397_8.stderr parsl.dataflow.dflow INFO: Task 3081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c724974e24ea43d0c72850a6f8505c with result from task 3081 parsl.dataflow.dflow DEBUG: Task 3081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c354db95-7e43-4db6-b10e-c746fb92e03f_isrForCrosstalkSources_855399_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/c354db95-7e43-4db6-b10e-c746fb92e03f_isrForCrosstalkSources_855399_54.stderr parsl.dataflow.dflow INFO: Task 3113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad0a32543c88f5ce72d79c3192af0db2 with result from task 3113 parsl.dataflow.dflow DEBUG: Task 3113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/19d7bf43-47ba-4194-868a-3e83f55ac243_isrForCrosstalkSources_855449_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/19d7bf43-47ba-4194-868a-3e83f55ac243_isrForCrosstalkSources_855449_14.stderr parsl.dataflow.dflow INFO: Task 3220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbdb3365b6d99ff65fbac108ef0646f8 with result from task 3220 parsl.dataflow.dflow DEBUG: Task 3220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a9dcd3d-0d81-427c-877a-a551b3563df2_isrForCrosstalkSources_855409_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a9dcd3d-0d81-427c-877a-a551b3563df2_isrForCrosstalkSources_855409_10.stderr parsl.dataflow.dflow INFO: Task 2992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a9fb8a8cecdf5d51d7cf5eec1386ef4 with result from task 2992 parsl.dataflow.dflow DEBUG: Task 2992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/6c023183-91b4-4da7-bf5f-02214072def7_isrForCrosstalkSources_855396_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/6c023183-91b4-4da7-bf5f-02214072def7_isrForCrosstalkSources_855396_58.stderr parsl.dataflow.dflow INFO: Task 3108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b0f12d8f4ada74d2e6c69d12e7948f0 with result from task 3108 parsl.dataflow.dflow DEBUG: Task 3108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5ca04777-70a4-4275-825c-1bfd9a66bb14_isrForCrosstalkSources_855407_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/5ca04777-70a4-4275-825c-1bfd9a66bb14_isrForCrosstalkSources_855407_12.stderr parsl.dataflow.dflow INFO: Task 3190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0ab080ded4a94cd5af1f77d51f35561 with result from task 3190 parsl.dataflow.dflow DEBUG: Task 3190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/988033b0-e16c-4e74-81a4-01a559c45d04_isrForCrosstalkSources_855440_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/988033b0-e16c-4e74-81a4-01a559c45d04_isrForCrosstalkSources_855440_51.stderr parsl.dataflow.dflow INFO: Task 3266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ad185b077b769f9776e73acc206d2c9 with result from task 3266 parsl.dataflow.dflow DEBUG: Task 3266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/577d4a80-272c-4522-86a4-25bdb6bda9e4_isrForCrosstalkSources_855400_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/577d4a80-272c-4522-86a4-25bdb6bda9e4_isrForCrosstalkSources_855400_34.stderr parsl.dataflow.dflow INFO: Task 3311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15126935e924ea7c6093e2cf94f5bc98 with result from task 3311 parsl.dataflow.dflow DEBUG: Task 3311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fe6c3c45-f3b1-4a7b-8e1d-0239a85b7fd5_isrForCrosstalkSources_855385_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fe6c3c45-f3b1-4a7b-8e1d-0239a85b7fd5_isrForCrosstalkSources_855385_29.stderr parsl.dataflow.dflow INFO: Task 3184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 368ab406d0f06617548bd52002b6fba3 with result from task 3184 parsl.dataflow.dflow DEBUG: Task 3184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d0961b78-a411-4800-b61e-cad83cf8106e_isrForCrosstalkSources_855389_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d0961b78-a411-4800-b61e-cad83cf8106e_isrForCrosstalkSources_855389_26.stderr parsl.dataflow.dflow INFO: Task 3127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3da5b7be1e7b34732320447e067d2c2e with result from task 3127 parsl.dataflow.dflow DEBUG: Task 3127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8e15fbda-a6cb-4a0b-9b5d-4818721b7ee8_isrForCrosstalkSources_855414_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8e15fbda-a6cb-4a0b-9b5d-4818721b7ee8_isrForCrosstalkSources_855414_1.stderr parsl.dataflow.dflow INFO: Task 3077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d62e8968d901d97a4011e9399bbce9c with result from task 3077 parsl.dataflow.dflow DEBUG: Task 3077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6c5da835-7bd4-4c98-b31a-b24e4fc46930_isrForCrosstalkSources_855412_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/6c5da835-7bd4-4c98-b31a-b24e4fc46930_isrForCrosstalkSources_855412_48.stderr parsl.dataflow.dflow INFO: Task 2953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae2f34bfea5a3eda44fc921523e9a84 with result from task 2953 parsl.dataflow.dflow DEBUG: Task 2953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3e32847b-bdf0-4f1a-b1b9-7f0e649fd223_isrForCrosstalkSources_855453_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3e32847b-bdf0-4f1a-b1b9-7f0e649fd223_isrForCrosstalkSources_855453_54.stderr parsl.dataflow.dflow INFO: Task 3259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da502217ab43eadf0931f56429edaf00 with result from task 3259 parsl.dataflow.dflow DEBUG: Task 3259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed2be071-0051-4713-99dc-ae6f7e449a13_isrForCrosstalkSources_855455_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ed2be071-0051-4713-99dc-ae6f7e449a13_isrForCrosstalkSources_855455_45.stderr parsl.dataflow.dflow INFO: Task 3200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69371c4da87acfe7721a313fe1668696 with result from task 3200 parsl.dataflow.dflow DEBUG: Task 3200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a9bf15d2-1648-4960-9e63-d1e8298385d8_isrForCrosstalkSources_855408_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a9bf15d2-1648-4960-9e63-d1e8298385d8_isrForCrosstalkSources_855408_51.stderr parsl.dataflow.dflow INFO: Task 3189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1759f88cbecf6a32fd16de16fcebc1c with result from task 3189 parsl.dataflow.dflow DEBUG: Task 3189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/46623f48-f95b-420b-80d3-cb470b12405e_isrForCrosstalkSources_855439_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/46623f48-f95b-420b-80d3-cb470b12405e_isrForCrosstalkSources_855439_45.stderr parsl.dataflow.dflow INFO: Task 3090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1ccb43a01662e1e8c40d8c6500a43a with result from task 3090 parsl.dataflow.dflow DEBUG: Task 3090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/47e26ca8-770a-4650-acc9-502b60959bb5_isrForCrosstalkSources_855394_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/47e26ca8-770a-4650-acc9-502b60959bb5_isrForCrosstalkSources_855394_54.stderr parsl.dataflow.dflow INFO: Task 3016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9886ae1ea83c30fffb5661306ac92a00 with result from task 3016 parsl.dataflow.dflow DEBUG: Task 3016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/75ac1539-149d-4131-8ea3-3b460ddc4313_isrForCrosstalkSources_855438_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/75ac1539-149d-4131-8ea3-3b460ddc4313_isrForCrosstalkSources_855438_58.stderr parsl.dataflow.dflow INFO: Task 2954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4b1edd54d04d44f26ae31e7d52c2169 with result from task 2954 parsl.dataflow.dflow DEBUG: Task 2954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/135e9e23-ff9d-4204-98a4-dcacfd683ae2_isrForCrosstalkSources_855455_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/135e9e23-ff9d-4204-98a4-dcacfd683ae2_isrForCrosstalkSources_855455_55.stderr parsl.dataflow.dflow INFO: Task 3440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 374ea5039284ace9c2497467980a0b01 with result from task 3440 parsl.dataflow.dflow DEBUG: Task 3440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/32f760e8-043c-4f7f-8699-200cda225b38_isrForCrosstalkSources_855428_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/32f760e8-043c-4f7f-8699-200cda225b38_isrForCrosstalkSources_855428_61.stderr parsl.dataflow.dflow INFO: Task 3245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5dccc61439221fc3ad4716d8b39525 with result from task 3245 parsl.dataflow.dflow DEBUG: Task 3245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8dc2818f-4db5-4aa4-874e-2e5066797d5a_isrForCrosstalkSources_855453_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8dc2818f-4db5-4aa4-874e-2e5066797d5a_isrForCrosstalkSources_855453_34.stderr parsl.dataflow.dflow INFO: Task 3212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38cee0b0800e6a2062944827883a5326 with result from task 3212 parsl.dataflow.dflow DEBUG: Task 3212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c1b84101-743f-4a65-ab9d-7c110ca7fe11_isrForCrosstalkSources_855421_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c1b84101-743f-4a65-ab9d-7c110ca7fe11_isrForCrosstalkSources_855421_11.stderr parsl.dataflow.dflow INFO: Task 2959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea3a3548a5e814d8ef376311c1d62a8 with result from task 2959 parsl.dataflow.dflow DEBUG: Task 2959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1b3a3a89-fea9-43ba-865f-2f26ec30ad52_isrForCrosstalkSources_855391_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1b3a3a89-fea9-43ba-865f-2f26ec30ad52_isrForCrosstalkSources_855391_49.stderr parsl.dataflow.dflow INFO: Task 3160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdaee9229635a81837a35202ba66ac9d with result from task 3160 parsl.dataflow.dflow DEBUG: Task 3160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11b27807-dbdf-4afe-9544-cef4ec6ff83e_isrForCrosstalkSources_855426_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11b27807-dbdf-4afe-9544-cef4ec6ff83e_isrForCrosstalkSources_855426_34.stderr parsl.dataflow.dflow INFO: Task 3403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53d90ec6b7f50c4940e0a1d6ab405fd with result from task 3403 parsl.dataflow.dflow DEBUG: Task 3403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/10dac8f3-51f7-4d55-874a-abeaefa8eba8_isrForCrosstalkSources_855391_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/10dac8f3-51f7-4d55-874a-abeaefa8eba8_isrForCrosstalkSources_855391_5.stderr parsl.dataflow.dflow INFO: Task 3222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c7880ff6937a65890d67a935f203ef6 with result from task 3222 parsl.dataflow.dflow DEBUG: Task 3222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fa784283-9a02-45ac-a008-562c75d4a439_isrForCrosstalkSources_855418_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fa784283-9a02-45ac-a008-562c75d4a439_isrForCrosstalkSources_855418_1.stderr parsl.dataflow.dflow INFO: Task 3179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a3304701f7e16527905693eea88e271 with result from task 3179 parsl.dataflow.dflow DEBUG: Task 3179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6d54d8cb-f27a-4b87-8bda-5042e4ccf43a_isrForCrosstalkSources_855415_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/6d54d8cb-f27a-4b87-8bda-5042e4ccf43a_isrForCrosstalkSources_855415_42.stderr parsl.dataflow.dflow INFO: Task 3357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1aed864508a9b3d8d0a9997fca68b2b with result from task 3357 parsl.dataflow.dflow DEBUG: Task 3357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/24093dfa-1f4d-477d-a510-7ea42bb9190d_isrForCrosstalkSources_855455_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/24093dfa-1f4d-477d-a510-7ea42bb9190d_isrForCrosstalkSources_855455_34.stderr parsl.dataflow.dflow INFO: Task 3142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e63944d71ac8a68d083b2dc704a748d3 with result from task 3142 parsl.dataflow.dflow DEBUG: Task 3142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5da6fbfc-4554-4bac-b9e8-2ae4849cf266_isrForCrosstalkSources_855444_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5da6fbfc-4554-4bac-b9e8-2ae4849cf266_isrForCrosstalkSources_855444_41.stderr parsl.dataflow.dflow INFO: Task 3194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41a3f17878f7375518668a024ef2b75 with result from task 3194 parsl.dataflow.dflow DEBUG: Task 3194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ea1dc659-90fa-4ba0-8980-2ef017ee73a4_isrForCrosstalkSources_855383_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ea1dc659-90fa-4ba0-8980-2ef017ee73a4_isrForCrosstalkSources_855383_37.stderr parsl.dataflow.dflow INFO: Task 3467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b24f687a331f61497007f06d67a93d2 with result from task 3467 parsl.dataflow.dflow DEBUG: Task 3467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2a8bd97-0e06-4552-8b62-d9c87d48ad06_isrForCrosstalkSources_855402_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/f2a8bd97-0e06-4552-8b62-d9c87d48ad06_isrForCrosstalkSources_855402_13.stderr parsl.dataflow.dflow INFO: Task 3351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8df01bbd456ce86ecbee0ea8cf0e6db3 with result from task 3351 parsl.dataflow.dflow DEBUG: Task 3351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e3fbfeea-08a1-4366-9d3c-88e819d73968_isrForCrosstalkSources_855391_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e3fbfeea-08a1-4366-9d3c-88e819d73968_isrForCrosstalkSources_855391_40.stderr parsl.dataflow.dflow INFO: Task 3424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38d1362a4669da004af1a102001b471a with result from task 3424 parsl.dataflow.dflow DEBUG: Task 3424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e50a56aa-3d4d-430d-8cc2-aa836de1be3d_isrForCrosstalkSources_855430_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e50a56aa-3d4d-430d-8cc2-aa836de1be3d_isrForCrosstalkSources_855430_37.stderr parsl.dataflow.dflow INFO: Task 3202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feea43eb493d8fb0bf4d36ee6f5ab0e0 with result from task 3202 parsl.dataflow.dflow DEBUG: Task 3202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e9b9c204-1019-46e2-b038-6616021399c5_isrForCrosstalkSources_855451_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e9b9c204-1019-46e2-b038-6616021399c5_isrForCrosstalkSources_855451_39.stderr parsl.dataflow.dflow INFO: Task 3354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 631008413c118d7321b2aa37165c4d44 with result from task 3354 2024-03-09 14:54:33 proc_lsst.multi:170 [INFO] [multi] job_id 16964284 proc_lsst.multi INFO: [multi] job_id 16964284 2024-03-09 14:54:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 195 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 195 2024-03-09 14:54:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964284 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964284 parsl.executors.status_handling DEBUG: Launched block 250 on executor multi with job ID 16964284 parsl.executors.status_handling INFO: Allocated block ID 251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-251 2024-03-09 14:54:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 2024-03-09 14:54:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 195 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 195 2024-03-09 14:54:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/f26ef99f-5411-4801-9458-7392eaa1149e_isrForCrosstalkSources_855415_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/f26ef99f-5411-4801-9458-7392eaa1149e_isrForCrosstalkSources_855415_56.stderr parsl.dataflow.dflow INFO: Task 3182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 207b26957c8c2b982f84e34ba656e474 with result from task 3182 parsl.dataflow.dflow DEBUG: Task 3182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3cc3d15a-3f84-41d8-a77b-f4847dd241c3_isrForCrosstalkSources_855424_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3cc3d15a-3f84-41d8-a77b-f4847dd241c3_isrForCrosstalkSources_855424_44.stderr parsl.dataflow.dflow INFO: Task 3437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9900fb052453001a91d60febd16f8ed0 with result from task 3437 parsl.dataflow.dflow DEBUG: Task 3437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0cd0dc86-c5da-4194-b0e8-69d7ff744a85_isrForCrosstalkSources_855383_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/0cd0dc86-c5da-4194-b0e8-69d7ff744a85_isrForCrosstalkSources_855383_40.stderr parsl.dataflow.dflow INFO: Task 3133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01745799e1ba4f117e137aa63169a09f with result from task 3133 parsl.dataflow.dflow DEBUG: Task 3133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bc24003f-15dd-4bda-8ebc-d74dab03707c_isrForCrosstalkSources_855386_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/bc24003f-15dd-4bda-8ebc-d74dab03707c_isrForCrosstalkSources_855386_28.stderr parsl.dataflow.dflow INFO: Task 3251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03c3c545f8ea2ca245f6df662829b137 with result from task 3251 parsl.dataflow.dflow DEBUG: Task 3251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cd9e5c1d-5b40-4c77-bc4b-150feb546ee4_isrForCrosstalkSources_855409_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/cd9e5c1d-5b40-4c77-bc4b-150feb546ee4_isrForCrosstalkSources_855409_32.stderr parsl.dataflow.dflow INFO: Task 3139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a25da4789a10d3d3621c2ff6460fc1fc with result from task 3139 parsl.dataflow.dflow DEBUG: Task 3139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/9d5a393f-8cd1-4e34-b786-2a4239b75dde_isrForCrosstalkSources_855419_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/9d5a393f-8cd1-4e34-b786-2a4239b75dde_isrForCrosstalkSources_855419_3.stderr parsl.dataflow.dflow INFO: Task 3404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 248644aaf35cd09b32a1d2b08af58a09 with result from task 3404 parsl.dataflow.dflow DEBUG: Task 3404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c67453f4-5275-443b-9e5a-b67b6094b44b_isrForCrosstalkSources_855389_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c67453f4-5275-443b-9e5a-b67b6094b44b_isrForCrosstalkSources_855389_46.stderr parsl.dataflow.dflow INFO: Task 3264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9f7b98f150f979b6dfa1314bdf6d324 with result from task 3264 parsl.dataflow.dflow DEBUG: Task 3264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b9a228c2-4de1-4e93-b8ba-877293c45e3d_isrForCrosstalkSources_855454_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b9a228c2-4de1-4e93-b8ba-877293c45e3d_isrForCrosstalkSources_855454_19.stderr parsl.dataflow.dflow INFO: Task 3470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e628f8e1217b798d72b619e9fb34fdb with result from task 3470 parsl.dataflow.dflow DEBUG: Task 3470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/55ec0e6e-2eae-473a-98b4-bc4f9db61dfc_isrForCrosstalkSources_855400_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/55ec0e6e-2eae-473a-98b4-bc4f9db61dfc_isrForCrosstalkSources_855400_22.stderr parsl.dataflow.dflow INFO: Task 3060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07acf34286f5bd5ef236af8a2f7872a8 with result from task 3060 parsl.dataflow.dflow DEBUG: Task 3060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ba79794e-922a-445d-b231-d5ecb0b92aa3_isrForCrosstalkSources_855439_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/ba79794e-922a-445d-b231-d5ecb0b92aa3_isrForCrosstalkSources_855439_60.stderr parsl.dataflow.dflow INFO: Task 3186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfe33d7e004494cb9ee17f294068d86d with result from task 3186 parsl.dataflow.dflow DEBUG: Task 3186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/05e77e9c-b34b-4aba-a2ed-5dedace27adf_isrForCrosstalkSources_855459_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/05e77e9c-b34b-4aba-a2ed-5dedace27adf_isrForCrosstalkSources_855459_43.stderr parsl.dataflow.dflow INFO: Task 3450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94d550d30dcc0a09aadd6bb3a3727193 with result from task 3450 parsl.dataflow.dflow DEBUG: Task 3450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3b2d67d5-15fd-4c28-a0db-616ff937f631_isrForCrosstalkSources_855429_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3b2d67d5-15fd-4c28-a0db-616ff937f631_isrForCrosstalkSources_855429_7.stderr parsl.dataflow.dflow INFO: Task 3154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0a0c7926fa467b993b5c2f1c7e9e5f1 with result from task 3154 parsl.dataflow.dflow DEBUG: Task 3154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/853e3782-652f-4dd3-9fd2-7c2d7bcee168_isrForCrosstalkSources_855415_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/853e3782-652f-4dd3-9fd2-7c2d7bcee168_isrForCrosstalkSources_855415_3.stderr parsl.dataflow.dflow INFO: Task 3225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08022ff6b6c1b4dcfaa3acfad57e707f with result from task 3225 parsl.dataflow.dflow DEBUG: Task 3225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a41bf571-cd49-4591-8719-14286d9aa538_isrForCrosstalkSources_855390_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/a41bf571-cd49-4591-8719-14286d9aa538_isrForCrosstalkSources_855390_33.stderr parsl.dataflow.dflow INFO: Task 3458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2317590b4d0c7f9e54fad753bcbfe0d with result from task 3458 parsl.dataflow.dflow DEBUG: Task 3458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/984f5a86-869f-4b02-a5d2-0b31ec7f7f89_isrForCrosstalkSources_855400_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/984f5a86-869f-4b02-a5d2-0b31ec7f7f89_isrForCrosstalkSources_855400_21.stderr parsl.dataflow.dflow INFO: Task 3164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9f21132d8a38ee789e7c575aa32b08d with result from task 3164 parsl.dataflow.dflow DEBUG: Task 3164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/836c955a-8f7b-4146-b905-6863a2bc45e5_isrForCrosstalkSources_855454_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/836c955a-8f7b-4146-b905-6863a2bc45e5_isrForCrosstalkSources_855454_21.stderr parsl.dataflow.dflow INFO: Task 3135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb6c269767549469c9d1bd0717f44443 with result from task 3135 parsl.dataflow.dflow DEBUG: Task 3135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/da2b94dc-ecee-4d84-b652-5d5831c5e327_isrForCrosstalkSources_855424_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/da2b94dc-ecee-4d84-b652-5d5831c5e327_isrForCrosstalkSources_855424_60.stderr parsl.dataflow.dflow INFO: Task 3368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad488ca53868a708ddab0b135baa5278 with result from task 3368 parsl.dataflow.dflow DEBUG: Task 3368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/27f95e20-8453-4972-847f-0837d2bd09d1_isrForCrosstalkSources_855439_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/27f95e20-8453-4972-847f-0837d2bd09d1_isrForCrosstalkSources_855439_5.stderr parsl.dataflow.dflow INFO: Task 3132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b36122832d4e6fe28e988729e9627e1 with result from task 3132 parsl.dataflow.dflow DEBUG: Task 3132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4c60a38c-b757-4cb1-abe5-c1cd681ae9db_isrForCrosstalkSources_855426_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/4c60a38c-b757-4cb1-abe5-c1cd681ae9db_isrForCrosstalkSources_855426_55.stderr parsl.dataflow.dflow INFO: Task 3233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 259bd14771ef4403d73dfef1eaa939e9 with result from task 3233 parsl.dataflow.dflow DEBUG: Task 3233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/722d0f14-1333-4133-b0bd-bc3ebce8487d_isrForCrosstalkSources_855388_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/722d0f14-1333-4133-b0bd-bc3ebce8487d_isrForCrosstalkSources_855388_47.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51c4a008b0b91d3170ccee7c72927f96 with result from task 3187 parsl.dataflow.dflow DEBUG: Task 3187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ab2ee0b6-6fe6-453a-b83a-48c7824426b5_isrForCrosstalkSources_855411_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/ab2ee0b6-6fe6-453a-b83a-48c7824426b5_isrForCrosstalkSources_855411_30.stderr parsl.dataflow.dflow INFO: Task 3218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0c7ae63b05175bd6df04c76d7ad48b with result from task 3218 parsl.dataflow.dflow DEBUG: Task 3218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a31cc8-b4d5-4f60-99d4-35cfb35a5697_isrForCrosstalkSources_855391_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b8a31cc8-b4d5-4f60-99d4-35cfb35a5697_isrForCrosstalkSources_855391_58.stderr parsl.dataflow.dflow INFO: Task 3334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da4765bbac46ea9d44d652304e371d6d with result from task 3334 parsl.dataflow.dflow DEBUG: Task 3334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d7c6a38a-14cc-4882-a742-76d60faf9017_isrForCrosstalkSources_855438_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d7c6a38a-14cc-4882-a742-76d60faf9017_isrForCrosstalkSources_855438_4.stderr parsl.dataflow.dflow INFO: Task 3214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ddf7ac90be41d6f4046e3a2ede224b2 with result from task 3214 parsl.dataflow.dflow DEBUG: Task 3214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/81806efd-31cb-47e1-8979-2644d4c6924a_isrForCrosstalkSources_855415_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/81806efd-31cb-47e1-8979-2644d4c6924a_isrForCrosstalkSources_855415_61.stderr parsl.dataflow.dflow INFO: Task 3493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bca4474c3d2f4a09a15e0379ed0a5d0 with result from task 3493 parsl.dataflow.dflow DEBUG: Task 3493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/be143c66-a5f8-4c88-89e1-2eb1aa80f41a_isrForCrosstalkSources_855392_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/be143c66-a5f8-4c88-89e1-2eb1aa80f41a_isrForCrosstalkSources_855392_8.stderr parsl.dataflow.dflow INFO: Task 3398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bda2473b83656e7a22d5aecf4739ba4 with result from task 3398 parsl.dataflow.dflow DEBUG: Task 3398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6d12d94-8fd4-444e-be24-fc715c82337f_isrForCrosstalkSources_855412_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6d12d94-8fd4-444e-be24-fc715c82337f_isrForCrosstalkSources_855412_17.stderr parsl.dataflow.dflow INFO: Task 3185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e56b78a87bc061593a8fec0050636a with result from task 3185 parsl.dataflow.dflow DEBUG: Task 3185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d716a157-5d95-4453-85bb-b06dd7f82de4_isrForCrosstalkSources_855457_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d716a157-5d95-4453-85bb-b06dd7f82de4_isrForCrosstalkSources_855457_42.stderr parsl.dataflow.dflow INFO: Task 3459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b52956f73a50e2fd8a9b928e90e32c27 with result from task 3459 parsl.dataflow.dflow DEBUG: Task 3459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3eb45eb1-2efd-48fa-9625-df920e9c0fe2_isrForCrosstalkSources_855402_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3eb45eb1-2efd-48fa-9625-df920e9c0fe2_isrForCrosstalkSources_855402_3.stderr parsl.dataflow.dflow INFO: Task 3210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8754b03d7a48b217140e30416b31c584 with result from task 3210 parsl.dataflow.dflow DEBUG: Task 3210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6cdc3705-e164-48e4-b8f6-dd789c590a05_isrForCrosstalkSources_855417_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6cdc3705-e164-48e4-b8f6-dd789c590a05_isrForCrosstalkSources_855417_1.stderr parsl.dataflow.dflow INFO: Task 3340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df4a1d4184b20f6b95993b219640abe9 with result from task 3340 parsl.dataflow.dflow DEBUG: Task 3340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d3c25951-4fa8-47f6-a0db-fb27a811949d_isrForCrosstalkSources_855439_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/d3c25951-4fa8-47f6-a0db-fb27a811949d_isrForCrosstalkSources_855439_22.stderr parsl.dataflow.dflow INFO: Task 3339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b154154ca28cedfac2ac6ec811989865 with result from task 3339 parsl.dataflow.dflow DEBUG: Task 3339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/67435a15-511c-4ff0-b467-3dfea52e54e6_isrForCrosstalkSources_855440_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/67435a15-511c-4ff0-b467-3dfea52e54e6_isrForCrosstalkSources_855440_38.stderr parsl.dataflow.dflow INFO: Task 3435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee8d4f74476e2c21b0c876df32e5f9df with result from task 3435 parsl.dataflow.dflow DEBUG: Task 3435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ba56838-af5b-4dab-8ece-4d94e75869b8_isrForCrosstalkSources_855442_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/5ba56838-af5b-4dab-8ece-4d94e75869b8_isrForCrosstalkSources_855442_51.stderr parsl.dataflow.dflow INFO: Task 3367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56aa915d2bd60ccf0867b0624604c7ee with result from task 3367 parsl.dataflow.dflow DEBUG: Task 3367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/be2beb1e-96b1-4d84-bacc-a3903841e90e_isrForCrosstalkSources_855385_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/be2beb1e-96b1-4d84-bacc-a3903841e90e_isrForCrosstalkSources_855385_27.stderr parsl.dataflow.dflow INFO: Task 3462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 341f27e7db41c99e32172e1ecb21b843 with result from task 3462 parsl.dataflow.dflow DEBUG: Task 3462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a3f82cc8-a7d7-4585-aa70-e0e00d23ab7a_isrForCrosstalkSources_855426_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a3f82cc8-a7d7-4585-aa70-e0e00d23ab7a_isrForCrosstalkSources_855426_29.stderr parsl.dataflow.dflow INFO: Task 3242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25147841631ecf8c231ed418464af47 with result from task 3242 parsl.dataflow.dflow DEBUG: Task 3242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/60826d6a-b235-4622-939d-93f808a52157_isrForCrosstalkSources_855388_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/60826d6a-b235-4622-939d-93f808a52157_isrForCrosstalkSources_855388_25.stderr parsl.dataflow.dflow INFO: Task 3211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3404eec7790213bdebcceba95b752c2 with result from task 3211 parsl.dataflow.dflow DEBUG: Task 3211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e75126fc-c7fc-4052-b59a-57b8d7f44e98_isrForCrosstalkSources_855445_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e75126fc-c7fc-4052-b59a-57b8d7f44e98_isrForCrosstalkSources_855445_36.stderr parsl.dataflow.dflow INFO: Task 3307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c8be923fef3d460b5b843c1764a6267 with result from task 3307 parsl.dataflow.dflow DEBUG: Task 3307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c9a6ccb3-5f20-4010-af03-f3d2abb4531d_isrForCrosstalkSources_855421_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c9a6ccb3-5f20-4010-af03-f3d2abb4531d_isrForCrosstalkSources_855421_26.stderr parsl.dataflow.dflow INFO: Task 3299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67a1fbd105aff003c94ef10c33fbc9bf with result from task 3299 parsl.dataflow.dflow DEBUG: Task 3299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/7c9c86c1-e793-4ee5-8658-7bf9d96a5c68_isrForCrosstalkSources_855395_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/7c9c86c1-e793-4ee5-8658-7bf9d96a5c68_isrForCrosstalkSources_855395_13.stderr parsl.dataflow.dflow INFO: Task 3243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b0f9e731f88fb657dd8d388c174ebdf with result from task 3243 parsl.dataflow.dflow DEBUG: Task 3243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7d6bc5f7-7ea2-44ba-a386-fc4cf159ac95_isrForCrosstalkSources_855380_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/7d6bc5f7-7ea2-44ba-a386-fc4cf159ac95_isrForCrosstalkSources_855380_24.stderr parsl.dataflow.dflow INFO: Task 3468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3af3e1825176e857bc280c3a9dd785d6 with result from task 3468 parsl.dataflow.dflow DEBUG: Task 3468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6279700-b9a7-4883-a26f-886b28501392_isrForCrosstalkSources_855412_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/a6279700-b9a7-4883-a26f-886b28501392_isrForCrosstalkSources_855412_28.stderr parsl.dataflow.dflow INFO: Task 3407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e7ef7f4ee12cf83dc22ec49b55288d4 with result from task 3407 parsl.dataflow.dflow DEBUG: Task 3407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/69966d3a-1bb9-4fb8-b22d-d0c73e63d892_isrForCrosstalkSources_855410_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/69966d3a-1bb9-4fb8-b22d-d0c73e63d892_isrForCrosstalkSources_855410_26.stderr parsl.dataflow.dflow INFO: Task 3474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24cbb86368ee23dbe0db9b56f6651897 with result from task 3474 parsl.dataflow.dflow DEBUG: Task 3474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/50480b30-9bb0-4f43-8329-bf71cec85de3_isrForCrosstalkSources_855406_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/50480b30-9bb0-4f43-8329-bf71cec85de3_isrForCrosstalkSources_855406_36.stderr parsl.dataflow.dflow INFO: Task 3253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53b42505f21b447de559c180474740f2 with result from task 3253 parsl.dataflow.dflow DEBUG: Task 3253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0aaa98c7-d054-4565-b89f-e2d6a371a7ac_isrForCrosstalkSources_855417_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/0aaa98c7-d054-4565-b89f-e2d6a371a7ac_isrForCrosstalkSources_855417_35.stderr parsl.dataflow.dflow INFO: Task 3216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5423d9c7bb1dec5287d9bf508330246 with result from task 3216 parsl.dataflow.dflow DEBUG: Task 3216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0bf3b81d-b0a1-467a-a3f7-cf677f301da1_isrForCrosstalkSources_855426_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0bf3b81d-b0a1-467a-a3f7-cf677f301da1_isrForCrosstalkSources_855426_32.stderr parsl.dataflow.dflow INFO: Task 3423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06abe4f24497e13954dbfc045f16ced5 with result from task 3423 parsl.dataflow.dflow DEBUG: Task 3423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/71f23d17-183b-4a79-add2-3eee295e02f8_isrForCrosstalkSources_855455_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/71f23d17-183b-4a79-add2-3eee295e02f8_isrForCrosstalkSources_855455_23.stderr parsl.dataflow.dflow INFO: Task 3276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29a89ce91564b54ca304feb561a9a87a with result from task 3276 parsl.dataflow.dflow DEBUG: Task 3276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e3c96271-9e98-4811-a02d-90f80ccc1287_isrForCrosstalkSources_855435_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/e3c96271-9e98-4811-a02d-90f80ccc1287_isrForCrosstalkSources_855435_8.stderr 2024-03-09 14:54:35 proc_lsst.multi:170 [INFO] [multi] job_id 16964286 proc_lsst.multi INFO: [multi] job_id 16964286 2024-03-09 14:54:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 196 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 196 2024-03-09 14:54:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964286 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964286 parsl.executors.status_handling DEBUG: Launched block 251 on executor multi with job ID 16964286 parsl.executors.status_handling INFO: Allocated block ID 252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-252 2024-03-09 14:54:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 2024-03-09 14:54:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 196 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 196 2024-03-09 14:54:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6167aa54134bf75f69c26941dc306650 with result from task 3335 parsl.dataflow.dflow DEBUG: Task 3335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c023d42e-59b1-41d9-8ade-1298a7d9456d_isrForCrosstalkSources_855456_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/c023d42e-59b1-41d9-8ade-1298a7d9456d_isrForCrosstalkSources_855456_25.stderr parsl.dataflow.dflow INFO: Task 3439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f87131eb4e2b5395e250cba6e2f5ef1f with result from task 3439 parsl.dataflow.dflow DEBUG: Task 3439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/affa7222-7884-44ab-b5db-d7cbf821b1b2_isrForCrosstalkSources_855390_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/affa7222-7884-44ab-b5db-d7cbf821b1b2_isrForCrosstalkSources_855390_19.stderr parsl.dataflow.dflow INFO: Task 3297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a521b03cd1c04ddb29ff481abc24890 with result from task 3297 parsl.dataflow.dflow DEBUG: Task 3297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6565babd-aef0-4e70-8e1a-e28032c8600a_isrForCrosstalkSources_855380_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/6565babd-aef0-4e70-8e1a-e28032c8600a_isrForCrosstalkSources_855380_20.stderr parsl.dataflow.dflow INFO: Task 3041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a9af6e12b21a5cc586c7bc8b1223bf2 with result from task 3041 parsl.dataflow.dflow DEBUG: Task 3041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e120e213-b551-4c37-a4e5-6d21d2746b5a_isrForCrosstalkSources_855379_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/e120e213-b551-4c37-a4e5-6d21d2746b5a_isrForCrosstalkSources_855379_59.stderr parsl.dataflow.dflow INFO: Task 3163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca1386793ef46a2bbad42fd07e42d3ba with result from task 3163 parsl.dataflow.dflow DEBUG: Task 3163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/7d45e839-aebd-4405-9d8c-72a4149f2fee_isrForCrosstalkSources_855435_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/7d45e839-aebd-4405-9d8c-72a4149f2fee_isrForCrosstalkSources_855435_60.stderr parsl.dataflow.dflow INFO: Task 3397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc812d81f10490f8e17cf899b31c12a with result from task 3397 parsl.dataflow.dflow DEBUG: Task 3397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5b77f20-46e3-4d6d-a39b-640c70576a84_isrForCrosstalkSources_855456_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5b77f20-46e3-4d6d-a39b-640c70576a84_isrForCrosstalkSources_855456_5.stderr parsl.dataflow.dflow INFO: Task 3342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a02f71c5322e229fcfaf29553f55ed with result from task 3342 parsl.dataflow.dflow DEBUG: Task 3342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0fa7e19a-79a3-4b3a-83c2-0687b45a7abf_isrForCrosstalkSources_855457_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/0fa7e19a-79a3-4b3a-83c2-0687b45a7abf_isrForCrosstalkSources_855457_14.stderr parsl.dataflow.dflow INFO: Task 3391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c612265e0e8c2d1abe2bde773cc28f5f with result from task 3391 parsl.dataflow.dflow DEBUG: Task 3391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7eb0d558-fc64-4987-ba09-f731e13f89b0_isrForCrosstalkSources_855386_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/7eb0d558-fc64-4987-ba09-f731e13f89b0_isrForCrosstalkSources_855386_44.stderr parsl.dataflow.dflow INFO: Task 3191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03a61fc8f2f032e194318ded02a92ad0 with result from task 3191 parsl.dataflow.dflow DEBUG: Task 3191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2fcdadca-4bbe-4417-a1d5-6b0e60a8b005_isrForCrosstalkSources_855442_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2fcdadca-4bbe-4417-a1d5-6b0e60a8b005_isrForCrosstalkSources_855442_35.stderr parsl.dataflow.dflow INFO: Task 3401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6cf237806ce6f3e16e74477b9d66197 with result from task 3401 parsl.dataflow.dflow DEBUG: Task 3401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f4386468-35f5-4f59-99e4-f4249136db28_isrForCrosstalkSources_855393_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/f4386468-35f5-4f59-99e4-f4249136db28_isrForCrosstalkSources_855393_46.stderr parsl.dataflow.dflow INFO: Task 3183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b78c1d471aad115b462e65f63208d63 with result from task 3183 parsl.dataflow.dflow DEBUG: Task 3183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ded8ac30-a29a-476e-9133-10af376514ec_isrForCrosstalkSources_855437_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ded8ac30-a29a-476e-9133-10af376514ec_isrForCrosstalkSources_855437_30.stderr parsl.dataflow.dflow INFO: Task 3231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4fd932c39c88f285dc818fd3275de11 with result from task 3231 parsl.dataflow.dflow DEBUG: Task 3231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/71f63767-67ca-4b66-b433-23d2039876d8_isrForCrosstalkSources_855379_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/71f63767-67ca-4b66-b433-23d2039876d8_isrForCrosstalkSources_855379_18.stderr parsl.dataflow.dflow INFO: Task 3272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 397e6278ef217732f429cc6e87512c61 with result from task 3272 parsl.dataflow.dflow DEBUG: Task 3272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/160ca8d3-9622-42e6-b54d-5ce55b7bbbe8_isrForCrosstalkSources_855438_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/160ca8d3-9622-42e6-b54d-5ce55b7bbbe8_isrForCrosstalkSources_855438_57.stderr parsl.dataflow.dflow INFO: Task 3384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 506c7cf1498a76c1bfd6dd298d826f39 with result from task 3384 parsl.dataflow.dflow DEBUG: Task 3384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/54c0b434-5401-49e8-8945-216d5043067c_isrForCrosstalkSources_855427_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/54c0b434-5401-49e8-8945-216d5043067c_isrForCrosstalkSources_855427_9.stderr parsl.dataflow.dflow INFO: Task 3250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7f102bcedfe8c924dce5f9155023d6 with result from task 3250 parsl.dataflow.dflow DEBUG: Task 3250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/183f17c2-4bfc-4d26-9001-7205f4206992_isrForCrosstalkSources_855390_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/183f17c2-4bfc-4d26-9001-7205f4206992_isrForCrosstalkSources_855390_39.stderr parsl.dataflow.dflow INFO: Task 3206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4300f048e93d298bdd16344bab81400 with result from task 3206 parsl.dataflow.dflow DEBUG: Task 3206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6fc6a114-9ce6-4ce9-ad7e-c5f4cf67e9c7_isrForCrosstalkSources_855447_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6fc6a114-9ce6-4ce9-ad7e-c5f4cf67e9c7_isrForCrosstalkSources_855447_23.stderr parsl.dataflow.dflow INFO: Task 3312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 346d4658336f0369465ebcc918a06ae0 with result from task 3312 parsl.dataflow.dflow DEBUG: Task 3312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/65117874-6b8e-49ca-9379-a6afcda90585_isrForCrosstalkSources_855457_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/65117874-6b8e-49ca-9379-a6afcda90585_isrForCrosstalkSources_855457_39.stderr parsl.dataflow.dflow INFO: Task 3286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1107c29669bcc1e5be171bd88beaf797 with result from task 3286 parsl.dataflow.dflow DEBUG: Task 3286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ffbd624b-bc17-4dbf-b53d-885172eb654f_isrForCrosstalkSources_855398_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/ffbd624b-bc17-4dbf-b53d-885172eb654f_isrForCrosstalkSources_855398_26.stderr parsl.dataflow.dflow INFO: Task 3453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adef6f68d621306c40b31b7a5c4c028a with result from task 3453 parsl.dataflow.dflow DEBUG: Task 3453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/e19f3b26-cd27-4d92-ab3c-5dd203c8eade_isrForCrosstalkSources_855446_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/e19f3b26-cd27-4d92-ab3c-5dd203c8eade_isrForCrosstalkSources_855446_34.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61fa030a397f79f380f61a3a10aafc5 with result from task 3444 parsl.dataflow.dflow DEBUG: Task 3444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/26c17532-63fa-41b7-b9b6-6d6f698c4999_isrForCrosstalkSources_855400_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/26c17532-63fa-41b7-b9b6-6d6f698c4999_isrForCrosstalkSources_855400_20.stderr parsl.dataflow.dflow INFO: Task 3205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c8b71e255a9ffcd09679f63138723b with result from task 3205 parsl.dataflow.dflow DEBUG: Task 3205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/23baf001-7978-4ede-ad42-1410b03ada2f_isrForCrosstalkSources_855417_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/23baf001-7978-4ede-ad42-1410b03ada2f_isrForCrosstalkSources_855417_8.stderr parsl.dataflow.dflow INFO: Task 3434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9a297d98a38916735f194682fa8643a with result from task 3434 parsl.dataflow.dflow DEBUG: Task 3434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5095af1-3adf-46a7-a434-b4244137a1b4_isrForCrosstalkSources_855421_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5095af1-3adf-46a7-a434-b4244137a1b4_isrForCrosstalkSources_855421_34.stderr parsl.dataflow.dflow INFO: Task 3314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4cdca9757a1d2c7b34375f3f6040023 with result from task 3314 parsl.dataflow.dflow DEBUG: Task 3314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b43d4b5c-8b36-4ba9-9d97-17ff7d607681_isrForCrosstalkSources_855448_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b43d4b5c-8b36-4ba9-9d97-17ff7d607681_isrForCrosstalkSources_855448_25.stderr parsl.dataflow.dflow INFO: Task 3238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7e2ba6ab998ab6ca728a0221993c0e4 with result from task 3238 parsl.dataflow.dflow DEBUG: Task 3238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/03a54ed6-ddff-413f-a546-4ef396d2485c_isrForCrosstalkSources_855390_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/03a54ed6-ddff-413f-a546-4ef396d2485c_isrForCrosstalkSources_855390_41.stderr parsl.dataflow.dflow INFO: Task 3162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc642c1625bfb190ba2426e889c98357 with result from task 3162 parsl.dataflow.dflow DEBUG: Task 3162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/65e8dd92-84a0-4604-9114-a2a252649ff9_isrForCrosstalkSources_855417_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/65e8dd92-84a0-4604-9114-a2a252649ff9_isrForCrosstalkSources_855417_28.stderr parsl.dataflow.dflow INFO: Task 3235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a171aa306502bf9e866a75f56419b64 with result from task 3235 parsl.dataflow.dflow DEBUG: Task 3235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2f298ecd-5578-4216-9a85-75e0c299c17f_isrForCrosstalkSources_855457_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2f298ecd-5578-4216-9a85-75e0c299c17f_isrForCrosstalkSources_855457_58.stderr 2024-03-09 14:54:36 proc_lsst.multi:170 [INFO] [multi] job_id 16964287 proc_lsst.multi INFO: [multi] job_id 16964287 2024-03-09 14:54:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 197 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 197 2024-03-09 14:54:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964287 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964287 parsl.executors.status_handling DEBUG: Launched block 252 on executor multi with job ID 16964287 parsl.executors.status_handling INFO: Allocated block ID 253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-253 2024-03-09 14:54:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 2024-03-09 14:54:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 197 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 197 2024-03-09 14:54:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc4b2d95d6d0697ae5d59a36117f8983 with result from task 3217 parsl.dataflow.dflow DEBUG: Task 3217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2c228c88-57ea-4181-a5a9-0303b75bc56b_isrForCrosstalkSources_855392_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2c228c88-57ea-4181-a5a9-0303b75bc56b_isrForCrosstalkSources_855392_47.stderr parsl.dataflow.dflow INFO: Task 3382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24457b5358b43ca3a39e060f9433e80f with result from task 3382 parsl.dataflow.dflow DEBUG: Task 3382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3e1d5f5-cb20-4b40-980e-c61a56917b8e_isrForCrosstalkSources_855414_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3e1d5f5-cb20-4b40-980e-c61a56917b8e_isrForCrosstalkSources_855414_2.stderr parsl.dataflow.dflow INFO: Task 3406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9fb32357d3f69f6d75298f67d81a4c1 with result from task 3406 parsl.dataflow.dflow DEBUG: Task 3406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/abe884fc-730f-4efe-b1c7-fc4e383f5e43_isrForCrosstalkSources_855407_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/abe884fc-730f-4efe-b1c7-fc4e383f5e43_isrForCrosstalkSources_855407_35.stderr parsl.dataflow.dflow INFO: Task 3279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39955790b5519373e6fc4995050ed8be with result from task 3279 parsl.dataflow.dflow DEBUG: Task 3279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/e683f363-f3e5-4d34-967c-36d087b1e99d_isrForCrosstalkSources_855442_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/e683f363-f3e5-4d34-967c-36d087b1e99d_isrForCrosstalkSources_855442_33.stderr parsl.dataflow.dflow INFO: Task 3370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 161ed729d2aebd220ca0bda435f0439d with result from task 3370 parsl.dataflow.dflow DEBUG: Task 3370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9277b11c-f790-456b-8478-c4bc9a5729f3_isrForCrosstalkSources_855396_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9277b11c-f790-456b-8478-c4bc9a5729f3_isrForCrosstalkSources_855396_14.stderr parsl.dataflow.dflow INFO: Task 3422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a6835eff538c5939fd207a85e56bc76 with result from task 3422 parsl.dataflow.dflow DEBUG: Task 3422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/64fa625a-9205-40fc-83d9-4f1c9068ced2_isrForCrosstalkSources_855409_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/64fa625a-9205-40fc-83d9-4f1c9068ced2_isrForCrosstalkSources_855409_58.stderr parsl.dataflow.dflow INFO: Task 3390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 066afb20fe0bfe41e2ac78dcd7510413 with result from task 3390 parsl.dataflow.dflow DEBUG: Task 3390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b9e8ff41-4703-4740-8356-2a69b46b1bd5_isrForCrosstalkSources_855402_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/b9e8ff41-4703-4740-8356-2a69b46b1bd5_isrForCrosstalkSources_855402_40.stderr parsl.dataflow.dflow INFO: Task 3375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dad7c228c244987804d31aa717ebabe3 with result from task 3375 parsl.dataflow.dflow DEBUG: Task 3375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/29698ad7-a62e-47d0-abe5-7ae6c92d1708_isrForCrosstalkSources_855456_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/29698ad7-a62e-47d0-abe5-7ae6c92d1708_isrForCrosstalkSources_855456_9.stderr parsl.dataflow.dflow INFO: Task 3269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdcfcde40311c67457feeb46fb73a6a6 with result from task 3269 parsl.dataflow.dflow DEBUG: Task 3269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2b946e7b-9fc9-408b-9237-3da1e6094eb9_isrForCrosstalkSources_855434_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2b946e7b-9fc9-408b-9237-3da1e6094eb9_isrForCrosstalkSources_855434_24.stderr parsl.dataflow.dflow INFO: Task 3166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87ae91884d3f88d959d155e8cb60ada3 with result from task 3166 parsl.dataflow.dflow DEBUG: Task 3166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1b655500-b1d0-4ddb-b037-f67e7f28e15f_isrForCrosstalkSources_855403_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/1b655500-b1d0-4ddb-b037-f67e7f28e15f_isrForCrosstalkSources_855403_44.stderr parsl.dataflow.dflow INFO: Task 3463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8e5bb911736a0e45ffd8be292af479 with result from task 3463 parsl.dataflow.dflow DEBUG: Task 3463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a5dcce0a-e390-44f2-8d39-250a7c2bb015_isrForCrosstalkSources_855398_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a5dcce0a-e390-44f2-8d39-250a7c2bb015_isrForCrosstalkSources_855398_47.stderr parsl.dataflow.dflow INFO: Task 3413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45c8a2cec3f6557941103180d76343b2 with result from task 3413 parsl.dataflow.dflow DEBUG: Task 3413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0e0a8bc0-dcc0-4a84-9cb1-33604e528266_isrForCrosstalkSources_855419_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/0e0a8bc0-dcc0-4a84-9cb1-33604e528266_isrForCrosstalkSources_855419_58.stderr parsl.dataflow.dflow INFO: Task 3465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 086ffe9f70b979effe04f5a28413047c with result from task 3465 parsl.dataflow.dflow DEBUG: Task 3465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f69c18bf-982b-48cb-9b13-7b4b9da6113b_isrForCrosstalkSources_855382_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f69c18bf-982b-48cb-9b13-7b4b9da6113b_isrForCrosstalkSources_855382_45.stderr parsl.dataflow.dflow INFO: Task 3306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b62c0a5f1b13549d408922954011a662 with result from task 3306 parsl.dataflow.dflow DEBUG: Task 3306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/111f4768-f00f-4d31-9a53-bef9c5faea42_isrForCrosstalkSources_855414_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/111f4768-f00f-4d31-9a53-bef9c5faea42_isrForCrosstalkSources_855414_8.stderr parsl.dataflow.dflow INFO: Task 3537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cdff3e16df7242e0be439661726f0c2 with result from task 3537 parsl.dataflow.dflow DEBUG: Task 3537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/51fa4cd9-3234-4a73-8370-e09ca19c8616_isrForCrosstalkSources_855441_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/51fa4cd9-3234-4a73-8370-e09ca19c8616_isrForCrosstalkSources_855441_10.stderr parsl.dataflow.dflow INFO: Task 3239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fb59274a4d85fbc13b96991e9b62752 with result from task 3239 parsl.dataflow.dflow DEBUG: Task 3239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/dd61b315-e3a1-415a-852d-5f24d6094767_isrForCrosstalkSources_855449_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/dd61b315-e3a1-415a-852d-5f24d6094767_isrForCrosstalkSources_855449_45.stderr parsl.dataflow.dflow INFO: Task 3261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd9e4b0317e0f0c0ce5730fe8be7cc7c with result from task 3261 parsl.dataflow.dflow DEBUG: Task 3261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/dcdc45c5-bde7-421e-8180-96c2ff6c41cc_isrForCrosstalkSources_855410_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/dcdc45c5-bde7-421e-8180-96c2ff6c41cc_isrForCrosstalkSources_855410_31.stderr parsl.dataflow.dflow INFO: Task 3399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f943a13467e37a9fe44f470f9867b6f2 with result from task 3399 parsl.dataflow.dflow DEBUG: Task 3399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0dbcd168-361b-4861-8ec8-00e8e3319314_isrForCrosstalkSources_855445_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/0dbcd168-361b-4861-8ec8-00e8e3319314_isrForCrosstalkSources_855445_25.stderr parsl.dataflow.dflow INFO: Task 3362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7c909cabe0faecc482469c32f3aa7c3 with result from task 3362 parsl.dataflow.dflow DEBUG: Task 3362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7ac05089-1c75-4531-8ba5-7af6a3212b8b_isrForCrosstalkSources_855391_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7ac05089-1c75-4531-8ba5-7af6a3212b8b_isrForCrosstalkSources_855391_8.stderr parsl.dataflow.dflow INFO: Task 3230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57a9aaef4f63f841da133ca40d83be46 with result from task 3230 parsl.dataflow.dflow DEBUG: Task 3230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/d770272a-b7cc-4b92-b647-41d377fef552_isrForCrosstalkSources_855413_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/d770272a-b7cc-4b92-b647-41d377fef552_isrForCrosstalkSources_855413_43.stderr parsl.dataflow.dflow INFO: Task 3471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6d1296afc89d901cc4bdf67c0e1a22 with result from task 3471 parsl.dataflow.dflow DEBUG: Task 3471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cc2091b6-682a-4b93-a569-37dd47227b22_isrForCrosstalkSources_855456_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cc2091b6-682a-4b93-a569-37dd47227b22_isrForCrosstalkSources_855456_33.stderr parsl.dataflow.dflow INFO: Task 3336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c88c1f502060bbce9acfeb3fcae6fd84 with result from task 3336 parsl.dataflow.dflow DEBUG: Task 3336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b9ff11c-c6bf-4b54-bef2-a0778e2379bc_isrForCrosstalkSources_855400_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/2b9ff11c-c6bf-4b54-bef2-a0778e2379bc_isrForCrosstalkSources_855400_15.stderr parsl.dataflow.dflow INFO: Task 3300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe4ab725b791b0769cc32d636409b4d0 with result from task 3300 parsl.dataflow.dflow DEBUG: Task 3300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f338996d-c0c1-4ee1-beef-2c0b243c5c6e_isrForCrosstalkSources_855404_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f338996d-c0c1-4ee1-beef-2c0b243c5c6e_isrForCrosstalkSources_855404_56.stderr parsl.dataflow.dflow INFO: Task 3317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81fa6e490e80de90dae4902a7aae2223 with result from task 3317 parsl.dataflow.dflow DEBUG: Task 3317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/bff0bb6d-7a32-44b3-a280-1d7cbf212e36_isrForCrosstalkSources_855404_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/bff0bb6d-7a32-44b3-a280-1d7cbf212e36_isrForCrosstalkSources_855404_46.stderr parsl.dataflow.dflow INFO: Task 3254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99756d5ff861b8d1452182dee08a8512 with result from task 3254 parsl.dataflow.dflow DEBUG: Task 3254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/203f96ea-7ef8-465e-89f8-3388727b1fe2_isrForCrosstalkSources_855437_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/203f96ea-7ef8-465e-89f8-3388727b1fe2_isrForCrosstalkSources_855437_59.stderr parsl.dataflow.dflow INFO: Task 3313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1160b7ba22f24bbfa38b606ea5db0f99 with result from task 3313 parsl.dataflow.dflow DEBUG: Task 3313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/77f0c514-8f89-450f-bc1e-0cd285c92b58_isrForCrosstalkSources_855438_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/77f0c514-8f89-450f-bc1e-0cd285c92b58_isrForCrosstalkSources_855438_17.stderr parsl.dataflow.dflow INFO: Task 3291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9bee3d00949492dc9fce3555f0df8a with result from task 3291 parsl.dataflow.dflow DEBUG: Task 3291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/8a6aaab0-90b2-4c84-982e-6944c2f8ef50_isrForCrosstalkSources_855405_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/8a6aaab0-90b2-4c84-982e-6944c2f8ef50_isrForCrosstalkSources_855405_33.stderr parsl.dataflow.dflow INFO: Task 3293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a04f6b7cd58de16f25e122ccf06fc580 with result from task 3293 parsl.dataflow.dflow DEBUG: Task 3293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d2bf9c0b-db94-4ffe-a4cc-b56f4502d375_isrForCrosstalkSources_855446_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d2bf9c0b-db94-4ffe-a4cc-b56f4502d375_isrForCrosstalkSources_855446_21.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53e03d4b73ce61f28f8eefc1293ff16 with result from task 3446 parsl.dataflow.dflow DEBUG: Task 3446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5b41249f-aefe-4c17-9173-a125efeaf69a_isrForCrosstalkSources_855445_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/5b41249f-aefe-4c17-9173-a125efeaf69a_isrForCrosstalkSources_855445_19.stderr parsl.dataflow.dflow INFO: Task 3426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa943888de40c5e2451eb9be5188cf8 with result from task 3426 parsl.dataflow.dflow DEBUG: Task 3426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/daea178a-440c-41f6-a8fb-fa174512c70b_isrForCrosstalkSources_855429_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/daea178a-440c-41f6-a8fb-fa174512c70b_isrForCrosstalkSources_855429_11.stderr parsl.dataflow.dflow INFO: Task 3394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3af2442d4d469bc06db6873c6408e5d8 with result from task 3394 parsl.dataflow.dflow DEBUG: Task 3394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fe0c2afe-735c-4468-8cb8-fb896061b297_isrForCrosstalkSources_855430_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fe0c2afe-735c-4468-8cb8-fb896061b297_isrForCrosstalkSources_855430_46.stderr parsl.dataflow.dflow INFO: Task 3420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d01d7876c6c811df0fd287e89b0a4930 with result from task 3420 parsl.dataflow.dflow DEBUG: Task 3420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f18b569c-8ae7-4fa3-8f4e-1e7f6b455960_isrForCrosstalkSources_855451_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/f18b569c-8ae7-4fa3-8f4e-1e7f6b455960_isrForCrosstalkSources_855451_52.stderr parsl.dataflow.dflow INFO: Task 3431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58285a047bd4d210959fdc887247e7c with result from task 3431 parsl.dataflow.dflow DEBUG: Task 3431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/3700e20e-9474-46d8-87a3-3a67ff26a03c_isrForCrosstalkSources_855446_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/3700e20e-9474-46d8-87a3-3a67ff26a03c_isrForCrosstalkSources_855446_39.stderr parsl.dataflow.dflow INFO: Task 3277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a81f2f016a4faf23acf1210911a52e87 with result from task 3277 parsl.dataflow.dflow DEBUG: Task 3277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7dbe451c-9e32-4245-8c96-f3124b354111_isrForCrosstalkSources_855382_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/7dbe451c-9e32-4245-8c96-f3124b354111_isrForCrosstalkSources_855382_23.stderr 2024-03-09 14:54:38 proc_lsst.multi:170 [INFO] [multi] job_id 16964288 proc_lsst.multi INFO: [multi] job_id 16964288 2024-03-09 14:54:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 198 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 198 2024-03-09 14:54:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964288 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964288 parsl.executors.status_handling DEBUG: Launched block 253 on executor multi with job ID 16964288 parsl.executors.status_handling INFO: Allocated block ID 254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-254 2024-03-09 14:54:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 2024-03-09 14:54:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 198 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 198 2024-03-09 14:54:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daa2aab53344ed7c17551cd3cef299b3 with result from task 3288 parsl.dataflow.dflow DEBUG: Task 3288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/177ec516-102d-41f5-8cb5-a41a18c243dd_isrForCrosstalkSources_855455_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/177ec516-102d-41f5-8cb5-a41a18c243dd_isrForCrosstalkSources_855455_27.stderr parsl.dataflow.dflow INFO: Task 3521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dc56c0e450b9570ca4279db566cbb74 with result from task 3521 parsl.dataflow.dflow DEBUG: Task 3521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a17d691-0212-47a4-b058-c5d07d721c20_isrForCrosstalkSources_855409_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0a17d691-0212-47a4-b058-c5d07d721c20_isrForCrosstalkSources_855409_26.stderr parsl.dataflow.dflow INFO: Task 3518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba3ab4523f3463263ffa259af85f82d8 with result from task 3518 parsl.dataflow.dflow DEBUG: Task 3518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32ab099e-41e0-4e13-bb99-95bac65ae896_isrForCrosstalkSources_855447_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/32ab099e-41e0-4e13-bb99-95bac65ae896_isrForCrosstalkSources_855447_25.stderr parsl.dataflow.dflow INFO: Task 3395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 927f167eeebc1632a0a734365237600e with result from task 3395 parsl.dataflow.dflow DEBUG: Task 3395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/562f57e2-5766-4f4a-9e41-f5e012465bb1_isrForCrosstalkSources_855401_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/562f57e2-5766-4f4a-9e41-f5e012465bb1_isrForCrosstalkSources_855401_6.stderr parsl.dataflow.dflow INFO: Task 3352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 330dcb591e6ad756bbfaa4237c32c353 with result from task 3352 parsl.dataflow.dflow DEBUG: Task 3352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/70094a6c-074f-404d-afd8-be75026cd251_isrForCrosstalkSources_855453_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/70094a6c-074f-404d-afd8-be75026cd251_isrForCrosstalkSources_855453_17.stderr parsl.dataflow.dflow INFO: Task 3353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d1a47b541174bd77429f573fc8dfdca with result from task 3353 parsl.dataflow.dflow DEBUG: Task 3353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/049984de-e4be-437c-b9ac-4efe10e0b8f7_isrForCrosstalkSources_855421_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/049984de-e4be-437c-b9ac-4efe10e0b8f7_isrForCrosstalkSources_855421_5.stderr parsl.dataflow.dflow INFO: Task 3416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ea7dc7712ff82456d47bc357519f150 with result from task 3416 parsl.dataflow.dflow DEBUG: Task 3416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/dcf9f9ac-f74a-414b-aa25-3d56ff424c0e_isrForCrosstalkSources_855414_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/dcf9f9ac-f74a-414b-aa25-3d56ff424c0e_isrForCrosstalkSources_855414_41.stderr parsl.dataflow.dflow INFO: Task 3432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c2e47753e25f29c031e9f9842f5492 with result from task 3432 parsl.dataflow.dflow DEBUG: Task 3432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85cc5a34-b767-40e7-beb9-90d268ac4fae_isrForCrosstalkSources_855413_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85cc5a34-b767-40e7-beb9-90d268ac4fae_isrForCrosstalkSources_855413_23.stderr parsl.dataflow.dflow INFO: Task 3411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55918bace708ec72d4db905a8493b45b with result from task 3411 parsl.dataflow.dflow DEBUG: Task 3411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac2b4776-0947-44f1-89e2-220f376b2eca_isrForCrosstalkSources_855394_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac2b4776-0947-44f1-89e2-220f376b2eca_isrForCrosstalkSources_855394_30.stderr parsl.dataflow.dflow INFO: Task 3100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6145a5dfd13612d19bfe6f9a87e8ab0 with result from task 3100 parsl.dataflow.dflow DEBUG: Task 3100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3bd27bd0-49db-48b8-9cc1-db6a907988ca_isrForCrosstalkSources_855400_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3bd27bd0-49db-48b8-9cc1-db6a907988ca_isrForCrosstalkSources_855400_57.stderr parsl.dataflow.dflow INFO: Task 3171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ffe08a44d93a186f3892667ede7d138 with result from task 3171 parsl.dataflow.dflow DEBUG: Task 3171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1be526a5-f47d-4508-9504-3b89d1d2a81c_isrForCrosstalkSources_855421_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/1be526a5-f47d-4508-9504-3b89d1d2a81c_isrForCrosstalkSources_855421_44.stderr parsl.dataflow.dflow INFO: Task 3495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6439b2d7dac99413fece1ee129700d7a with result from task 3495 parsl.dataflow.dflow DEBUG: Task 3495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/40c2a8cc-ea0a-497b-89e1-effeadf60cae_isrForCrosstalkSources_855438_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/40c2a8cc-ea0a-497b-89e1-effeadf60cae_isrForCrosstalkSources_855438_53.stderr parsl.dataflow.dflow INFO: Task 3125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7598cc8acb2b2ac04ddf2a9fe51fee0 with result from task 3125 parsl.dataflow.dflow DEBUG: Task 3125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/acca1293-970f-4be7-b386-1b7fc82ceffd_isrForCrosstalkSources_855459_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/acca1293-970f-4be7-b386-1b7fc82ceffd_isrForCrosstalkSources_855459_62.stderr parsl.dataflow.dflow INFO: Task 3229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf6387dc0f986611e5bc01f32299e5e with result from task 3229 parsl.dataflow.dflow DEBUG: Task 3229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f085d25f-fc94-48c0-b06a-7f785bd70bd9_isrForCrosstalkSources_855458_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/f085d25f-fc94-48c0-b06a-7f785bd70bd9_isrForCrosstalkSources_855458_47.stderr parsl.dataflow.dflow INFO: Task 3295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e27a9f4eaf8b236c8620f9a748704f with result from task 3295 parsl.dataflow.dflow DEBUG: Task 3295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46171d7c-d804-4393-bc58-87d9baf436f0_isrForCrosstalkSources_855419_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/46171d7c-d804-4393-bc58-87d9baf436f0_isrForCrosstalkSources_855419_31.stderr parsl.dataflow.dflow INFO: Task 3405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03b4bca069ceb22d643d0f2111d64c91 with result from task 3405 parsl.dataflow.dflow DEBUG: Task 3405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/39688a18-723d-4234-a925-9552850744ab_isrForCrosstalkSources_855436_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/39688a18-723d-4234-a925-9552850744ab_isrForCrosstalkSources_855436_52.stderr parsl.dataflow.dflow INFO: Task 3213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 190cbeb1595c7525bf621cb1997fb8cb with result from task 3213 parsl.dataflow.dflow DEBUG: Task 3213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/12ada047-5aee-430d-b7c8-57ac303eb900_isrForCrosstalkSources_855409_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/12ada047-5aee-430d-b7c8-57ac303eb900_isrForCrosstalkSources_855409_47.stderr parsl.dataflow.dflow INFO: Task 3436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0f1bbd64d4a7eaa83c5b0a6cf4820ce with result from task 3436 parsl.dataflow.dflow DEBUG: Task 3436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8ecb4921-441c-463a-ae48-086d8c792534_isrForCrosstalkSources_855400_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8ecb4921-441c-463a-ae48-086d8c792534_isrForCrosstalkSources_855400_8.stderr parsl.dataflow.dflow INFO: Task 3198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 056421ecf787f556f8ec34c254443b6d with result from task 3198 parsl.dataflow.dflow DEBUG: Task 3198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a854591f-3c71-402e-8cfc-a29992c8cf5a_isrForCrosstalkSources_855441_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a854591f-3c71-402e-8cfc-a29992c8cf5a_isrForCrosstalkSources_855441_60.stderr parsl.dataflow.dflow INFO: Task 3429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5523f4da28b60cfdbb3ee66d3ffd1a8 with result from task 3429 parsl.dataflow.dflow DEBUG: Task 3429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7cddf91d-1fe6-4c69-8f5d-0ae668d1739f_isrForCrosstalkSources_855419_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7cddf91d-1fe6-4c69-8f5d-0ae668d1739f_isrForCrosstalkSources_855419_10.stderr parsl.dataflow.dflow INFO: Task 3386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ad019fa327205bf31bed7350626bfd5 with result from task 3386 parsl.dataflow.dflow DEBUG: Task 3386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/789a794b-d699-4e6b-b710-a2954a6ee26e_isrForCrosstalkSources_855399_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/789a794b-d699-4e6b-b710-a2954a6ee26e_isrForCrosstalkSources_855399_38.stderr parsl.dataflow.dflow INFO: Task 3419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e9e1b45ff75b7e07f0615a506f3976e with result from task 3419 parsl.dataflow.dflow DEBUG: Task 3419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2c246f36-383e-4555-949b-e3ce2c96abfc_isrForCrosstalkSources_855441_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/2c246f36-383e-4555-949b-e3ce2c96abfc_isrForCrosstalkSources_855441_57.stderr parsl.dataflow.dflow INFO: Task 3318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e1d5b8e53584cbd8bd9bccd75e02f03 with result from task 3318 parsl.dataflow.dflow DEBUG: Task 3318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/16e40b65-7be4-4539-a079-cb5101eaaa34_isrForCrosstalkSources_855401_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/16e40b65-7be4-4539-a079-cb5101eaaa34_isrForCrosstalkSources_855401_22.stderr parsl.dataflow.dflow INFO: Task 3469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf76b745c8c4c92632a76a0f32497c11 with result from task 3469 parsl.dataflow.dflow DEBUG: Task 3469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/87d72053-8afe-4112-9021-03148a11cf2f_isrForCrosstalkSources_855430_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/87d72053-8afe-4112-9021-03148a11cf2f_isrForCrosstalkSources_855430_1.stderr parsl.dataflow.dflow INFO: Task 3549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 735021a4a53e23025745f4845338f3cf with result from task 3549 parsl.dataflow.dflow DEBUG: Task 3549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2a1baa68-dcdc-49a9-8884-b7d84ed20cbd_isrForCrosstalkSources_855406_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2a1baa68-dcdc-49a9-8884-b7d84ed20cbd_isrForCrosstalkSources_855406_13.stderr parsl.dataflow.dflow INFO: Task 3451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e6e827f052039d421127a7441931ad5 with result from task 3451 parsl.dataflow.dflow DEBUG: Task 3451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c2141cea-ebe4-47e8-8cef-eaaf6af81824_isrForCrosstalkSources_855393_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/c2141cea-ebe4-47e8-8cef-eaaf6af81824_isrForCrosstalkSources_855393_29.stderr parsl.dataflow.dflow INFO: Task 3478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8a4f9bfc21352a58c7a915d6dccc70 with result from task 3478 parsl.dataflow.dflow DEBUG: Task 3478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/aaef5961-5040-4b4c-b788-79902e9bfd96_isrForCrosstalkSources_855414_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/aaef5961-5040-4b4c-b788-79902e9bfd96_isrForCrosstalkSources_855414_13.stderr parsl.dataflow.dflow INFO: Task 3505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de3e715d003dd378bcf58343ebc3f32e with result from task 3505 parsl.dataflow.dflow DEBUG: Task 3505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/63188aaa-4701-4f5e-8e8a-24fd74815db4_isrForCrosstalkSources_855392_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/63188aaa-4701-4f5e-8e8a-24fd74815db4_isrForCrosstalkSources_855392_33.stderr parsl.dataflow.dflow INFO: Task 3550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b01d771752b11fb35209640c32b6177f with result from task 3550 parsl.dataflow.dflow DEBUG: Task 3550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7140b809-35f7-44cb-95f4-c7500be2b39f_isrForCrosstalkSources_855412_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/7140b809-35f7-44cb-95f4-c7500be2b39f_isrForCrosstalkSources_855412_2.stderr parsl.dataflow.dflow INFO: Task 3153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def4eaf0e232762496253ed09df3e33b with result from task 3153 parsl.dataflow.dflow DEBUG: Task 3153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8908981e-9194-4a74-a90b-4e88cff77c5f_isrForCrosstalkSources_855445_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/8908981e-9194-4a74-a90b-4e88cff77c5f_isrForCrosstalkSources_855445_60.stderr parsl.dataflow.dflow INFO: Task 3400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7e3b992b7b16fae61dfc7829ad11ee9 with result from task 3400 parsl.dataflow.dflow DEBUG: Task 3400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/aa55b2aa-806b-431c-94f7-cb7b37273d1a_isrForCrosstalkSources_855390_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/aa55b2aa-806b-431c-94f7-cb7b37273d1a_isrForCrosstalkSources_855390_22.stderr parsl.dataflow.dflow INFO: Task 3294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14498b1ae78b1008d019b01d679343c4 with result from task 3294 parsl.dataflow.dflow DEBUG: Task 3294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4ebaab23-e92b-411c-8e71-9f7cb67e4aa7_isrForCrosstalkSources_855451_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/4ebaab23-e92b-411c-8e71-9f7cb67e4aa7_isrForCrosstalkSources_855451_18.stderr parsl.dataflow.dflow INFO: Task 3255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9964029dc45be9be020725329c09b7f8 with result from task 3255 parsl.dataflow.dflow DEBUG: Task 3255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/14350e7b-5db1-43b1-9cf4-89fc90f0354e_isrForCrosstalkSources_855421_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/14350e7b-5db1-43b1-9cf4-89fc90f0354e_isrForCrosstalkSources_855421_12.stderr parsl.dataflow.dflow INFO: Task 3262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e81b834a31db9a7bbc82a643367ca610 with result from task 3262 parsl.dataflow.dflow DEBUG: Task 3262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/4f54f249-ef20-4716-91e4-ac0cd64d1a3d_isrForCrosstalkSources_855431_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/4f54f249-ef20-4716-91e4-ac0cd64d1a3d_isrForCrosstalkSources_855431_28.stderr parsl.dataflow.dflow INFO: Task 3365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35f952c4a7d03480e1f28b56cd9c3eb4 with result from task 3365 parsl.dataflow.dflow DEBUG: Task 3365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/c6d741d9-414b-4f5c-b33b-0d7aace65a3e_isrForCrosstalkSources_855396_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/c6d741d9-414b-4f5c-b33b-0d7aace65a3e_isrForCrosstalkSources_855396_32.stderr parsl.dataflow.dflow INFO: Task 3484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79e36ab0b0c8b235a7133f00e26efdd1 with result from task 3484 parsl.dataflow.dflow DEBUG: Task 3484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4eaf419b-9dc1-4f88-beb7-8cda288fdbff_isrForCrosstalkSources_855404_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4eaf419b-9dc1-4f88-beb7-8cda288fdbff_isrForCrosstalkSources_855404_19.stderr parsl.dataflow.dflow INFO: Task 3337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac7f38688a2739ad50204664f338d622 with result from task 3337 parsl.dataflow.dflow DEBUG: Task 3337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7e7039a7-6744-4675-b91f-65409ce8a0a2_isrForCrosstalkSources_855393_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7e7039a7-6744-4675-b91f-65409ce8a0a2_isrForCrosstalkSources_855393_10.stderr parsl.dataflow.dflow INFO: Task 3289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3156662feb49ebd30e3ed7f70f9da440 with result from task 3289 parsl.dataflow.dflow DEBUG: Task 3289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/927f96ba-d510-476c-a656-d02ea7969808_isrForCrosstalkSources_855404_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/927f96ba-d510-476c-a656-d02ea7969808_isrForCrosstalkSources_855404_12.stderr parsl.dataflow.dflow INFO: Task 3298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 823732da5636908422190aaa30df20f3 with result from task 3298 parsl.dataflow.dflow DEBUG: Task 3298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/45f54443-35bd-4317-93f5-a59cf0e73ae4_isrForCrosstalkSources_855453_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/45f54443-35bd-4317-93f5-a59cf0e73ae4_isrForCrosstalkSources_855453_9.stderr parsl.dataflow.dflow INFO: Task 3452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8b0d1957803033c0f03c191ebe478f9 with result from task 3452 parsl.dataflow.dflow DEBUG: Task 3452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/907572b2-f7cd-49a4-b7bb-c823864b0a57_isrForCrosstalkSources_855433_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/907572b2-f7cd-49a4-b7bb-c823864b0a57_isrForCrosstalkSources_855433_8.stderr parsl.dataflow.dflow INFO: Task 3472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 996df03a3c2dffd6cf3368926b4bc266 with result from task 3472 parsl.dataflow.dflow DEBUG: Task 3472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cccb1036-0f3e-4259-b2b9-178cf974c4fa_isrForCrosstalkSources_855441_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cccb1036-0f3e-4259-b2b9-178cf974c4fa_isrForCrosstalkSources_855441_5.stderr parsl.dataflow.dflow INFO: Task 3457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ed8e3569eb5ff61cc7b9edc9ce3fd7 with result from task 3457 parsl.dataflow.dflow DEBUG: Task 3457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/166ca430-e38c-4a5b-a186-df6e3a4884e2_isrForCrosstalkSources_855444_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/166ca430-e38c-4a5b-a186-df6e3a4884e2_isrForCrosstalkSources_855444_33.stderr parsl.dataflow.dflow INFO: Task 3378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c9811655269a9fa9fd1b4f7fc92e69c with result from task 3378 parsl.dataflow.dflow DEBUG: Task 3378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/4f69030a-4e82-4cf2-b87a-2e1e01ba6e0c_isrForCrosstalkSources_855441_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/4f69030a-4e82-4cf2-b87a-2e1e01ba6e0c_isrForCrosstalkSources_855441_20.stderr parsl.dataflow.dflow INFO: Task 2833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f808aa30c124a72f7e0875666ff85caa with result from task 2833 parsl.dataflow.dflow DEBUG: Task 2833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e636e68a-4ebd-402d-af07-9aebbed7465e_isrForCrosstalkSources_855428_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/e636e68a-4ebd-402d-af07-9aebbed7465e_isrForCrosstalkSources_855428_35.stderr parsl.dataflow.dflow INFO: Task 3173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db9a4235045eba93d6d8dab3c663eb51 with result from task 3173 parsl.dataflow.dflow DEBUG: Task 3173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1e1f2582-3b63-4b56-8635-66af830575b7_isrForCrosstalkSources_855380_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1e1f2582-3b63-4b56-8635-66af830575b7_isrForCrosstalkSources_855380_42.stderr parsl.dataflow.dflow INFO: Task 3559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a67c16c2b626211776ccaba5193f7b0 with result from task 3559 parsl.dataflow.dflow DEBUG: Task 3559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a67a79e6-b962-40ef-85da-aa440cb252da_isrForCrosstalkSources_855395_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a67a79e6-b962-40ef-85da-aa440cb252da_isrForCrosstalkSources_855395_27.stderr parsl.dataflow.dflow INFO: Task 3283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a03b45590d13178b7a3ec9078f560dc with result from task 3283 parsl.dataflow.dflow DEBUG: Task 3283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eb9a7d0-6658-481b-b398-74da10b51297_isrForCrosstalkSources_855434_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eb9a7d0-6658-481b-b398-74da10b51297_isrForCrosstalkSources_855434_59.stderr parsl.dataflow.dflow INFO: Task 3234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9fdf3851f4c9bb0cd88ff0fa7a7fda2 with result from task 3234 parsl.dataflow.dflow DEBUG: Task 3234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a4d96484-14cf-41e8-9409-0c5a418bc2cc_isrForCrosstalkSources_855421_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a4d96484-14cf-41e8-9409-0c5a418bc2cc_isrForCrosstalkSources_855421_29.stderr parsl.dataflow.dflow INFO: Task 3356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9eb54f79ee5e90b956fa953d5991f9 with result from task 3356 parsl.dataflow.dflow DEBUG: Task 3356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/87cbe565-d88e-4664-a78a-54dbca48a1b1_isrForCrosstalkSources_855388_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/87cbe565-d88e-4664-a78a-54dbca48a1b1_isrForCrosstalkSources_855388_38.stderr parsl.dataflow.dflow INFO: Task 3284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44e88399ff603a81c830cea4440182d2 with result from task 3284 parsl.dataflow.dflow DEBUG: Task 3284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/abdd798f-a489-44c2-82cc-8a81041c2d4b_isrForCrosstalkSources_855396_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/abdd798f-a489-44c2-82cc-8a81041c2d4b_isrForCrosstalkSources_855396_55.stderr parsl.dataflow.dflow INFO: Task 3244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e9fc3097d958c27913aa48fb2a7f73 with result from task 3244 parsl.dataflow.dflow DEBUG: Task 3244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/531fa6b5-6af4-4d1b-9b0c-75863d646999_isrForCrosstalkSources_855429_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/531fa6b5-6af4-4d1b-9b0c-75863d646999_isrForCrosstalkSources_855429_21.stderr parsl.dataflow.dflow INFO: Task 3569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbd42eae016554e12883c64d09061b1a with result from task 3569 parsl.dataflow.dflow DEBUG: Task 3569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4094669c-c20f-4c4f-aec8-c2223058782b_isrForCrosstalkSources_855415_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4094669c-c20f-4c4f-aec8-c2223058782b_isrForCrosstalkSources_855415_51.stderr parsl.dataflow.dflow INFO: Task 3271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 461630d395f899e5673a1b3a9a59e403 with result from task 3271 parsl.dataflow.dflow DEBUG: Task 3271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/35a3c222-9ed2-480f-840f-98d7ce9c9297_isrForCrosstalkSources_855407_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/35a3c222-9ed2-480f-840f-98d7ce9c9297_isrForCrosstalkSources_855407_50.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfc7e77799827a12b903d3c7a8fe56c6 with result from task 3372 parsl.dataflow.dflow DEBUG: Task 3372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f5541ea-5ede-4116-af1d-9c330e20ccc7_isrForCrosstalkSources_855387_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f5541ea-5ede-4116-af1d-9c330e20ccc7_isrForCrosstalkSources_855387_16.stderr 2024-03-09 14:54:40 proc_lsst.multi:170 [INFO] [multi] job_id 16964289 proc_lsst.multi INFO: [multi] job_id 16964289 2024-03-09 14:54:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 199 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 199 2024-03-09 14:54:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964289 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964289 parsl.dataflow.dflow INFO: Task 3296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7483eb6e0659ed55346b5cd37d9308a6 with result from task 3296 parsl.executors.status_handling DEBUG: Launched block 254 on executor multi with job ID 16964289 parsl.executors.status_handling INFO: Allocated block ID 255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-255 2024-03-09 14:54:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 2024-03-09 14:54:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 199 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 199 2024-03-09 14:54:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec9896af-2420-4be4-8b05-f07330c9f444_isrForCrosstalkSources_855391_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/ec9896af-2420-4be4-8b05-f07330c9f444_isrForCrosstalkSources_855391_22.stderr parsl.dataflow.dflow INFO: Task 3442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfce970a5f0390beab6f04c0f56b947a with result from task 3442 parsl.dataflow.dflow DEBUG: Task 3442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c8df001a-7dea-4ae0-948c-4bd62e1a8020_isrForCrosstalkSources_855413_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c8df001a-7dea-4ae0-948c-4bd62e1a8020_isrForCrosstalkSources_855413_12.stderr parsl.dataflow.dflow INFO: Task 3473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b7c17f8ce0f5d2551cd8f4ff89ab3f8 with result from task 3473 parsl.dataflow.dflow DEBUG: Task 3473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a4703adb-6136-45e1-ba0e-bf1b5b9da8d0_isrForCrosstalkSources_855388_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/a4703adb-6136-45e1-ba0e-bf1b5b9da8d0_isrForCrosstalkSources_855388_29.stderr parsl.dataflow.dflow INFO: Task 3418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfb0563dca82bc450b96fd3a4da07ebf with result from task 3418 parsl.dataflow.dflow DEBUG: Task 3418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/52a2b556-151e-472f-9a8e-a7f6de621d20_isrForCrosstalkSources_855402_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/52a2b556-151e-472f-9a8e-a7f6de621d20_isrForCrosstalkSources_855402_28.stderr parsl.dataflow.dflow INFO: Task 3497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8327a67250b072a5b7360743e403cd8d with result from task 3497 parsl.dataflow.dflow DEBUG: Task 3497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/eb9ae3e2-dc59-47a6-a9d3-4f95b0e01b33_isrForCrosstalkSources_855394_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/eb9ae3e2-dc59-47a6-a9d3-4f95b0e01b33_isrForCrosstalkSources_855394_17.stderr parsl.dataflow.dflow INFO: Task 3358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a545bb4a65f3bd5546237f364804673e with result from task 3358 parsl.dataflow.dflow DEBUG: Task 3358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1702a573-18f8-4c82-b87d-6134152070ee_isrForCrosstalkSources_855454_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1702a573-18f8-4c82-b87d-6134152070ee_isrForCrosstalkSources_855454_15.stderr parsl.dataflow.dflow INFO: Task 3302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f721a729ebf3d64ef6d8cfb486f70f91 with result from task 3302 parsl.dataflow.dflow DEBUG: Task 3302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3491356d-45b2-4bb0-9d8f-08d1ac06762e_isrForCrosstalkSources_855420_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/3491356d-45b2-4bb0-9d8f-08d1ac06762e_isrForCrosstalkSources_855420_50.stderr parsl.dataflow.dflow INFO: Task 3443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6834abf18b0fb9b57f8e3f3e5fd7c788 with result from task 3443 parsl.dataflow.dflow DEBUG: Task 3443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/5df8726b-a1f6-49dd-8b4e-5ef4b2af8857_isrForCrosstalkSources_855380_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/5df8726b-a1f6-49dd-8b4e-5ef4b2af8857_isrForCrosstalkSources_855380_57.stderr parsl.dataflow.dflow INFO: Task 3350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 115275db087fdfc37b09ccea55586339 with result from task 3350 parsl.dataflow.dflow DEBUG: Task 3350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/19a61281-f927-4d1c-a52b-c48fd421ebe2_isrForCrosstalkSources_855454_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/19a61281-f927-4d1c-a52b-c48fd421ebe2_isrForCrosstalkSources_855454_23.stderr parsl.dataflow.dflow INFO: Task 3268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91179648853a43d1eef7c0f5283a194a with result from task 3268 parsl.dataflow.dflow DEBUG: Task 3268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/edec96b1-9fae-4fd5-8ec9-48bea8875438_isrForCrosstalkSources_855399_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/edec96b1-9fae-4fd5-8ec9-48bea8875438_isrForCrosstalkSources_855399_23.stderr parsl.dataflow.dflow INFO: Task 3281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6d2c38ff0cea8e0c9a85952a19896b7 with result from task 3281 parsl.dataflow.dflow DEBUG: Task 3281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e0507c7-991f-4e4f-9e21-08e7c30ce0e3_isrForCrosstalkSources_855438_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/7e0507c7-991f-4e4f-9e21-08e7c30ce0e3_isrForCrosstalkSources_855438_9.stderr parsl.dataflow.dflow INFO: Task 3223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13083f73a916915eed36bb5f2bd2dae1 with result from task 3223 parsl.dataflow.dflow DEBUG: Task 3223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f85253cd-caea-47ba-9d52-96d6f6ff13cc_isrForCrosstalkSources_855454_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f85253cd-caea-47ba-9d52-96d6f6ff13cc_isrForCrosstalkSources_855454_57.stderr parsl.dataflow.dflow INFO: Task 3265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cead9dc45af259c0d994eb52b33d83f5 with result from task 3265 parsl.dataflow.dflow DEBUG: Task 3265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e7ff9757-7191-4a82-ba64-d028a2847ebf_isrForCrosstalkSources_855409_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e7ff9757-7191-4a82-ba64-d028a2847ebf_isrForCrosstalkSources_855409_1.stderr parsl.dataflow.dflow INFO: Task 3282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78af73007fc783bd443ec75cca4d2564 with result from task 3282 parsl.dataflow.dflow DEBUG: Task 3282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0a4d14ed-1502-4ed6-88f1-073f70ea5ccd_isrForCrosstalkSources_855405_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/0a4d14ed-1502-4ed6-88f1-073f70ea5ccd_isrForCrosstalkSources_855405_17.stderr parsl.dataflow.dflow INFO: Task 3326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb673b19bf9b34d3abeed1d81a88e3de with result from task 3326 parsl.dataflow.dflow DEBUG: Task 3326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6d4b2595-12b7-4b5e-85bb-5b9ec084790f_isrForCrosstalkSources_855421_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/6d4b2595-12b7-4b5e-85bb-5b9ec084790f_isrForCrosstalkSources_855421_6.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b13a05f02151b1adf5525c1c9fe977d with result from task 3525 parsl.dataflow.dflow DEBUG: Task 3525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1111bb95-9e35-4fc8-9955-e6df5be5db26_isrForCrosstalkSources_855380_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/1111bb95-9e35-4fc8-9955-e6df5be5db26_isrForCrosstalkSources_855380_31.stderr parsl.dataflow.dflow INFO: Task 3290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cf89ab3852867b3eca88ff404c298b0 with result from task 3290 parsl.dataflow.dflow DEBUG: Task 3290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/54932d80-3a26-4ff9-b768-7019f65053b2_isrForCrosstalkSources_855393_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/54932d80-3a26-4ff9-b768-7019f65053b2_isrForCrosstalkSources_855393_1.stderr parsl.dataflow.dflow INFO: Task 3315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cb64ecce9e51644067c9f65eb318d4d with result from task 3315 parsl.dataflow.dflow DEBUG: Task 3315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcc0b76-2fc5-4a79-9c44-5571bbaa0027_isrForCrosstalkSources_855409_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/1bcc0b76-2fc5-4a79-9c44-5571bbaa0027_isrForCrosstalkSources_855409_23.stderr parsl.dataflow.dflow INFO: Task 3466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 644228c7152b17457666b4a835c24110 with result from task 3466 parsl.dataflow.dflow DEBUG: Task 3466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f8ae4a49-6851-4550-a6a1-98140540ad13_isrForCrosstalkSources_855455_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f8ae4a49-6851-4550-a6a1-98140540ad13_isrForCrosstalkSources_855455_16.stderr parsl.dataflow.dflow INFO: Task 3355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11535d976e9cb8f6a04de4a57079cfe6 with result from task 3355 parsl.dataflow.dflow DEBUG: Task 3355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/15c6b4f1-956b-4787-b031-26d7161018a9_isrForCrosstalkSources_855391_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/15c6b4f1-956b-4787-b031-26d7161018a9_isrForCrosstalkSources_855391_37.stderr parsl.dataflow.dflow INFO: Task 3524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff62663d6389d19b161c52d7c598eff3 with result from task 3524 parsl.dataflow.dflow DEBUG: Task 3524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b86a07a6-a973-4e8a-9e66-1c2bdf32e4e7_isrForCrosstalkSources_855407_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b86a07a6-a973-4e8a-9e66-1c2bdf32e4e7_isrForCrosstalkSources_855407_60.stderr parsl.dataflow.dflow INFO: Task 3333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03d6398a94448818f09c503c29161605 with result from task 3333 parsl.dataflow.dflow DEBUG: Task 3333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1d8473fd-5d56-48b2-ad58-85e120e58ec1_isrForCrosstalkSources_855446_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1d8473fd-5d56-48b2-ad58-85e120e58ec1_isrForCrosstalkSources_855446_12.stderr parsl.dataflow.dflow INFO: Task 3274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec299d1e48c93ca10e4eec0ff80a3e79 with result from task 3274 parsl.dataflow.dflow DEBUG: Task 3274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d3e936d6-6163-4849-ab60-30ab27b390a4_isrForCrosstalkSources_855438_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d3e936d6-6163-4849-ab60-30ab27b390a4_isrForCrosstalkSources_855438_11.stderr parsl.dataflow.dflow INFO: Task 3595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b6002ea242aa1edf8a8fe8d1ad00406 with result from task 3595 parsl.dataflow.dflow DEBUG: Task 3595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4d8e5c40-5d78-4144-ad25-eb1a6a51d134_isrForCrosstalkSources_855439_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/4d8e5c40-5d78-4144-ad25-eb1a6a51d134_isrForCrosstalkSources_855439_34.stderr parsl.dataflow.dflow INFO: Task 3499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce5781cb2430f7a3e2fbe9f12f117ca0 with result from task 3499 2024-03-09 14:54:41 proc_lsst.multi:170 [INFO] [multi] job_id 16964290 proc_lsst.multi INFO: [multi] job_id 16964290 2024-03-09 14:54:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 200 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 200 2024-03-09 14:54:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964290 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964290 parsl.executors.status_handling DEBUG: Launched block 255 on executor multi with job ID 16964290 parsl.executors.status_handling INFO: Allocated block ID 256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-256 2024-03-09 14:54:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 200 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 200 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9fa9e5ca-ca7d-4c04-8d26-932324bc785f_isrForCrosstalkSources_855427_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/9fa9e5ca-ca7d-4c04-8d26-932324bc785f_isrForCrosstalkSources_855427_53.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fc73e438e59f2fe80f0fd9072636e46 with result from task 3346 parsl.dataflow.dflow DEBUG: Task 3346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b5378692-ec56-47de-936f-4721b1f11595_isrForCrosstalkSources_855458_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/b5378692-ec56-47de-936f-4721b1f11595_isrForCrosstalkSources_855458_49.stderr parsl.dataflow.dflow INFO: Task 3464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d70387b413a9a7a9315dc8ba4bcef36 with result from task 3464 parsl.dataflow.dflow DEBUG: Task 3464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/517b2a07-83f5-468a-81bb-b79b7869a36c_isrForCrosstalkSources_855457_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/517b2a07-83f5-468a-81bb-b79b7869a36c_isrForCrosstalkSources_855457_23.stderr parsl.dataflow.dflow INFO: Task 3331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6052bd26c98173b7d3f19d52e59eb1c9 with result from task 3331 parsl.dataflow.dflow DEBUG: Task 3331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c338c74e-2b20-48d8-9759-4a910be38d4b_isrForCrosstalkSources_855442_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c338c74e-2b20-48d8-9759-4a910be38d4b_isrForCrosstalkSources_855442_18.stderr parsl.dataflow.dflow INFO: Task 3303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e747b9a254858efee844e4ec585207ad with result from task 3303 parsl.dataflow.dflow DEBUG: Task 3303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0940b16c-c90f-44c5-b5fc-b420fd90ad91_isrForCrosstalkSources_855407_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0940b16c-c90f-44c5-b5fc-b420fd90ad91_isrForCrosstalkSources_855407_28.stderr parsl.dataflow.dflow INFO: Task 3584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97c2aa90adb1d45acbc897b402386d07 with result from task 3584 parsl.dataflow.dflow DEBUG: Task 3584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f601338e-8b75-4c17-9d5e-7849db377a02_isrForCrosstalkSources_855425_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f601338e-8b75-4c17-9d5e-7849db377a02_isrForCrosstalkSources_855425_15.stderr parsl.dataflow.dflow INFO: Task 3409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 367628b9d71f343d58c7938a1b7a9f6c with result from task 3409 parsl.dataflow.dflow DEBUG: Task 3409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ae3a5866-67e9-46c1-bc7f-620176f12045_isrForCrosstalkSources_855405_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ae3a5866-67e9-46c1-bc7f-620176f12045_isrForCrosstalkSources_855405_12.stderr parsl.dataflow.dflow INFO: Task 3456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dae8e510434defca6f9dc92f0ba64cbb with result from task 3456 parsl.dataflow.dflow DEBUG: Task 3456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d56b2bf1-bef8-49a9-bca5-2c7b6dc28a36_isrForCrosstalkSources_855459_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d56b2bf1-bef8-49a9-bca5-2c7b6dc28a36_isrForCrosstalkSources_855459_30.stderr 2024-03-09 14:54:41 proc_lsst.multi:170 [INFO] [multi] job_id 16964291 proc_lsst.multi INFO: [multi] job_id 16964291 2024-03-09 14:54:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 201 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 201 2024-03-09 14:54:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964291 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964291 parsl.executors.status_handling DEBUG: Launched block 256 on executor multi with job ID 16964291 parsl.executors.status_handling INFO: Allocated block ID 257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-257 2024-03-09 14:54:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 201 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 201 2024-03-09 14:54:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab8f7954a53c1c0744487bacbe8a2bba with result from task 3502 parsl.dataflow.dflow DEBUG: Task 3502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/736b0e7f-bfec-40a0-bfae-d210a796d396_isrForCrosstalkSources_855438_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/736b0e7f-bfec-40a0-bfae-d210a796d396_isrForCrosstalkSources_855438_39.stderr parsl.dataflow.dflow INFO: Task 3445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee34f1245bd28546ce995dfc159df9ef with result from task 3445 parsl.dataflow.dflow DEBUG: Task 3445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c001cc44-4abc-4809-9f3d-ccd46e2116e5_isrForCrosstalkSources_855459_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/c001cc44-4abc-4809-9f3d-ccd46e2116e5_isrForCrosstalkSources_855459_23.stderr parsl.dataflow.dflow INFO: Task 3526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7e2f326ef7344cddfaf9fe24deb2fe4 with result from task 3526 parsl.dataflow.dflow DEBUG: Task 3526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ec06bbbe-b3b1-4ae2-a987-e93ca6910221_isrForCrosstalkSources_855388_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ec06bbbe-b3b1-4ae2-a987-e93ca6910221_isrForCrosstalkSources_855388_39.stderr parsl.dataflow.dflow INFO: Task 3209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1fead7ce76ebf556f6444b5c251ead8 with result from task 3209 parsl.dataflow.dflow DEBUG: Task 3209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c05924d7-effd-4459-a74d-ad1ca096ae3f_isrForCrosstalkSources_855418_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c05924d7-effd-4459-a74d-ad1ca096ae3f_isrForCrosstalkSources_855418_16.stderr parsl.dataflow.dflow INFO: Task 3292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a47eee0c531ae171332e5dc7b168fd05 with result from task 3292 parsl.dataflow.dflow DEBUG: Task 3292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/86e09342-1ec0-4e83-82c3-e45202c05f5e_isrForCrosstalkSources_855430_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/86e09342-1ec0-4e83-82c3-e45202c05f5e_isrForCrosstalkSources_855430_12.stderr parsl.dataflow.dflow INFO: Task 3500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bbb3e7ce5179ef67b2d3577f7211103 with result from task 3500 parsl.dataflow.dflow DEBUG: Task 3500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f1b248ee-09cf-483d-8d15-e0a64f889688_isrForCrosstalkSources_855391_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f1b248ee-09cf-483d-8d15-e0a64f889688_isrForCrosstalkSources_855391_1.stderr parsl.dataflow.dflow INFO: Task 3415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0446782af5a38da56615f7ccab8c809c with result from task 3415 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 3415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e6e5bd5d-38c9-402b-89b4-605d8d08d500_isrForCrosstalkSources_855424_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/e6e5bd5d-38c9-402b-89b4-605d8d08d500_isrForCrosstalkSources_855424_49.stderr parsl.dataflow.dflow INFO: Task 3123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d4cc64163c822399093f01ba4488821 with result from task 3123 parsl.dataflow.dflow DEBUG: Task 3123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7c90ac0f-cc74-408a-875a-7eff7f214ae1_isrForCrosstalkSources_855390_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7c90ac0f-cc74-408a-875a-7eff7f214ae1_isrForCrosstalkSources_855390_55.stderr parsl.dataflow.dflow INFO: Task 3280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b154a898f1a2feb2e8c72db802bde7dd with result from task 3280 parsl.dataflow.dflow DEBUG: Task 3280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/36db1cbb-f095-4ac2-9283-2df3aa427d5a_isrForCrosstalkSources_855459_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/36db1cbb-f095-4ac2-9283-2df3aa427d5a_isrForCrosstalkSources_855459_40.stderr parsl.dataflow.dflow INFO: Task 3482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f6023aaeaa9d5cec887b04352067732 with result from task 3482 parsl.dataflow.dflow DEBUG: Task 3482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d7d1069f-117c-4885-9a91-d49179ad93dd_isrForCrosstalkSources_855454_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/d7d1069f-117c-4885-9a91-d49179ad93dd_isrForCrosstalkSources_855454_38.stderr parsl.dataflow.dflow INFO: Task 3515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acfd8d940d5a3d68d77ea32934a2e5ea with result from task 3515 parsl.dataflow.dflow DEBUG: Task 3515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bcadaf5a-64cc-4f2e-9a60-0b935ffd5f72_isrForCrosstalkSources_855408_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bcadaf5a-64cc-4f2e-9a60-0b935ffd5f72_isrForCrosstalkSources_855408_44.stderr parsl.dataflow.dflow INFO: Task 3548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34fb9123731e2584109df57386edb7f7 with result from task 3548 parsl.dataflow.dflow DEBUG: Task 3548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/075fb689-7fba-4377-bdab-2d5b3a9abe3b_isrForCrosstalkSources_855435_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/075fb689-7fba-4377-bdab-2d5b3a9abe3b_isrForCrosstalkSources_855435_3.stderr parsl.dataflow.dflow INFO: Task 3528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21ae3d9f70288980de8e593597ff62cc with result from task 3528 parsl.dataflow.dflow DEBUG: Task 3528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/af05fd9f-5e99-4e0b-825c-60aeafeee911_isrForCrosstalkSources_855417_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/af05fd9f-5e99-4e0b-825c-60aeafeee911_isrForCrosstalkSources_855417_4.stderr parsl.dataflow.dflow INFO: Task 3323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4454e568a6c2d9fdb1f9ebadbc6104e8 with result from task 3323 parsl.dataflow.dflow DEBUG: Task 3323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ea92af3f-4923-4fb1-879f-98b52c785648_isrForCrosstalkSources_855459_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ea92af3f-4923-4fb1-879f-98b52c785648_isrForCrosstalkSources_855459_60.stderr parsl.dataflow.dflow INFO: Task 3506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b97629b05f8a0e2e10902f90ef6dc5c8 with result from task 3506 parsl.dataflow.dflow DEBUG: Task 3506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aa38a2f-5ac7-444e-8bc9-0ac99179b63a_isrForCrosstalkSources_855411_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/4aa38a2f-5ac7-444e-8bc9-0ac99179b63a_isrForCrosstalkSources_855411_4.stderr parsl.dataflow.dflow INFO: Task 3396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4df547fc47e0c4750dfafe52c62687 with result from task 3396 parsl.dataflow.dflow DEBUG: Task 3396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/523ef4a9-7b9c-42e6-af9b-8aeee1f4bed0_isrForCrosstalkSources_855450_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/523ef4a9-7b9c-42e6-af9b-8aeee1f4bed0_isrForCrosstalkSources_855450_53.stderr parsl.dataflow.dflow INFO: Task 3624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a452bf647bbfbd66743d92a710df0536 with result from task 3624 parsl.dataflow.dflow DEBUG: Task 3624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1cef0e58-784c-4197-bb53-3d169b704f77_isrForCrosstalkSources_855399_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1cef0e58-784c-4197-bb53-3d169b704f77_isrForCrosstalkSources_855399_8.stderr 2024-03-09 14:54:42 proc_lsst.multi:170 [INFO] [multi] job_id 16964292 proc_lsst.multi INFO: [multi] job_id 16964292 2024-03-09 14:54:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 202 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 202 2024-03-09 14:54:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964292 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964292 parsl.executors.status_handling DEBUG: Launched block 257 on executor multi with job ID 16964292 parsl.executors.status_handling INFO: Allocated block ID 258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-258 2024-03-09 14:54:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 2024-03-09 14:54:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 202 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 202 2024-03-09 14:54:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac6b3dd68b4f6500e805bf56777250fa with result from task 3208 parsl.dataflow.dflow DEBUG: Task 3208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5ca0df74-ad1b-465e-95ee-d654dc279e2b_isrForCrosstalkSources_855395_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/5ca0df74-ad1b-465e-95ee-d654dc279e2b_isrForCrosstalkSources_855395_17.stderr parsl.dataflow.dflow INFO: Task 3278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51a8cdc4d186d843a71332c0c600e1da with result from task 3278 parsl.dataflow.dflow DEBUG: Task 3278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea44618e-751c-4ece-9905-f5c10a54dcca_isrForCrosstalkSources_855382_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/ea44618e-751c-4ece-9905-f5c10a54dcca_isrForCrosstalkSources_855382_36.stderr parsl.dataflow.dflow INFO: Task 3380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d51011386f3f17c73742f971d4001010 with result from task 3380 parsl.dataflow.dflow DEBUG: Task 3380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/8b1647d2-9a27-41d5-b3af-b27dd8e946f1_isrForCrosstalkSources_855415_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/8b1647d2-9a27-41d5-b3af-b27dd8e946f1_isrForCrosstalkSources_855415_48.stderr parsl.dataflow.dflow INFO: Task 3516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e36b8a1fa54b8de35aa81f8a47a758 with result from task 3516 parsl.dataflow.dflow DEBUG: Task 3516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e722c02-c54f-461e-8449-4204f0b2280c_isrForCrosstalkSources_855451_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/6e722c02-c54f-461e-8449-4204f0b2280c_isrForCrosstalkSources_855451_38.stderr parsl.dataflow.dflow INFO: Task 3417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 526537aee93bf17347a090db3e86a889 with result from task 3417 parsl.dataflow.dflow DEBUG: Task 3417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/87ac3516-649e-4402-921c-29cab0773676_isrForCrosstalkSources_855419_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/87ac3516-649e-4402-921c-29cab0773676_isrForCrosstalkSources_855419_7.stderr parsl.dataflow.dflow INFO: Task 3428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 777ca244d48d2bc47f75c7ad37997a92 with result from task 3428 parsl.dataflow.dflow DEBUG: Task 3428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f0a4a63a-e794-4fb3-9105-b4219a67592b_isrForCrosstalkSources_855424_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f0a4a63a-e794-4fb3-9105-b4219a67592b_isrForCrosstalkSources_855424_48.stderr parsl.dataflow.dflow INFO: Task 3309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 043e3698fa7541a222fe409d3d4c3999 with result from task 3309 parsl.dataflow.dflow DEBUG: Task 3309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/89e618a8-b74d-4413-9b46-73c5884eb316_isrForCrosstalkSources_855441_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/89e618a8-b74d-4413-9b46-73c5884eb316_isrForCrosstalkSources_855441_38.stderr parsl.dataflow.dflow INFO: Task 3338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f634e31eccc4ece2ee86a3b6ceab490 with result from task 3338 parsl.dataflow.dflow DEBUG: Task 3338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6ca0c531-4eee-4836-a916-e2f15f7d92a2_isrForCrosstalkSources_855447_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/6ca0c531-4eee-4836-a916-e2f15f7d92a2_isrForCrosstalkSources_855447_53.stderr parsl.dataflow.dflow INFO: Task 3512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f51b4c827d51c95a8c0b92446f72d57 with result from task 3512 parsl.dataflow.dflow DEBUG: Task 3512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/eea59fe7-88b4-4995-b3f7-8f1897b38c09_isrForCrosstalkSources_855453_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/eea59fe7-88b4-4995-b3f7-8f1897b38c09_isrForCrosstalkSources_855453_3.stderr parsl.dataflow.dflow INFO: Task 3542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3abd9d35777170fc77bbcaaf24717c99 with result from task 3542 parsl.dataflow.dflow DEBUG: Task 3542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/4d93cb8d-526e-496f-89c7-659ae8fe8e4b_isrForCrosstalkSources_855453_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/4d93cb8d-526e-496f-89c7-659ae8fe8e4b_isrForCrosstalkSources_855453_44.stderr parsl.dataflow.dflow INFO: Task 3364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1145aa37034b6d392f7fe8bdd53bd190 with result from task 3364 parsl.dataflow.dflow DEBUG: Task 3364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/61f50193-ee37-4641-80c4-3d732595a32f_isrForCrosstalkSources_855419_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/61f50193-ee37-4641-80c4-3d732595a32f_isrForCrosstalkSources_855419_61.stderr parsl.dataflow.dflow INFO: Task 3483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 567246927b6881bbd324538badb876b9 with result from task 3483 parsl.dataflow.dflow DEBUG: Task 3483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/d3e366d4-fb5e-4d52-a0ce-3a1403076d7e_isrForCrosstalkSources_855452_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/d3e366d4-fb5e-4d52-a0ce-3a1403076d7e_isrForCrosstalkSources_855452_12.stderr parsl.dataflow.dflow INFO: Task 3308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89bf9b43dd741e18f46e470c020c3dd5 with result from task 3308 parsl.dataflow.dflow DEBUG: Task 3308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/365eb78c-ff63-41c8-9c68-e94e091f4583_isrForCrosstalkSources_855402_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/365eb78c-ff63-41c8-9c68-e94e091f4583_isrForCrosstalkSources_855402_7.stderr parsl.dataflow.dflow INFO: Task 3488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3338eb778acf353eb32e9214e11dbbdf with result from task 3488 parsl.dataflow.dflow DEBUG: Task 3488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f453cbd2-37b3-40d7-b4cd-9b5c14045173_isrForCrosstalkSources_855444_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/f453cbd2-37b3-40d7-b4cd-9b5c14045173_isrForCrosstalkSources_855444_17.stderr parsl.dataflow.dflow INFO: Task 3267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfe1b001d999af508d9fb0ee0b1ee70b with result from task 3267 parsl.dataflow.dflow DEBUG: Task 3267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6e0853a6-2a6b-47b0-9b1e-c0f8e6da957a_isrForCrosstalkSources_855411_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/6e0853a6-2a6b-47b0-9b1e-c0f8e6da957a_isrForCrosstalkSources_855411_2.stderr parsl.dataflow.dflow INFO: Task 3534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd685c9b0e2f72c4027f2836ccfcaaf5 with result from task 3534 parsl.dataflow.dflow DEBUG: Task 3534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/89d331b8-4c9f-4e05-a1f7-cff9fb66e961_isrForCrosstalkSources_855400_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/89d331b8-4c9f-4e05-a1f7-cff9fb66e961_isrForCrosstalkSources_855400_27.stderr parsl.dataflow.dflow INFO: Task 3589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d13a5ff29968f217933deef74a827f8d with result from task 3589 parsl.dataflow.dflow DEBUG: Task 3589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b8b5ea09-cd27-4eae-8b2b-109c0c88657d_isrForCrosstalkSources_855453_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b8b5ea09-cd27-4eae-8b2b-109c0c88657d_isrForCrosstalkSources_855453_56.stderr parsl.dataflow.dflow INFO: Task 3304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65a4b7cb3981e0538bb6c4ad18f2be0c with result from task 3304 parsl.dataflow.dflow DEBUG: Task 3304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/205f4a96-6b34-4924-868f-df8b74a70a9c_isrForCrosstalkSources_855399_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/205f4a96-6b34-4924-868f-df8b74a70a9c_isrForCrosstalkSources_855399_42.stderr parsl.dataflow.dflow INFO: Task 3392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9165593dca94f7b2cc1ee6500f506197 with result from task 3392 parsl.dataflow.dflow DEBUG: Task 3392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a0dc2f0e-24ec-4990-8a3a-b88320f20964_isrForCrosstalkSources_855385_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a0dc2f0e-24ec-4990-8a3a-b88320f20964_isrForCrosstalkSources_855385_30.stderr parsl.dataflow.dflow INFO: Task 3118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ebd3712a5d4dadfa697f1a2cf0088b1 with result from task 3118 parsl.dataflow.dflow DEBUG: Task 3118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/81e7cd0e-81ee-48b6-b6a2-e6af2cc7093c_isrForCrosstalkSources_855418_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/81e7cd0e-81ee-48b6-b6a2-e6af2cc7093c_isrForCrosstalkSources_855418_54.stderr parsl.dataflow.dflow INFO: Task 3322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 038d69a0315ae60a3515da256ec87b0e with result from task 3322 parsl.dataflow.dflow DEBUG: Task 3322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e428c86d-57a7-41ce-859d-20110be2c55e_isrForCrosstalkSources_855411_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/e428c86d-57a7-41ce-859d-20110be2c55e_isrForCrosstalkSources_855411_28.stderr parsl.dataflow.dflow INFO: Task 3408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 974ce5981df1a63d6832e1ae9cfeee60 with result from task 3408 parsl.dataflow.dflow DEBUG: Task 3408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/367da439-8cc9-46eb-9289-7a8e47221f11_isrForCrosstalkSources_855423_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/367da439-8cc9-46eb-9289-7a8e47221f11_isrForCrosstalkSources_855423_59.stderr parsl.dataflow.dflow INFO: Task 3325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43a9623cf626f3829c6ad5922ce1871a with result from task 3325 parsl.dataflow.dflow DEBUG: Task 3325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/46fdd612-820d-4183-a519-730c0ec547e6_isrForCrosstalkSources_855385_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/46fdd612-820d-4183-a519-730c0ec547e6_isrForCrosstalkSources_855385_41.stderr parsl.dataflow.dflow INFO: Task 3161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa10f3dc3c09f60da28da9a247a7d4b2 with result from task 3161 parsl.dataflow.dflow DEBUG: Task 3161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5a9557ac-0f3b-4db3-a104-2b4242c46285_isrForCrosstalkSources_855382_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5a9557ac-0f3b-4db3-a104-2b4242c46285_isrForCrosstalkSources_855382_60.stderr parsl.dataflow.dflow INFO: Task 3556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e67449ba4d73d7e5751bb0785edcc14 with result from task 3556 parsl.dataflow.dflow DEBUG: Task 3556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/83f2759d-27b2-4012-8972-d5a6ce1fd9bf_isrForCrosstalkSources_855450_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/83f2759d-27b2-4012-8972-d5a6ce1fd9bf_isrForCrosstalkSources_855450_4.stderr parsl.dataflow.dflow INFO: Task 3327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0438d112ebab515ef55a731cd29d849f with result from task 3327 parsl.dataflow.dflow DEBUG: Task 3327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/29a520b1-18b7-44d5-889c-630966fead6b_isrForCrosstalkSources_855457_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/29a520b1-18b7-44d5-889c-630966fead6b_isrForCrosstalkSources_855457_25.stderr parsl.dataflow.dflow INFO: Task 3241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2105285125aad0332ede01d9e980d699 with result from task 3241 parsl.dataflow.dflow DEBUG: Task 3241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e804c7ee-adbb-4106-a515-86f9ff88ca12_isrForCrosstalkSources_855382_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e804c7ee-adbb-4106-a515-86f9ff88ca12_isrForCrosstalkSources_855382_57.stderr parsl.dataflow.dflow INFO: Task 3533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05581a07f84a4c5c98c3952a0933106f with result from task 3533 parsl.dataflow.dflow DEBUG: Task 3533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/9060c877-9179-42bc-81a5-50f3036ec3a1_isrForCrosstalkSources_855433_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/9060c877-9179-42bc-81a5-50f3036ec3a1_isrForCrosstalkSources_855433_9.stderr parsl.dataflow.dflow INFO: Task 3348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 556207e9396dd4cf605092b6e27aff1b with result from task 3348 parsl.dataflow.dflow DEBUG: Task 3348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/181468a7-9bb0-49ef-a35f-8319c917053a_isrForCrosstalkSources_855431_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/181468a7-9bb0-49ef-a35f-8319c917053a_isrForCrosstalkSources_855431_22.stderr parsl.dataflow.dflow INFO: Task 3558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f62823c6030141fd828bbeb8c9ceb512 with result from task 3558 parsl.dataflow.dflow DEBUG: Task 3558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b017210-7aeb-4dfd-a6e7-511c7893ea6f_isrForCrosstalkSources_855398_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b017210-7aeb-4dfd-a6e7-511c7893ea6f_isrForCrosstalkSources_855398_3.stderr parsl.dataflow.dflow INFO: Task 3498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d6a189002cfcd06e940ebbbeb20dd9 with result from task 3498 parsl.dataflow.dflow DEBUG: Task 3498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5db3f52f-effc-4255-81cd-c2e86405c5e6_isrForCrosstalkSources_855382_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5db3f52f-effc-4255-81cd-c2e86405c5e6_isrForCrosstalkSources_855382_18.stderr parsl.dataflow.dflow INFO: Task 3479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 890837931747e95cb8f5f5fb08ec5531 with result from task 3479 parsl.dataflow.dflow DEBUG: Task 3479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e2c96db1-e0e6-4420-a5a0-d7ebc64236ee_isrForCrosstalkSources_855408_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e2c96db1-e0e6-4420-a5a0-d7ebc64236ee_isrForCrosstalkSources_855408_62.stderr parsl.dataflow.dflow INFO: Task 3529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17fb824cdcf30e0228f008a376f93139 with result from task 3529 parsl.dataflow.dflow DEBUG: Task 3529 checkpointed parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff419ba9-31c3-4dbb-b78f-86a0def16b7b_isrForCrosstalkSources_855405_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ff419ba9-31c3-4dbb-b78f-86a0def16b7b_isrForCrosstalkSources_855405_48.stderr parsl.dataflow.dflow INFO: Task 3363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13dbfd5a7c296e37ef1124acd7d9e45b with result from task 3363 parsl.dataflow.dflow DEBUG: Task 3363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80e88815-87b0-4a0a-a9ec-9406785584da_isrForCrosstalkSources_855426_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/80e88815-87b0-4a0a-a9ec-9406785584da_isrForCrosstalkSources_855426_40.stderr parsl.dataflow.dflow INFO: Task 3461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cec3543fe957d33f680e0397cc9269e with result from task 3461 parsl.dataflow.dflow DEBUG: Task 3461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5a2cc04c-1889-489d-b5ed-199f4d7812c8_isrForCrosstalkSources_855424_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/5a2cc04c-1889-489d-b5ed-199f4d7812c8_isrForCrosstalkSources_855424_12.stderr 2024-03-09 14:54:45 proc_lsst.multi:170 [INFO] [multi] job_id 16964293 proc_lsst.multi INFO: [multi] job_id 16964293 2024-03-09 14:54:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 203 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 203 2024-03-09 14:54:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964293 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964293 parsl.executors.status_handling DEBUG: Launched block 258 on executor multi with job ID 16964293 parsl.executors.status_handling INFO: Allocated block ID 259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-259 2024-03-09 14:54:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 2024-03-09 14:54:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:54:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:54:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:54:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:54:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:54:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:54:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:54:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:54:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:54:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:54:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 203 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 203 2024-03-09 14:54:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:54:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4210b4d964607f9290cd86ffb441648c with result from task 3574 parsl.dataflow.dflow DEBUG: Task 3574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ffc4a808-5dfc-482e-b4e1-b2d89266990f_isrForCrosstalkSources_855436_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ffc4a808-5dfc-482e-b4e1-b2d89266990f_isrForCrosstalkSources_855436_1.stderr parsl.dataflow.dflow INFO: Task 3389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6069938b714966539b65884ed69712fd with result from task 3389 parsl.dataflow.dflow DEBUG: Task 3389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/168baace-19a4-4344-b925-af9ee984bbd8_isrForCrosstalkSources_855407_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/168baace-19a4-4344-b925-af9ee984bbd8_isrForCrosstalkSources_855407_3.stderr parsl.dataflow.dflow INFO: Task 3252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 166f4f7f39c908fa22ed5fe1f792b3d8 with result from task 3252 parsl.dataflow.dflow DEBUG: Task 3252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/090d2c07-7edd-45eb-902e-bed49b82a37a_isrForCrosstalkSources_855432_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/090d2c07-7edd-45eb-902e-bed49b82a37a_isrForCrosstalkSources_855432_9.stderr parsl.dataflow.dflow INFO: Task 3501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3842905ff48376dffa3ff118e9f1af6 with result from task 3501 parsl.dataflow.dflow DEBUG: Task 3501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/495c4db4-2bcc-4f6e-aab3-b19195e05d2e_isrForCrosstalkSources_855415_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/495c4db4-2bcc-4f6e-aab3-b19195e05d2e_isrForCrosstalkSources_855415_39.stderr parsl.dataflow.dflow INFO: Task 3433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2559acdb34c2fa6bb759b31e04e4f91 with result from task 3433 parsl.dataflow.dflow DEBUG: Task 3433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/0237d247-1bec-4ac7-bf8c-5d55d0a380d7_isrForCrosstalkSources_855448_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/0237d247-1bec-4ac7-bf8c-5d55d0a380d7_isrForCrosstalkSources_855448_11.stderr parsl.dataflow.dflow INFO: Task 3228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a5d57e3f4c0d836cadc1db2f44fb7b with result from task 3228 parsl.dataflow.dflow DEBUG: Task 3228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/82342c8c-1628-4f59-a68b-6e71b77bd72b_isrForCrosstalkSources_855435_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/82342c8c-1628-4f59-a68b-6e71b77bd72b_isrForCrosstalkSources_855435_37.stderr parsl.dataflow.dflow INFO: Task 3414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9430fd2c2448b8c337d5f547ccfe32c1 with result from task 3414 parsl.dataflow.dflow DEBUG: Task 3414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/b2e8baf0-5372-4e63-a0c4-e5dca8aee349_isrForCrosstalkSources_855401_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/b2e8baf0-5372-4e63-a0c4-e5dca8aee349_isrForCrosstalkSources_855401_48.stderr parsl.dataflow.dflow INFO: Task 3551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f941ca4373482e8eefa580b62dc8dbea with result from task 3551 parsl.dataflow.dflow DEBUG: Task 3551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5002d2ee-b24d-4f20-afc9-2e0d4773e111_isrForCrosstalkSources_855425_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5002d2ee-b24d-4f20-afc9-2e0d4773e111_isrForCrosstalkSources_855425_20.stderr parsl.dataflow.dflow INFO: Task 3359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e50a0f8a243f648381b333a0dcf5026f with result from task 3359 parsl.dataflow.dflow DEBUG: Task 3359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a4d4bc13-1207-4fbb-898f-2718e2e1d708_isrForCrosstalkSources_855456_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/a4d4bc13-1207-4fbb-898f-2718e2e1d708_isrForCrosstalkSources_855456_50.stderr parsl.dataflow.dflow INFO: Task 3508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e48a8091641ed7ef3250cd3b34d277b with result from task 3508 parsl.dataflow.dflow DEBUG: Task 3508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/14425dfb-5615-4049-901d-662a0203d68d_isrForCrosstalkSources_855388_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/14425dfb-5615-4049-901d-662a0203d68d_isrForCrosstalkSources_855388_33.stderr parsl.dataflow.dflow INFO: Task 3285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b5b4122d86225c07a1ddd5537caa13 with result from task 3285 parsl.dataflow.dflow DEBUG: Task 3285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/24760688-6112-446b-af2b-eacbf0f0b025_isrForCrosstalkSources_855420_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/24760688-6112-446b-af2b-eacbf0f0b025_isrForCrosstalkSources_855420_48.stderr parsl.dataflow.dflow INFO: Task 3332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe9f60551a07e998d27dfbfa53fd4769 with result from task 3332 parsl.dataflow.dflow DEBUG: Task 3332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/106ca6c3-f4f4-46e6-afe2-4c7565a39d11_isrForCrosstalkSources_855414_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/106ca6c3-f4f4-46e6-afe2-4c7565a39d11_isrForCrosstalkSources_855414_20.stderr parsl.dataflow.dflow INFO: Task 3376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf3b5abb528b22b2d829ff413971acea with result from task 3376 parsl.dataflow.dflow DEBUG: Task 3376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9062b442-854e-4d95-b469-19b7ca1468d9_isrForCrosstalkSources_855418_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/9062b442-854e-4d95-b469-19b7ca1468d9_isrForCrosstalkSources_855418_15.stderr parsl.dataflow.dflow INFO: Task 3553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 590795d2b8881da06ad4de6db2e1ef5f with result from task 3553 parsl.dataflow.dflow DEBUG: Task 3553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/73675dac-6580-47b5-8c83-807d806e9c7b_isrForCrosstalkSources_855431_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/73675dac-6580-47b5-8c83-807d806e9c7b_isrForCrosstalkSources_855431_45.stderr parsl.dataflow.dflow INFO: Task 3448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a878768a7d5e97fc2c7f5233ff7716b with result from task 3448 parsl.dataflow.dflow DEBUG: Task 3448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/12a9d19e-9e59-4625-92c1-44c1fc93a643_isrForCrosstalkSources_855428_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/12a9d19e-9e59-4625-92c1-44c1fc93a643_isrForCrosstalkSources_855428_5.stderr parsl.dataflow.dflow INFO: Task 3454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5e6a5f12018921aff03dd2783c147b9 with result from task 3454 parsl.dataflow.dflow DEBUG: Task 3454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ed2b08-0c5c-42c7-a18e-a542a009413e_isrForCrosstalkSources_855396_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ed2b08-0c5c-42c7-a18e-a542a009413e_isrForCrosstalkSources_855396_11.stderr parsl.dataflow.dflow INFO: Task 3544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 823ad9d0f1be4ee1932e22b62c76d181 with result from task 3544 parsl.dataflow.dflow DEBUG: Task 3544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/74f4c5f0-49e0-4499-b999-7d4e2d1e34f4_isrForCrosstalkSources_855413_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/74f4c5f0-49e0-4499-b999-7d4e2d1e34f4_isrForCrosstalkSources_855413_9.stderr parsl.dataflow.dflow INFO: Task 3538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21a686607d97a085dd73767fdaac511c with result from task 3538 parsl.dataflow.dflow DEBUG: Task 3538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c2bac645-4dd9-4b97-afe0-928ae0897ce5_isrForCrosstalkSources_855394_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c2bac645-4dd9-4b97-afe0-928ae0897ce5_isrForCrosstalkSources_855394_29.stderr parsl.dataflow.dflow INFO: Task 3263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2dbeac2b93e186b2812d9afe8706280 with result from task 3263 parsl.dataflow.dflow DEBUG: Task 3263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cdbc92ab-6cef-4b46-8e53-b384abdb789f_isrForCrosstalkSources_855412_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/cdbc92ab-6cef-4b46-8e53-b384abdb789f_isrForCrosstalkSources_855412_35.stderr parsl.dataflow.dflow INFO: Task 3517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a919e94faa609388a121a9b44aff372e with result from task 3517 parsl.dataflow.dflow DEBUG: Task 3517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/344b2869-e8bd-4393-8075-44ac5b673c93_isrForCrosstalkSources_855446_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/344b2869-e8bd-4393-8075-44ac5b673c93_isrForCrosstalkSources_855446_29.stderr parsl.dataflow.dflow INFO: Task 3330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 551927b34fb40cc5c09ac1baa64d250e with result from task 3330 parsl.dataflow.dflow DEBUG: Task 3330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3c426b99-c7d7-4c82-b1aa-ad387efcf019_isrForCrosstalkSources_855405_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3c426b99-c7d7-4c82-b1aa-ad387efcf019_isrForCrosstalkSources_855405_55.stderr parsl.dataflow.dflow INFO: Task 3455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b824932bb767db7acbc3842b0ca20291 with result from task 3455 parsl.dataflow.dflow DEBUG: Task 3455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/a505484b-7c28-465d-91d6-a9c0aec6a4a2_isrForCrosstalkSources_855425_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/a505484b-7c28-465d-91d6-a9c0aec6a4a2_isrForCrosstalkSources_855425_12.stderr parsl.dataflow.dflow INFO: Task 3236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1cf5779dafa0514ef29b9322171a9be with result from task 3236 parsl.dataflow.dflow DEBUG: Task 3236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/182de1bc-f421-40c9-a3c0-602eb6bc4bf0_isrForCrosstalkSources_855444_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/182de1bc-f421-40c9-a3c0-602eb6bc4bf0_isrForCrosstalkSources_855444_48.stderr parsl.dataflow.dflow INFO: Task 3301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef91110dc5eeef6372dc7cd38e5b4e2 with result from task 3301 parsl.dataflow.dflow DEBUG: Task 3301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/df80d7bc-c85d-4306-b3fa-f8f6ec0a3b4e_isrForCrosstalkSources_855427_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/df80d7bc-c85d-4306-b3fa-f8f6ec0a3b4e_isrForCrosstalkSources_855427_35.stderr parsl.dataflow.dflow INFO: Task 3449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbef0870c140d8f47b13c6ccb746786d with result from task 3449 parsl.dataflow.dflow DEBUG: Task 3449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/bdc9a131-1688-40fa-8ddd-2da3ff2ecf7b_isrForCrosstalkSources_855449_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/bdc9a131-1688-40fa-8ddd-2da3ff2ecf7b_isrForCrosstalkSources_855449_43.stderr parsl.dataflow.dflow INFO: Task 3522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 063ac63f91996f374fb064bfaf2a500c with result from task 3522 parsl.dataflow.dflow DEBUG: Task 3522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/48b28375-c8bc-4f21-9b09-7ecb4f0b602d_isrForCrosstalkSources_855415_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/48b28375-c8bc-4f21-9b09-7ecb4f0b602d_isrForCrosstalkSources_855415_1.stderr parsl.dataflow.dflow INFO: Task 3535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08a509d5197e67ae505ba459b95d15a9 with result from task 3535 parsl.dataflow.dflow DEBUG: Task 3535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c14149e0-3e36-4cb6-97ad-222e666a73d5_isrForCrosstalkSources_855397_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c14149e0-3e36-4cb6-97ad-222e666a73d5_isrForCrosstalkSources_855397_29.stderr parsl.dataflow.dflow INFO: Task 3580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08399a30416a83d4573a4ef73baa1be2 with result from task 3580 parsl.dataflow.dflow DEBUG: Task 3580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b7236540-7e34-4a25-9820-d653658c898f_isrForCrosstalkSources_855440_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b7236540-7e34-4a25-9820-d653658c898f_isrForCrosstalkSources_855440_13.stderr parsl.dataflow.dflow INFO: Task 3260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6450a931ee5991448e438c1b5209e92f with result from task 3260 parsl.dataflow.dflow DEBUG: Task 3260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/77ca0aa2-61bc-4402-b96b-16071fd832b8_isrForCrosstalkSources_855399_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/77ca0aa2-61bc-4402-b96b-16071fd832b8_isrForCrosstalkSources_855399_62.stderr parsl.dataflow.dflow INFO: Task 3270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b4650df60176552418b8da2c4dad47a with result from task 3270 parsl.dataflow.dflow DEBUG: Task 3270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a48cd0f6-7729-4428-8e8d-06ad827c31b3_isrForCrosstalkSources_855452_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a48cd0f6-7729-4428-8e8d-06ad827c31b3_isrForCrosstalkSources_855452_42.stderr parsl.dataflow.dflow INFO: Task 3447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32ae092917af3a23a35e316d587f0fd3 with result from task 3447 parsl.dataflow.dflow DEBUG: Task 3447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d3102e7a-571b-4672-a616-481c6766a3a6_isrForCrosstalkSources_855407_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d3102e7a-571b-4672-a616-481c6766a3a6_isrForCrosstalkSources_855407_36.stderr parsl.dataflow.dflow INFO: Task 3310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aa1becbbf22094bd860e79a382d390d with result from task 3310 parsl.dataflow.dflow DEBUG: Task 3310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/112f10eb-0231-475f-94b9-f970cd9d98da_isrForCrosstalkSources_855459_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/112f10eb-0231-475f-94b9-f970cd9d98da_isrForCrosstalkSources_855459_16.stderr parsl.dataflow.dflow INFO: Task 3531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a94bcf387ac4be152f7988ec6ce7f947 with result from task 3531 parsl.dataflow.dflow DEBUG: Task 3531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/67987cbf-cd26-4f2a-8b0f-aa70c5536033_isrForCrosstalkSources_855407_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/67987cbf-cd26-4f2a-8b0f-aa70c5536033_isrForCrosstalkSources_855407_56.stderr parsl.dataflow.dflow INFO: Task 3543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11f506ba076b9d72b655bf2ee2baf826 with result from task 3543 parsl.dataflow.dflow DEBUG: Task 3543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b22d770a-7622-4780-bb88-8658c0181dab_isrForCrosstalkSources_855438_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b22d770a-7622-4780-bb88-8658c0181dab_isrForCrosstalkSources_855438_30.stderr parsl.dataflow.dflow INFO: Task 3704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7a2e3523ae325749eeafc2841703a19 with result from task 3704 parsl.dataflow.dflow DEBUG: Task 3704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0a1973c5-f59c-4ee4-b72f-7bdbbe4ae6ef_isrForCrosstalkSources_855418_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0a1973c5-f59c-4ee4-b72f-7bdbbe4ae6ef_isrForCrosstalkSources_855418_32.stderr parsl.dataflow.dflow INFO: Task 3475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e82ba1de454785fc730d684e98b305fd with result from task 3475 parsl.dataflow.dflow DEBUG: Task 3475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3ee0f50e-03df-4478-88ea-1d3fbdf03441_isrForCrosstalkSources_855400_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/3ee0f50e-03df-4478-88ea-1d3fbdf03441_isrForCrosstalkSources_855400_49.stderr parsl.dataflow.dflow INFO: Task 3373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d950695bbcb8def6a36f769882fbd848 with result from task 3373 parsl.dataflow.dflow DEBUG: Task 3373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/72d91b75-1529-4760-98de-116815f11a66_isrForCrosstalkSources_855408_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/72d91b75-1529-4760-98de-116815f11a66_isrForCrosstalkSources_855408_47.stderr parsl.dataflow.dflow INFO: Task 3349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 055b0070861bf2b3ae08e94b9629c5e7 with result from task 3349 parsl.dataflow.dflow DEBUG: Task 3349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/20814ec4-7b25-40ac-8912-bf446f434f55_isrForCrosstalkSources_855387_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/20814ec4-7b25-40ac-8912-bf446f434f55_isrForCrosstalkSources_855387_28.stderr parsl.dataflow.dflow INFO: Task 3789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 618b74762ad5f07d29374fe67d828245 with result from task 3789 parsl.dataflow.dflow DEBUG: Task 3789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bd1844c6-3d71-4a22-916f-dffec30e8ebf_isrForCrosstalkSources_855405_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/bd1844c6-3d71-4a22-916f-dffec30e8ebf_isrForCrosstalkSources_855405_23.stderr parsl.dataflow.dflow INFO: Task 3698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ef5270fe9037f256d6ad2550b6de4d7 with result from task 3698 parsl.dataflow.dflow DEBUG: Task 3698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f52a0f10-3780-4bc4-af41-23ef9c2dff81_isrForCrosstalkSources_855400_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f52a0f10-3780-4bc4-af41-23ef9c2dff81_isrForCrosstalkSources_855400_19.stderr parsl.dataflow.dflow INFO: Task 3511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5caffc90d45d69c47f0bdf0cd65fc5a7 with result from task 3511 parsl.dataflow.dflow DEBUG: Task 3511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/61041d6d-ffa2-4c31-b270-46e53784f74b_isrForCrosstalkSources_855434_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/61041d6d-ffa2-4c31-b270-46e53784f74b_isrForCrosstalkSources_855434_21.stderr parsl.dataflow.dflow INFO: Task 3756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93f9bf49ede719842c94e49fa6e2a75d with result from task 3756 parsl.dataflow.dflow DEBUG: Task 3756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/058f818a-dc13-4dd4-a4c5-0527ccc87076_isrForCrosstalkSources_855443_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/058f818a-dc13-4dd4-a4c5-0527ccc87076_isrForCrosstalkSources_855443_25.stderr parsl.dataflow.dflow INFO: Task 3145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d0e65fa279d6e139f73a6a3e2b3d22f with result from task 3145 parsl.dataflow.dflow DEBUG: Task 3145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/aaa52fb6-6f71-474d-bf70-75c34f2b9560_isrForCrosstalkSources_855441_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/aaa52fb6-6f71-474d-bf70-75c34f2b9560_isrForCrosstalkSources_855441_42.stderr parsl.dataflow.dflow INFO: Task 3248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd7994351275e11afeab30ae4c500eb0 with result from task 3248 parsl.dataflow.dflow DEBUG: Task 3248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/21b88370-a577-4dda-ba3a-a323d21263e9_isrForCrosstalkSources_855445_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/21b88370-a577-4dda-ba3a-a323d21263e9_isrForCrosstalkSources_855445_53.stderr parsl.dataflow.dflow INFO: Task 3712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b904f4be2bf273c2a332c20c68979630 with result from task 3712 parsl.dataflow.dflow DEBUG: Task 3712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f764e681-afb0-45ff-be79-27cadb5c2dff_isrForCrosstalkSources_855390_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f764e681-afb0-45ff-be79-27cadb5c2dff_isrForCrosstalkSources_855390_45.stderr parsl.dataflow.dflow INFO: Task 3275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2a9ee8f588fd453591022c0eb41e1e8 with result from task 3275 parsl.dataflow.dflow DEBUG: Task 3275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/2ad3dedf-3ef1-4307-9b11-b3aed27ebaf3_isrForCrosstalkSources_855446_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/2ad3dedf-3ef1-4307-9b11-b3aed27ebaf3_isrForCrosstalkSources_855446_52.stderr parsl.dataflow.dflow INFO: Task 3596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e11c718d290dcdb3e04c447231861e5 with result from task 3596 parsl.dataflow.dflow DEBUG: Task 3596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e2ba2e16-92cf-4c29-8807-d70701f6f9f0_isrForCrosstalkSources_855458_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e2ba2e16-92cf-4c29-8807-d70701f6f9f0_isrForCrosstalkSources_855458_36.stderr parsl.dataflow.dflow INFO: Task 3344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85607e86f48d4d6d46dc54b0d5a6af7a with result from task 3344 parsl.dataflow.dflow DEBUG: Task 3344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/bfdaf816-be8f-4f54-830b-fac0e1c859e2_isrForCrosstalkSources_855445_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/bfdaf816-be8f-4f54-830b-fac0e1c859e2_isrForCrosstalkSources_855445_58.stderr parsl.dataflow.dflow INFO: Task 3635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77d14e30431d3c7f4bee7932b6534bdc with result from task 3635 parsl.dataflow.dflow DEBUG: Task 3635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cbf5e53b-33a7-4807-be5b-73c7c79a1c0b_isrForCrosstalkSources_855441_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/cbf5e53b-33a7-4807-be5b-73c7c79a1c0b_isrForCrosstalkSources_855441_45.stderr parsl.dataflow.dflow INFO: Task 3629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e108675d90f005934ad672d167356c4f with result from task 3629 parsl.dataflow.dflow DEBUG: Task 3629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fcd156-11b8-439a-9a66-9482c859bf71_isrForCrosstalkSources_855420_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fcd156-11b8-439a-9a66-9482c859bf71_isrForCrosstalkSources_855420_17.stderr parsl.dataflow.dflow INFO: Task 3343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc7e19ebdccff9cabf61200a69f70387 with result from task 3343 parsl.dataflow.dflow DEBUG: Task 3343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a54b702d-c78a-4474-a73b-197dce6bb0d4_isrForCrosstalkSources_855448_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a54b702d-c78a-4474-a73b-197dce6bb0d4_isrForCrosstalkSources_855448_57.stderr parsl.dataflow.dflow INFO: Task 3490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b10fa431fc7c96307af9855dbbfc66c with result from task 3490 parsl.dataflow.dflow DEBUG: Task 3490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/31a51583-e414-4591-982f-c026508d2be3_isrForCrosstalkSources_855426_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/31a51583-e414-4591-982f-c026508d2be3_isrForCrosstalkSources_855426_25.stderr parsl.dataflow.dflow INFO: Task 3328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 292d649cdb7256c3a58a526f9e19a79f with result from task 3328 parsl.dataflow.dflow DEBUG: Task 3328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/cf2baab9-89ff-40b0-91bd-c3ab17dcce72_isrForCrosstalkSources_855438_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/cf2baab9-89ff-40b0-91bd-c3ab17dcce72_isrForCrosstalkSources_855438_35.stderr parsl.dataflow.dflow INFO: Task 3822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5da492c30873ee35b87319d1bdb1760 with result from task 3822 parsl.dataflow.dflow DEBUG: Task 3822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1208293f-d019-4d7d-9d4b-ae3b888274f1_isrForCrosstalkSources_855427_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/1208293f-d019-4d7d-9d4b-ae3b888274f1_isrForCrosstalkSources_855427_19.stderr parsl.dataflow.dflow INFO: Task 3781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50a91dd64a23930f8ce1be0b44ff936f with result from task 3781 parsl.dataflow.dflow DEBUG: Task 3781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/1c5a289b-9930-4005-a318-33039e735fab_isrForCrosstalkSources_855386_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/1c5a289b-9930-4005-a318-33039e735fab_isrForCrosstalkSources_855386_27.stderr parsl.dataflow.dflow INFO: Task 3713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0313692aa505c34c25a485f7228ccc57 with result from task 3713 parsl.dataflow.dflow DEBUG: Task 3713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2c954487-cb98-400e-afa2-33b989965dbd_isrForCrosstalkSources_855401_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2c954487-cb98-400e-afa2-33b989965dbd_isrForCrosstalkSources_855401_8.stderr parsl.dataflow.dflow INFO: Task 3753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3413995975bfef0027007e2a874b0abd with result from task 3753 parsl.dataflow.dflow DEBUG: Task 3753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c2c2d003-5a87-4ab2-a93e-60d5fb18893e_isrForCrosstalkSources_855435_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c2c2d003-5a87-4ab2-a93e-60d5fb18893e_isrForCrosstalkSources_855435_56.stderr parsl.dataflow.dflow INFO: Task 3494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d068899311ff4d963a61d7596619f095 with result from task 3494 parsl.dataflow.dflow DEBUG: Task 3494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2fb5ac56-f970-42af-a369-f419b1dd4308_isrForCrosstalkSources_855397_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2fb5ac56-f970-42af-a369-f419b1dd4308_isrForCrosstalkSources_855397_20.stderr parsl.dataflow.dflow INFO: Task 3530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eddd0a428d85ab659a68d4efa0823707 with result from task 3530 parsl.dataflow.dflow DEBUG: Task 3530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/895aa623-9951-4abe-8e1d-af2d89568384_isrForCrosstalkSources_855395_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/895aa623-9951-4abe-8e1d-af2d89568384_isrForCrosstalkSources_855395_28.stderr parsl.dataflow.dflow INFO: Task 3656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3fa7a7b794472f536d1e144961e4a5e with result from task 3656 parsl.dataflow.dflow DEBUG: Task 3656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5b43d080-841d-4deb-bb4f-9ea7ddd8ed7e_isrForCrosstalkSources_855381_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5b43d080-841d-4deb-bb4f-9ea7ddd8ed7e_isrForCrosstalkSources_855381_15.stderr parsl.dataflow.dflow INFO: Task 3492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1de74ced59c8e3eb92b567e8a3f40794 with result from task 3492 parsl.dataflow.dflow DEBUG: Task 3492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04de1dc1-0b18-4abd-a233-8797ee17829e_isrForCrosstalkSources_855440_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/04de1dc1-0b18-4abd-a233-8797ee17829e_isrForCrosstalkSources_855440_50.stderr parsl.dataflow.dflow INFO: Task 3402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c2b846eaa10161c55f3e0bfb88343c6 with result from task 3402 parsl.dataflow.dflow DEBUG: Task 3402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3c737218-d2b3-468a-b682-950d691b21d4_isrForCrosstalkSources_855404_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/3c737218-d2b3-468a-b682-950d691b21d4_isrForCrosstalkSources_855404_39.stderr parsl.dataflow.dflow INFO: Task 3692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7deb1a38a98a3cc286d7fd667b8c6f0a with result from task 3692 parsl.dataflow.dflow DEBUG: Task 3692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3c483bc5-b8c8-427f-b564-b2da634349ca_isrForCrosstalkSources_855453_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3c483bc5-b8c8-427f-b564-b2da634349ca_isrForCrosstalkSources_855453_39.stderr parsl.dataflow.dflow INFO: Task 3760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7973393121f7472bfa295425c609321 with result from task 3760 parsl.dataflow.dflow DEBUG: Task 3760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3c871020-392a-4320-a18b-f12ef6d3c61d_isrForCrosstalkSources_855433_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3c871020-392a-4320-a18b-f12ef6d3c61d_isrForCrosstalkSources_855433_39.stderr parsl.dataflow.dflow INFO: Task 3561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a2c881f7ea3a974e19e2d173c794896 with result from task 3561 parsl.dataflow.dflow DEBUG: Task 3561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bf861680-9d92-4426-ade4-e0cb1dcf3923_isrForCrosstalkSources_855408_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/bf861680-9d92-4426-ade4-e0cb1dcf3923_isrForCrosstalkSources_855408_60.stderr parsl.dataflow.dflow INFO: Task 3347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 936d7af57c2e567ad8b8cdff270721fb with result from task 3347 parsl.dataflow.dflow DEBUG: Task 3347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/badd45d1-4e6a-412f-bd81-ce4838bdcb0e_isrForCrosstalkSources_855453_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/badd45d1-4e6a-412f-bd81-ce4838bdcb0e_isrForCrosstalkSources_855453_47.stderr parsl.dataflow.dflow INFO: Task 3679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27c06e11b71f747e356c2c223498bea1 with result from task 3679 parsl.dataflow.dflow DEBUG: Task 3679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d7ecc438-1bbb-4ff2-9869-3b96ed3cdda2_isrForCrosstalkSources_855430_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d7ecc438-1bbb-4ff2-9869-3b96ed3cdda2_isrForCrosstalkSources_855430_13.stderr parsl.dataflow.dflow INFO: Task 3477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec742e97abc506df9b48ce382982e95e with result from task 3477 parsl.dataflow.dflow DEBUG: Task 3477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7050be6d-e5aa-4d90-ab5f-505446a2eb79_isrForCrosstalkSources_855405_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7050be6d-e5aa-4d90-ab5f-505446a2eb79_isrForCrosstalkSources_855405_18.stderr parsl.dataflow.dflow INFO: Task 3319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fda0d79a961a169ce2012b0d602470ea with result from task 3319 parsl.dataflow.dflow DEBUG: Task 3319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/37c56ddf-5ed2-4a71-b4a0-89c67ea2006c_isrForCrosstalkSources_855393_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/37c56ddf-5ed2-4a71-b4a0-89c67ea2006c_isrForCrosstalkSources_855393_50.stderr parsl.dataflow.dflow INFO: Task 3509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e0480802544b04963c4138386bb7db7 with result from task 3509 parsl.dataflow.dflow DEBUG: Task 3509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e3ff23f6-1177-41b5-997d-1ab2a11ecafc_isrForCrosstalkSources_855380_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/e3ff23f6-1177-41b5-997d-1ab2a11ecafc_isrForCrosstalkSources_855380_21.stderr parsl.dataflow.dflow INFO: Task 3690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c9085075d49645ed10559e485287b4 with result from task 3690 parsl.dataflow.dflow DEBUG: Task 3690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/cac471ea-97b7-4dd4-9c0c-ce32958feeec_isrForCrosstalkSources_855459_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/cac471ea-97b7-4dd4-9c0c-ce32958feeec_isrForCrosstalkSources_855459_5.stderr parsl.dataflow.dflow INFO: Task 3249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a7584782297d98a6fc7427b1a1a99f7 with result from task 3249 parsl.dataflow.dflow DEBUG: Task 3249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3d81103c-db4f-4eda-aabf-555fc04bdbe7_isrForCrosstalkSources_855390_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/3d81103c-db4f-4eda-aabf-555fc04bdbe7_isrForCrosstalkSources_855390_57.stderr parsl.dataflow.dflow INFO: Task 3412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 314ce39031984b7e9165233999ac1e86 with result from task 3412 parsl.dataflow.dflow DEBUG: Task 3412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9cd63c91-ef38-4ced-97f7-c544c7b96ef3_isrForCrosstalkSources_855446_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/9cd63c91-ef38-4ced-97f7-c544c7b96ef3_isrForCrosstalkSources_855446_49.stderr parsl.dataflow.dflow INFO: Task 3666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 120d044cc95f10458842e9f17f70f514 with result from task 3666 parsl.dataflow.dflow DEBUG: Task 3666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9f013ba4-f0d0-4f4e-97fb-d7a58bb9627a_isrForCrosstalkSources_855389_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9f013ba4-f0d0-4f4e-97fb-d7a58bb9627a_isrForCrosstalkSources_855389_43.stderr parsl.dataflow.dflow INFO: Task 3625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f592a8fa2f1d0672e83f2a5882a4a66 with result from task 3625 parsl.dataflow.dflow DEBUG: Task 3625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4b7e71-b2ec-4ff6-82ae-ec6c0b1c5c7b_isrForCrosstalkSources_855393_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/6e4b7e71-b2ec-4ff6-82ae-ec6c0b1c5c7b_isrForCrosstalkSources_855393_32.stderr parsl.dataflow.dflow INFO: Task 3755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd97916757bb659bf39b1e6a335e98af with result from task 3755 parsl.dataflow.dflow DEBUG: Task 3755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/51afda2f-366a-49db-8572-c2459bd61021_isrForCrosstalkSources_855391_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/51afda2f-366a-49db-8572-c2459bd61021_isrForCrosstalkSources_855391_42.stderr parsl.dataflow.dflow INFO: Task 3324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c588fda30ac590d265d8c10f31c375f0 with result from task 3324 parsl.dataflow.dflow DEBUG: Task 3324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/19a9d313-ade1-4d42-aa60-306b254669f6_isrForCrosstalkSources_855443_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/19a9d313-ade1-4d42-aa60-306b254669f6_isrForCrosstalkSources_855443_41.stderr parsl.dataflow.dflow INFO: Task 3841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f24148e1f35ac59f680a9c033ca6573 with result from task 3841 parsl.dataflow.dflow DEBUG: Task 3841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b65b25af-fe5e-42d0-9be5-16c11c42cad8_isrForCrosstalkSources_855389_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/b65b25af-fe5e-42d0-9be5-16c11c42cad8_isrForCrosstalkSources_855389_37.stderr parsl.dataflow.dflow INFO: Task 3808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eb38a0e900c778c89026ac1486b0c3d with result from task 3808 parsl.dataflow.dflow DEBUG: Task 3808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9fdffaab-1b15-41f9-a627-ac4f59d8d086_isrForCrosstalkSources_855393_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9fdffaab-1b15-41f9-a627-ac4f59d8d086_isrForCrosstalkSources_855393_19.stderr parsl.dataflow.dflow INFO: Task 3641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e022231f3f1938cf2cf1dcfabf390107 with result from task 3641 parsl.dataflow.dflow DEBUG: Task 3641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8d7879fd-d442-4fa7-9f5b-7308057a5f54_isrForCrosstalkSources_855379_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8d7879fd-d442-4fa7-9f5b-7308057a5f54_isrForCrosstalkSources_855379_47.stderr parsl.dataflow.dflow INFO: Task 3519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8584768beb4971c9fefdf58db1992e3 with result from task 3519 parsl.dataflow.dflow DEBUG: Task 3519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5ad89aa5-14cf-49af-adea-6a98351c8baf_isrForCrosstalkSources_855400_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/5ad89aa5-14cf-49af-adea-6a98351c8baf_isrForCrosstalkSources_855400_58.stderr parsl.dataflow.dflow INFO: Task 3637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bdc9391774816dbd43c5ef197caedf2 with result from task 3637 parsl.dataflow.dflow DEBUG: Task 3637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c7fe6c11-d608-49dd-85e5-bbed650e8d79_isrForCrosstalkSources_855402_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/c7fe6c11-d608-49dd-85e5-bbed650e8d79_isrForCrosstalkSources_855402_55.stderr parsl.dataflow.dflow INFO: Task 3742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6dbd0053ae90306a613ee2577c9153a with result from task 3742 parsl.dataflow.dflow DEBUG: Task 3742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e63ee2f9-c6c0-4915-bdee-3042666095e3_isrForCrosstalkSources_855419_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e63ee2f9-c6c0-4915-bdee-3042666095e3_isrForCrosstalkSources_855419_29.stderr parsl.dataflow.dflow INFO: Task 3246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed27b71f815e98155f990881f544506e with result from task 3246 parsl.dataflow.dflow DEBUG: Task 3246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66f053dd-54fd-45af-9fab-58bb78f1a522_isrForCrosstalkSources_855390_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/66f053dd-54fd-45af-9fab-58bb78f1a522_isrForCrosstalkSources_855390_61.stderr parsl.dataflow.dflow INFO: Task 3438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9296e54d6a4ee15910d043df0eeb8552 with result from task 3438 parsl.dataflow.dflow DEBUG: Task 3438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7cfdac6c-1089-41a0-880a-78126cb2ca10_isrForCrosstalkSources_855414_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7cfdac6c-1089-41a0-880a-78126cb2ca10_isrForCrosstalkSources_855414_47.stderr parsl.dataflow.dflow INFO: Task 3379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e3b3af61e5e28bab3d7c22556017a0 with result from task 3379 parsl.dataflow.dflow DEBUG: Task 3379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4ebd1c0d-f607-4879-898d-4aea448111e0_isrForCrosstalkSources_855435_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/4ebd1c0d-f607-4879-898d-4aea448111e0_isrForCrosstalkSources_855435_55.stderr parsl.dataflow.dflow INFO: Task 3671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d48ced4520c538053d66001bf0e9b7f with result from task 3671 parsl.dataflow.dflow DEBUG: Task 3671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/91b2ce67-441a-487a-ba59-070dc9e29924_isrForCrosstalkSources_855393_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/91b2ce67-441a-487a-ba59-070dc9e29924_isrForCrosstalkSources_855393_48.stderr parsl.dataflow.dflow INFO: Task 3831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e18b3bda6978031c2ea449d8c2a07198 with result from task 3831 parsl.dataflow.dflow DEBUG: Task 3831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/49473dc9-3190-4d54-8ca2-c45bb54f7681_isrForCrosstalkSources_855410_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/49473dc9-3190-4d54-8ca2-c45bb54f7681_isrForCrosstalkSources_855410_13.stderr parsl.dataflow.dflow INFO: Task 3361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0e845a8908d08c29b65769a12fbd0b with result from task 3361 parsl.dataflow.dflow DEBUG: Task 3361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4070d7eb-1362-4cf7-a9c9-d2185a396d53_isrForCrosstalkSources_855392_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4070d7eb-1362-4cf7-a9c9-d2185a396d53_isrForCrosstalkSources_855392_52.stderr parsl.dataflow.dflow INFO: Task 3672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51e37653e93a857f098e11719a78e0cd with result from task 3672 parsl.dataflow.dflow DEBUG: Task 3672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b80c660e-2db0-4f9e-b6cb-55162e5ddfff_isrForCrosstalkSources_855454_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b80c660e-2db0-4f9e-b6cb-55162e5ddfff_isrForCrosstalkSources_855454_35.stderr parsl.dataflow.dflow INFO: Task 3715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7200cc01d49f7cd1e9cb65973e3bd6f with result from task 3715 parsl.dataflow.dflow DEBUG: Task 3715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/2a1f095c-8609-4104-b46b-2cd9e783560f_isrForCrosstalkSources_855427_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/2a1f095c-8609-4104-b46b-2cd9e783560f_isrForCrosstalkSources_855427_61.stderr parsl.dataflow.dflow INFO: Task 3794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1f35403f47ac6151390008fa7d8f36 with result from task 3794 parsl.dataflow.dflow DEBUG: Task 3794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/117c9559-6110-4eee-b836-7a575ca368b2_isrForCrosstalkSources_855440_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/117c9559-6110-4eee-b836-7a575ca368b2_isrForCrosstalkSources_855440_8.stderr parsl.dataflow.dflow INFO: Task 3752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d997f418ad0c23d9718ac088dbf630 with result from task 3752 parsl.dataflow.dflow DEBUG: Task 3752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/2ee62b5e-619e-4de4-8762-00fcfe02efcf_isrForCrosstalkSources_855445_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/2ee62b5e-619e-4de4-8762-00fcfe02efcf_isrForCrosstalkSources_855445_10.stderr parsl.dataflow.dflow INFO: Task 3636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfd5165d0b3e48f8fbb3d7167f54968c with result from task 3636 parsl.dataflow.dflow DEBUG: Task 3636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3b7f65dc-0e2a-46ab-af3e-04540511270f_isrForCrosstalkSources_855440_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/3b7f65dc-0e2a-46ab-af3e-04540511270f_isrForCrosstalkSources_855440_47.stderr parsl.dataflow.dflow INFO: Task 3682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e84fa4c3bcd6f372e9ac1cc9c21f56 with result from task 3682 parsl.dataflow.dflow DEBUG: Task 3682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/b88a13ad-47a9-42ba-a0f2-80751f47c78d_isrForCrosstalkSources_855409_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/b88a13ad-47a9-42ba-a0f2-80751f47c78d_isrForCrosstalkSources_855409_8.stderr parsl.dataflow.dflow INFO: Task 3694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8243bb10441d3ca98afe2208b064b90 with result from task 3694 parsl.dataflow.dflow DEBUG: Task 3694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2d320fa2-ff81-4d6b-9c41-09c7a3047275_isrForCrosstalkSources_855406_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2d320fa2-ff81-4d6b-9c41-09c7a3047275_isrForCrosstalkSources_855406_48.stderr parsl.dataflow.dflow INFO: Task 3859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0aa2bfd9488e49f217378b189a9d087 with result from task 3859 parsl.dataflow.dflow DEBUG: Task 3859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8aa1e1a8-5863-4618-8724-90cda2593111_isrForCrosstalkSources_855440_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/8aa1e1a8-5863-4618-8724-90cda2593111_isrForCrosstalkSources_855440_52.stderr parsl.dataflow.dflow INFO: Task 3710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6011e958e67cd6aaea12e58e2a7f190 with result from task 3710 parsl.dataflow.dflow DEBUG: Task 3710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/52b109a9-4641-49ac-8251-341ab42110c2_isrForCrosstalkSources_855429_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/52b109a9-4641-49ac-8251-341ab42110c2_isrForCrosstalkSources_855429_4.stderr parsl.dataflow.dflow INFO: Task 3645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd744de4b62302e02ebdf1cec156ada9 with result from task 3645 parsl.dataflow.dflow DEBUG: Task 3645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/18e5b8c2-5089-434f-8c7a-ea97415ec242_isrForCrosstalkSources_855459_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/18e5b8c2-5089-434f-8c7a-ea97415ec242_isrForCrosstalkSources_855459_15.stderr parsl.dataflow.dflow INFO: Task 3687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 193c4d7b8c97b211f6ddf521dc2879f2 with result from task 3687 parsl.dataflow.dflow DEBUG: Task 3687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1fc7e0e1-8ec7-4559-bb59-0dd3dd1432d8_isrForCrosstalkSources_855393_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/1fc7e0e1-8ec7-4559-bb59-0dd3dd1432d8_isrForCrosstalkSources_855393_39.stderr parsl.dataflow.dflow INFO: Task 3441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc83e33c85d9061e250454222d74a19e with result from task 3441 parsl.dataflow.dflow DEBUG: Task 3441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/88f573b8-081a-4ddd-83d8-2cc1b595e838_isrForCrosstalkSources_855424_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/88f573b8-081a-4ddd-83d8-2cc1b595e838_isrForCrosstalkSources_855424_35.stderr parsl.dataflow.dflow INFO: Task 3771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78896fbba3d1c813c976d246ed9d3bca with result from task 3771 parsl.dataflow.dflow DEBUG: Task 3771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/1e8a9629-ac39-47c9-b75e-ac54a0aca148_isrForCrosstalkSources_855404_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/1e8a9629-ac39-47c9-b75e-ac54a0aca148_isrForCrosstalkSources_855404_33.stderr parsl.dataflow.dflow INFO: Task 3481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfc547e63a37c0fa4fecbbe1794bc1e5 with result from task 3481 parsl.dataflow.dflow DEBUG: Task 3481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/6c4cad36-72a7-4f78-af82-962dcd1d5c55_isrForCrosstalkSources_855406_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/6c4cad36-72a7-4f78-af82-962dcd1d5c55_isrForCrosstalkSources_855406_53.stderr parsl.dataflow.dflow INFO: Task 3305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d1f28cf93dd0d041c99b552f2cf84a1 with result from task 3305 parsl.dataflow.dflow DEBUG: Task 3305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2f4618eb-7bdf-4c76-af6a-2ef0c568f00a_isrForCrosstalkSources_855398_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2f4618eb-7bdf-4c76-af6a-2ef0c568f00a_isrForCrosstalkSources_855398_42.stderr parsl.dataflow.dflow INFO: Task 3476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c0429bdd4d6acf5479b685bed3ac47d with result from task 3476 parsl.dataflow.dflow DEBUG: Task 3476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b9786dde-fdfd-44ee-808a-eca086ea3077_isrForCrosstalkSources_855439_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/b9786dde-fdfd-44ee-808a-eca086ea3077_isrForCrosstalkSources_855439_29.stderr parsl.dataflow.dflow INFO: Task 3668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7158d9a62b2c1e6d0a03c9b94dffbd0 with result from task 3668 parsl.dataflow.dflow DEBUG: Task 3668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/47bbc988-fa0e-4c5e-9c2b-c82c2b3efa15_isrForCrosstalkSources_855422_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/47bbc988-fa0e-4c5e-9c2b-c82c2b3efa15_isrForCrosstalkSources_855422_49.stderr parsl.dataflow.dflow INFO: Task 3510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 584d02c33198b605250c57133261bfb9 with result from task 3510 parsl.dataflow.dflow DEBUG: Task 3510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/f9f94987-86a9-43d5-9c44-b46cf1e5ba04_isrForCrosstalkSources_855408_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/f9f94987-86a9-43d5-9c44-b46cf1e5ba04_isrForCrosstalkSources_855408_58.stderr parsl.dataflow.dflow INFO: Task 3731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18828405c4f55ce14a1cc6d2df27515f with result from task 3731 parsl.dataflow.dflow DEBUG: Task 3731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f49f7637-c71d-4b0f-ab4c-1e82f0af05d5_isrForCrosstalkSources_855459_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f49f7637-c71d-4b0f-ab4c-1e82f0af05d5_isrForCrosstalkSources_855459_46.stderr parsl.dataflow.dflow INFO: Task 3669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b82e7c5473eca909a6c0ef1f2a01e60 with result from task 3669 parsl.dataflow.dflow DEBUG: Task 3669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/db281a04-69c2-4d5c-b52e-18933cd1f200_isrForCrosstalkSources_855439_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/db281a04-69c2-4d5c-b52e-18933cd1f200_isrForCrosstalkSources_855439_44.stderr parsl.dataflow.dflow INFO: Task 3683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f5c3a656c4c3c803474316d178613e with result from task 3683 parsl.dataflow.dflow DEBUG: Task 3683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/fcac4def-d984-4b5a-8b8e-dd23aa33027f_isrForCrosstalkSources_855427_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/fcac4def-d984-4b5a-8b8e-dd23aa33027f_isrForCrosstalkSources_855427_21.stderr parsl.dataflow.dflow INFO: Task 3643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04822c43e6fed03775da5a7dd30e86ab with result from task 3643 parsl.dataflow.dflow DEBUG: Task 3643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b8dfe603-054a-4ed5-84d2-7fb0f463e5e9_isrForCrosstalkSources_855422_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/b8dfe603-054a-4ed5-84d2-7fb0f463e5e9_isrForCrosstalkSources_855422_56.stderr parsl.dataflow.dflow INFO: Task 3491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc9410951f80ff43b38fbfbd7d4aa7a with result from task 3491 parsl.dataflow.dflow DEBUG: Task 3491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/743af19f-3743-4b75-afd5-164e3c9afcbd_isrForCrosstalkSources_855401_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/743af19f-3743-4b75-afd5-164e3c9afcbd_isrForCrosstalkSources_855401_41.stderr parsl.dataflow.dflow INFO: Task 3507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbbd6ccabef1ecd4134739345070d27a with result from task 3507 parsl.dataflow.dflow DEBUG: Task 3507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/91cebbaf-5877-4c1a-9bd9-0858e995f7dd_isrForCrosstalkSources_855419_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/91cebbaf-5877-4c1a-9bd9-0858e995f7dd_isrForCrosstalkSources_855419_52.stderr parsl.dataflow.dflow INFO: Task 3677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5e312cfcc745e64b1f3bdbb2accbbc6 with result from task 3677 parsl.dataflow.dflow DEBUG: Task 3677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/affe1d0e-7d37-48fd-8ba2-df395102153e_isrForCrosstalkSources_855408_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/affe1d0e-7d37-48fd-8ba2-df395102153e_isrForCrosstalkSources_855408_23.stderr parsl.dataflow.dflow INFO: Task 3814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24987f2215b2b02ff8221d83f141c576 with result from task 3814 parsl.dataflow.dflow DEBUG: Task 3814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/36dbcbc0-5c49-417e-b8d7-30976f3dd49f_isrForCrosstalkSources_855433_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/36dbcbc0-5c49-417e-b8d7-30976f3dd49f_isrForCrosstalkSources_855433_14.stderr parsl.dataflow.dflow INFO: Task 3632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a45ad3e53784184add40a48d564be67 with result from task 3632 parsl.dataflow.dflow DEBUG: Task 3632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e508acd9-9f1b-4a00-8699-b9e218300c28_isrForCrosstalkSources_855420_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e508acd9-9f1b-4a00-8699-b9e218300c28_isrForCrosstalkSources_855420_55.stderr parsl.dataflow.dflow INFO: Task 3603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e277ecb6a4ea8fc98f0586b6bd2cd568 with result from task 3603 parsl.dataflow.dflow DEBUG: Task 3603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/c54ab6a1-cc2f-483c-8b3c-8f8b6273198c_isrForCrosstalkSources_855386_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/c54ab6a1-cc2f-483c-8b3c-8f8b6273198c_isrForCrosstalkSources_855386_17.stderr parsl.dataflow.dflow INFO: Task 3708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c2315cda6074b543c04f900d256bb97 with result from task 3708 parsl.dataflow.dflow DEBUG: Task 3708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c78972bb-7cc9-477d-894b-1575b4a4220e_isrForCrosstalkSources_855451_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/c78972bb-7cc9-477d-894b-1575b4a4220e_isrForCrosstalkSources_855451_1.stderr parsl.dataflow.dflow INFO: Task 3593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aabdf3ce770d6b721eb641b86262216 with result from task 3593 parsl.dataflow.dflow DEBUG: Task 3593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4c7b5dea-8ad2-4be7-924a-56c0545dfbea_isrForCrosstalkSources_855443_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/4c7b5dea-8ad2-4be7-924a-56c0545dfbea_isrForCrosstalkSources_855443_10.stderr parsl.dataflow.dflow INFO: Task 3639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dbeae94f86e5b84f931c00f0fa93307 with result from task 3639 parsl.dataflow.dflow DEBUG: Task 3639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/624008d9-0a44-4c38-ac20-b12b4ff6fac2_isrForCrosstalkSources_855380_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/624008d9-0a44-4c38-ac20-b12b4ff6fac2_isrForCrosstalkSources_855380_61.stderr parsl.dataflow.dflow INFO: Task 3809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d424dc49a72931c0a53a06cd71d76cf with result from task 3809 parsl.dataflow.dflow DEBUG: Task 3809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b29adea0-8dbe-4282-9806-486e2e364c3e_isrForCrosstalkSources_855456_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b29adea0-8dbe-4282-9806-486e2e364c3e_isrForCrosstalkSources_855456_57.stderr parsl.dataflow.dflow INFO: Task 3693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138e8e27fa0e4063098e46295c74e355 with result from task 3693 parsl.dataflow.dflow DEBUG: Task 3693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f462d030-b462-4440-b154-c4c02d0e3b9b_isrForCrosstalkSources_855434_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/f462d030-b462-4440-b154-c4c02d0e3b9b_isrForCrosstalkSources_855434_17.stderr parsl.dataflow.dflow INFO: Task 3652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8e7bee939676d17bb67addfb4d8b59 with result from task 3652 parsl.dataflow.dflow DEBUG: Task 3652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a2e3da9f-ca02-49d1-ae0b-dc94973b8e29_isrForCrosstalkSources_855383_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/a2e3da9f-ca02-49d1-ae0b-dc94973b8e29_isrForCrosstalkSources_855383_29.stderr parsl.dataflow.dflow INFO: Task 3587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81332768d8426de860a928a31b651077 with result from task 3587 parsl.dataflow.dflow DEBUG: Task 3587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/225067e1-ab8d-42e9-801c-31458bc2c3c7_isrForCrosstalkSources_855381_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/225067e1-ab8d-42e9-801c-31458bc2c3c7_isrForCrosstalkSources_855381_1.stderr parsl.dataflow.dflow INFO: Task 3785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57f8c31c38077216bef3d543aab3a6f1 with result from task 3785 parsl.dataflow.dflow DEBUG: Task 3785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/07f0fa20-5457-485c-816e-ad4f31061d11_isrForCrosstalkSources_855457_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/07f0fa20-5457-485c-816e-ad4f31061d11_isrForCrosstalkSources_855457_57.stderr parsl.dataflow.dflow INFO: Task 3723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89aa5ddbf08e3cb6643e367721296516 with result from task 3723 parsl.dataflow.dflow DEBUG: Task 3723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a4b72e4a-c459-4ac1-abbe-8ef3b2b28836_isrForCrosstalkSources_855393_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a4b72e4a-c459-4ac1-abbe-8ef3b2b28836_isrForCrosstalkSources_855393_52.stderr parsl.dataflow.dflow INFO: Task 3697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21a7995c5e6ae474cfda4382bac0b87f with result from task 3697 parsl.dataflow.dflow DEBUG: Task 3697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d365306a-dd7e-4585-ba1f-23b3656d3180_isrForCrosstalkSources_855446_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/d365306a-dd7e-4585-ba1f-23b3656d3180_isrForCrosstalkSources_855446_7.stderr parsl.dataflow.dflow INFO: Task 3667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c84cad47cdd572036ae0f094bfc6cbf with result from task 3667 parsl.dataflow.dflow DEBUG: Task 3667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/03bdc9fe-3513-4f76-9a59-2b2081246165_isrForCrosstalkSources_855437_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/03bdc9fe-3513-4f76-9a59-2b2081246165_isrForCrosstalkSources_855437_23.stderr parsl.dataflow.dflow INFO: Task 3627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 912792969aefb417da22a7c4b32697dc with result from task 3627 parsl.dataflow.dflow DEBUG: Task 3627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1440ae3e-1b79-4a1d-83ee-3c5398968b62_isrForCrosstalkSources_855442_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1440ae3e-1b79-4a1d-83ee-3c5398968b62_isrForCrosstalkSources_855442_7.stderr parsl.dataflow.dflow INFO: Task 3614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc699752903dfb01254500535aa15afa with result from task 3614 parsl.dataflow.dflow DEBUG: Task 3614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/4127ef1f-9cab-4a13-a5b5-4c17e1138b4c_isrForCrosstalkSources_855422_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/4127ef1f-9cab-4a13-a5b5-4c17e1138b4c_isrForCrosstalkSources_855422_11.stderr parsl.dataflow.dflow INFO: Task 3695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 560ee2ddb802a537c20b4627dd9658f7 with result from task 3695 parsl.dataflow.dflow DEBUG: Task 3695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a09acec7-1983-4e3d-9ed9-275e072b6bcb_isrForCrosstalkSources_855451_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a09acec7-1983-4e3d-9ed9-275e072b6bcb_isrForCrosstalkSources_855451_35.stderr parsl.dataflow.dflow INFO: Task 3681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d8fb85c6be12d05478137cb430a91fb with result from task 3681 parsl.dataflow.dflow DEBUG: Task 3681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0fc3e8b7-67cb-4b67-8ac0-1af7a4198248_isrForCrosstalkSources_855384_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/0fc3e8b7-67cb-4b67-8ac0-1af7a4198248_isrForCrosstalkSources_855384_47.stderr parsl.dataflow.dflow INFO: Task 3727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 332a791bd3e3747227b5c97e75dfbe14 with result from task 3727 parsl.dataflow.dflow DEBUG: Task 3727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a1b8fdf7-fd6a-4664-8e2a-4d942857ab34_isrForCrosstalkSources_855421_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a1b8fdf7-fd6a-4664-8e2a-4d942857ab34_isrForCrosstalkSources_855421_56.stderr parsl.dataflow.dflow INFO: Task 3646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b9af8d4d706879d7f3aa0df82303f4f with result from task 3646 parsl.dataflow.dflow DEBUG: Task 3646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/8bff1e85-8a3e-4843-9f6f-8f7a436d1a84_isrForCrosstalkSources_855455_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/8bff1e85-8a3e-4843-9f6f-8f7a436d1a84_isrForCrosstalkSources_855455_17.stderr parsl.dataflow.dflow INFO: Task 3618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7bbdb2980b69515fa064a1e837d095a with result from task 3618 parsl.dataflow.dflow DEBUG: Task 3618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9513d3c0-7f29-40d6-9809-71f4cd085536_isrForCrosstalkSources_855398_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/9513d3c0-7f29-40d6-9809-71f4cd085536_isrForCrosstalkSources_855398_15.stderr parsl.dataflow.dflow INFO: Task 3664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daed0e3236e7bdf790db27ac7ab19e47 with result from task 3664 parsl.dataflow.dflow DEBUG: Task 3664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fe2f294e-7455-421d-a06e-422cb47ecbea_isrForCrosstalkSources_855420_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fe2f294e-7455-421d-a06e-422cb47ecbea_isrForCrosstalkSources_855420_52.stderr parsl.dataflow.dflow INFO: Task 3607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 236f93b602b290eceb42682490d8b586 with result from task 3607 parsl.dataflow.dflow DEBUG: Task 3607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/48447d67-adbd-4710-bb27-d99c2a6b232b_isrForCrosstalkSources_855398_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/48447d67-adbd-4710-bb27-d99c2a6b232b_isrForCrosstalkSources_855398_36.stderr parsl.dataflow.dflow INFO: Task 3620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32c9a19f8aa5e21f1b91c06f85cf287d with result from task 3620 parsl.dataflow.dflow DEBUG: Task 3620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/361861f7-6723-46fd-b49b-225de6392422_isrForCrosstalkSources_855388_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/361861f7-6723-46fd-b49b-225de6392422_isrForCrosstalkSources_855388_6.stderr parsl.dataflow.dflow INFO: Task 3514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe9360d589d08c8380813ed9a3af113 with result from task 3514 parsl.dataflow.dflow DEBUG: Task 3514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6e5ff343-1f86-41a3-bcd0-a77f410dcbca_isrForCrosstalkSources_855457_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/6e5ff343-1f86-41a3-bcd0-a77f410dcbca_isrForCrosstalkSources_855457_29.stderr parsl.dataflow.dflow INFO: Task 3754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74752565bf728611cb185d23f3c8963e with result from task 3754 parsl.dataflow.dflow DEBUG: Task 3754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2ba87b9a-ce05-4e39-ab1e-5ae37c126d2d_isrForCrosstalkSources_855398_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/2ba87b9a-ce05-4e39-ab1e-5ae37c126d2d_isrForCrosstalkSources_855398_48.stderr parsl.dataflow.dflow INFO: Task 3674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa31053d8850f9df8676a0e75aebe34f with result from task 3674 parsl.dataflow.dflow DEBUG: Task 3674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2d25e4bb-2b3b-46dc-80f7-bc1f7b2aa659_isrForCrosstalkSources_855395_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2d25e4bb-2b3b-46dc-80f7-bc1f7b2aa659_isrForCrosstalkSources_855395_30.stderr parsl.dataflow.dflow INFO: Task 3612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06f8e9481093c5bb6ffd92dc325a44d4 with result from task 3612 parsl.dataflow.dflow DEBUG: Task 3612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/756c58c9-8fe8-463b-83f2-4f48a14029fc_isrForCrosstalkSources_855456_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/756c58c9-8fe8-463b-83f2-4f48a14029fc_isrForCrosstalkSources_855456_43.stderr parsl.dataflow.dflow INFO: Task 3385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce5193c4126cf218a8783f601090c3d with result from task 3385 parsl.dataflow.dflow DEBUG: Task 3385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23bd345c-6299-4cea-998a-f3726dd9ee87_isrForCrosstalkSources_855396_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23bd345c-6299-4cea-998a-f3726dd9ee87_isrForCrosstalkSources_855396_35.stderr parsl.dataflow.dflow INFO: Task 3700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1457d983bd03030b0745d7be635ad95b with result from task 3700 parsl.dataflow.dflow DEBUG: Task 3700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c6a30af4-2100-4441-b261-5947c242bbb9_isrForCrosstalkSources_855431_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c6a30af4-2100-4441-b261-5947c242bbb9_isrForCrosstalkSources_855431_9.stderr parsl.dataflow.dflow INFO: Task 3735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6badcdf0ab28edbf4e519bd3c905aa5a with result from task 3735 parsl.dataflow.dflow DEBUG: Task 3735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b8814759-4238-45ba-b39d-6106eacd30ac_isrForCrosstalkSources_855454_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/b8814759-4238-45ba-b39d-6106eacd30ac_isrForCrosstalkSources_855454_36.stderr parsl.dataflow.dflow INFO: Task 3827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a8cc7e6dff7200707f541b6d2e4bfd4 with result from task 3827 parsl.dataflow.dflow DEBUG: Task 3827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c17bffe7-b847-4100-b555-7d75941258de_isrForCrosstalkSources_855414_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/c17bffe7-b847-4100-b555-7d75941258de_isrForCrosstalkSources_855414_58.stderr parsl.dataflow.dflow INFO: Task 3638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 615a76989bcc72ea2c39c8897e8a48c6 with result from task 3638 parsl.dataflow.dflow DEBUG: Task 3638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c97f978a-0409-4831-a8bb-9bf3e6601adb_isrForCrosstalkSources_855452_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c97f978a-0409-4831-a8bb-9bf3e6601adb_isrForCrosstalkSources_855452_31.stderr parsl.dataflow.dflow INFO: Task 3388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ca263493ba6c9ec3bb4a6eea01dd5e with result from task 3388 parsl.dataflow.dflow DEBUG: Task 3388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/8fd63271-097e-40b9-8ec3-bac9321bd005_isrForCrosstalkSources_855383_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/8fd63271-097e-40b9-8ec3-bac9321bd005_isrForCrosstalkSources_855383_60.stderr parsl.dataflow.dflow INFO: Task 3633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdd2ccbb6d802e1ff95e5a73ed155ff1 with result from task 3633 parsl.dataflow.dflow DEBUG: Task 3633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/84d3b206-b3fe-4a27-84d7-011ed74d0b07_isrForCrosstalkSources_855395_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/84d3b206-b3fe-4a27-84d7-011ed74d0b07_isrForCrosstalkSources_855395_48.stderr parsl.dataflow.dflow INFO: Task 3571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6236400eb79acfbcbd37d6b1675d2816 with result from task 3571 parsl.dataflow.dflow DEBUG: Task 3571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/486fcda9-77aa-490a-8097-91bcb88e125b_isrForCrosstalkSources_855394_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/486fcda9-77aa-490a-8097-91bcb88e125b_isrForCrosstalkSources_855394_59.stderr parsl.dataflow.dflow INFO: Task 3766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d28400259f11418164c362c2b40363fa with result from task 3766 parsl.dataflow.dflow DEBUG: Task 3766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b63987e-e9e7-4863-8239-b8c175189c2d_isrForCrosstalkSources_855449_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b63987e-e9e7-4863-8239-b8c175189c2d_isrForCrosstalkSources_855449_32.stderr parsl.dataflow.dflow INFO: Task 3740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0407c41cb4f80d87450563246895d27 with result from task 3740 parsl.dataflow.dflow DEBUG: Task 3740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/916fe9ac-3cf7-456d-bdbc-52602123b326_isrForCrosstalkSources_855417_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/916fe9ac-3cf7-456d-bdbc-52602123b326_isrForCrosstalkSources_855417_24.stderr parsl.dataflow.dflow INFO: Task 3718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6799ef143b7abe1d3d30e1a4dc24b7e with result from task 3718 parsl.dataflow.dflow DEBUG: Task 3718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3bb48352-9f4a-44c6-a25b-a9312ca93808_isrForCrosstalkSources_855424_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/3bb48352-9f4a-44c6-a25b-a9312ca93808_isrForCrosstalkSources_855424_21.stderr parsl.dataflow.dflow INFO: Task 3770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87be5e0de037315cd7911703b3e84bc7 with result from task 3770 parsl.dataflow.dflow DEBUG: Task 3770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9a20a477-3f55-4090-891f-5d6ba793daf1_isrForCrosstalkSources_855456_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/9a20a477-3f55-4090-891f-5d6ba793daf1_isrForCrosstalkSources_855456_48.stderr parsl.dataflow.dflow INFO: Task 3686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0fc28f63d2e5efd8a73c2c87b18d181 with result from task 3686 parsl.dataflow.dflow DEBUG: Task 3686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/67291f22-13fc-42f8-afdf-1e423eb845cd_isrForCrosstalkSources_855456_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/67291f22-13fc-42f8-afdf-1e423eb845cd_isrForCrosstalkSources_855456_39.stderr parsl.dataflow.dflow INFO: Task 3640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7449d6744c00678a5789a10a31af94f with result from task 3640 parsl.dataflow.dflow DEBUG: Task 3640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6a1a403-6f43-40f1-9c7e-cbedb4afd4d5_isrForCrosstalkSources_855380_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6a1a403-6f43-40f1-9c7e-cbedb4afd4d5_isrForCrosstalkSources_855380_25.stderr parsl.dataflow.dflow INFO: Task 3844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed3aef0f498472e76fa409d6b304ed95 with result from task 3844 parsl.dataflow.dflow DEBUG: Task 3844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f9e3e46d-1a7a-44c2-8c9d-e4cb91153b51_isrForCrosstalkSources_855401_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f9e3e46d-1a7a-44c2-8c9d-e4cb91153b51_isrForCrosstalkSources_855401_17.stderr parsl.dataflow.dflow INFO: Task 3743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7ca162e0d3c9093a305a4c38809a4a7 with result from task 3743 parsl.dataflow.dflow DEBUG: Task 3743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/1c7e0d12-f2d0-4801-9192-1cecba439eba_isrForCrosstalkSources_855451_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/1c7e0d12-f2d0-4801-9192-1cecba439eba_isrForCrosstalkSources_855451_22.stderr parsl.dataflow.dflow INFO: Task 3642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 938bd85f2f8970092d18b85b112e4362 with result from task 3642 parsl.dataflow.dflow DEBUG: Task 3642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7f7bf617-f0c3-4cf2-9d99-d6716fd37507_isrForCrosstalkSources_855406_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/7f7bf617-f0c3-4cf2-9d99-d6716fd37507_isrForCrosstalkSources_855406_27.stderr parsl.dataflow.dflow INFO: Task 3812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ce27384ae9c318ae0aa8530b6c7cf79 with result from task 3812 parsl.dataflow.dflow DEBUG: Task 3812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8464ad49-f405-4e87-8f54-42f2d8ed6c39_isrForCrosstalkSources_855459_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/8464ad49-f405-4e87-8f54-42f2d8ed6c39_isrForCrosstalkSources_855459_33.stderr parsl.dataflow.dflow INFO: Task 3611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 208a52d5f0b42ddbb3e80a7d2939909c with result from task 3611 parsl.dataflow.dflow DEBUG: Task 3611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1c0d908-a620-4f0d-9025-150302b8e34e_isrForCrosstalkSources_855412_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c1c0d908-a620-4f0d-9025-150302b8e34e_isrForCrosstalkSources_855412_25.stderr parsl.dataflow.dflow INFO: Task 3828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f4be8c8d698ae1a489ed28ebc1ee62 with result from task 3828 parsl.dataflow.dflow DEBUG: Task 3828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/dbbf3121-53e4-411f-b4b9-4378f0ddd7e6_isrForCrosstalkSources_855380_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/dbbf3121-53e4-411f-b4b9-4378f0ddd7e6_isrForCrosstalkSources_855380_45.stderr parsl.dataflow.dflow INFO: Task 3737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b59e0222590f0ffcdc0df098e5c205da with result from task 3737 parsl.dataflow.dflow DEBUG: Task 3737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/95e18f2a-ba45-4f40-b320-d392c1c86488_isrForCrosstalkSources_855413_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/95e18f2a-ba45-4f40-b320-d392c1c86488_isrForCrosstalkSources_855413_19.stderr parsl.dataflow.dflow INFO: Task 3807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5609c2eda301da29abde8ba142e889ad with result from task 3807 parsl.dataflow.dflow DEBUG: Task 3807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a9f8169d-9e52-40db-95b6-007b700ed966_isrForCrosstalkSources_855401_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/a9f8169d-9e52-40db-95b6-007b700ed966_isrForCrosstalkSources_855401_46.stderr parsl.dataflow.dflow INFO: Task 3626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1e4567d430deb2314208ccad8cf786b with result from task 3626 parsl.dataflow.dflow DEBUG: Task 3626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a960ec8a-05a6-48d0-8c7d-3c80806f95a2_isrForCrosstalkSources_855422_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a960ec8a-05a6-48d0-8c7d-3c80806f95a2_isrForCrosstalkSources_855422_25.stderr parsl.dataflow.dflow INFO: Task 3868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da9dec055ea66fadf597a9d504824f4d with result from task 3868 parsl.dataflow.dflow DEBUG: Task 3868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d9ada8ad-ce6f-4197-8127-ce8e44f5d56d_isrForCrosstalkSources_855444_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/d9ada8ad-ce6f-4197-8127-ce8e44f5d56d_isrForCrosstalkSources_855444_8.stderr parsl.dataflow.dflow INFO: Task 3871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e8979d7e6eb357db3d0c78f7b48bd02 with result from task 3871 parsl.dataflow.dflow DEBUG: Task 3871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b41767e5-7cc7-414a-b5a1-6c92a14557f3_isrForCrosstalkSources_855457_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/b41767e5-7cc7-414a-b5a1-6c92a14557f3_isrForCrosstalkSources_855457_51.stderr parsl.dataflow.dflow INFO: Task 3875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23fdf50a3ef2c491479051a7d5764848 with result from task 3875 parsl.dataflow.dflow DEBUG: Task 3875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0c1a7bd8-4354-44d7-b387-ed1c53bfbbbd_isrForCrosstalkSources_855435_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0c1a7bd8-4354-44d7-b387-ed1c53bfbbbd_isrForCrosstalkSources_855435_46.stderr parsl.dataflow.dflow INFO: Task 3767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 826c03120526513386571500a302daf3 with result from task 3767 parsl.dataflow.dflow DEBUG: Task 3767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/0c5a1922-d189-4299-a447-7c12ea6dcaa3_isrForCrosstalkSources_855410_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/0c5a1922-d189-4299-a447-7c12ea6dcaa3_isrForCrosstalkSources_855410_2.stderr parsl.dataflow.dflow INFO: Task 3706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ef6eaf421a3566b789f0fc2fb68885 with result from task 3706 parsl.dataflow.dflow DEBUG: Task 3706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9592d78c-c0ae-4bf9-bb81-20db09af7e2c_isrForCrosstalkSources_855441_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9592d78c-c0ae-4bf9-bb81-20db09af7e2c_isrForCrosstalkSources_855441_32.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6918a7c52f38cbc727e8ee07d74296e4 with result from task 3608 parsl.dataflow.dflow DEBUG: Task 3608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/803aacb8-5f3f-4dd4-bc42-f043ca576d09_isrForCrosstalkSources_855446_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/803aacb8-5f3f-4dd4-bc42-f043ca576d09_isrForCrosstalkSources_855446_32.stderr parsl.dataflow.dflow INFO: Task 3659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f438786142b857467189c2e8bb34ae4b with result from task 3659 parsl.dataflow.dflow DEBUG: Task 3659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/76e6d36b-3571-4a31-a572-99a87ea60b51_isrForCrosstalkSources_855440_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/76e6d36b-3571-4a31-a572-99a87ea60b51_isrForCrosstalkSources_855440_33.stderr parsl.dataflow.dflow INFO: Task 3655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2bd7a90549d7f08fde66ee71a72a11b with result from task 3655 parsl.dataflow.dflow DEBUG: Task 3655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/31d3ad1a-92ce-4006-bd57-3b1fb3f9dcaf_isrForCrosstalkSources_855437_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/31d3ad1a-92ce-4006-bd57-3b1fb3f9dcaf_isrForCrosstalkSources_855437_4.stderr parsl.dataflow.dflow INFO: Task 3861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1901aeea47ad7a14311ccf616734aa19 with result from task 3861 parsl.dataflow.dflow DEBUG: Task 3861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/195650d4-6bc3-4692-b0a9-36b2105d2d6e_isrForCrosstalkSources_855437_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/195650d4-6bc3-4692-b0a9-36b2105d2d6e_isrForCrosstalkSources_855437_8.stderr parsl.dataflow.dflow INFO: Task 3791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26d429ea5f4a943146667e27ad0f1a48 with result from task 3791 parsl.dataflow.dflow DEBUG: Task 3791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/990c8d36-02f4-4de2-b6fa-04bdd7d8f0d2_isrForCrosstalkSources_855407_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/990c8d36-02f4-4de2-b6fa-04bdd7d8f0d2_isrForCrosstalkSources_855407_11.stderr 2024-03-09 14:55:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964294 proc_lsst.multi INFO: [multi] job_id 16964294 2024-03-09 14:55:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 204 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 204 2024-03-09 14:55:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964294 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964294 parsl.executors.status_handling DEBUG: Launched block 259 on executor multi with job ID 16964294 parsl.executors.status_handling INFO: Allocated block ID 260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-260 2024-03-09 14:55:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 2024-03-09 14:55:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 204 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 204 2024-03-09 14:55:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8876db91f8717c86863d8e59f7af65f1 with result from task 3876 parsl.dataflow.dflow DEBUG: Task 3876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cf74abc9-5de9-4ad1-8ce6-37c318440bc6_isrForCrosstalkSources_855435_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/cf74abc9-5de9-4ad1-8ce6-37c318440bc6_isrForCrosstalkSources_855435_32.stderr parsl.dataflow.dflow INFO: Task 3591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc58648d5b1f150fa1a32a9857f2a886 with result from task 3591 parsl.dataflow.dflow DEBUG: Task 3591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/75d74f6a-e9ff-41db-9647-5dcdee1c4c36_isrForCrosstalkSources_855411_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/75d74f6a-e9ff-41db-9647-5dcdee1c4c36_isrForCrosstalkSources_855411_22.stderr parsl.dataflow.dflow INFO: Task 3634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a451bfd3dd56b443a95212d53e952384 with result from task 3634 parsl.dataflow.dflow DEBUG: Task 3634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d4fd2b5-de8e-4a34-83ef-536ef5bd0aab_isrForCrosstalkSources_855444_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d4fd2b5-de8e-4a34-83ef-536ef5bd0aab_isrForCrosstalkSources_855444_34.stderr parsl.dataflow.dflow INFO: Task 3908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19d2168243c3bbbf8b084da84f2d3ea3 with result from task 3908 parsl.dataflow.dflow DEBUG: Task 3908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4bc6cc34-d0d3-498d-988b-ae0dab3716dd_isrForCrosstalkSources_855382_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/4bc6cc34-d0d3-498d-988b-ae0dab3716dd_isrForCrosstalkSources_855382_20.stderr parsl.dataflow.dflow INFO: Task 3819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7605fd66cc96c5d42b83fe61c714155d with result from task 3819 parsl.dataflow.dflow DEBUG: Task 3819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ff2ecbd1-79b5-4919-9f0e-cef076aa97b5_isrForCrosstalkSources_855440_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ff2ecbd1-79b5-4919-9f0e-cef076aa97b5_isrForCrosstalkSources_855440_15.stderr parsl.dataflow.dflow INFO: Task 3776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab3cd72556a86c9d91a188364b38511c with result from task 3776 parsl.dataflow.dflow DEBUG: Task 3776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/111f52e3-e56b-40e6-87bc-354b29b3d88d_isrForCrosstalkSources_855440_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/111f52e3-e56b-40e6-87bc-354b29b3d88d_isrForCrosstalkSources_855440_16.stderr parsl.dataflow.dflow INFO: Task 3536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f0fbbab9c657d6daf310f737e254b6 with result from task 3536 parsl.dataflow.dflow DEBUG: Task 3536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7dd824e9-7a35-42c7-a512-c948c8d5c366_isrForCrosstalkSources_855424_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/7dd824e9-7a35-42c7-a512-c948c8d5c366_isrForCrosstalkSources_855424_6.stderr parsl.dataflow.dflow INFO: Task 3852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3277591ba1b5ca5b65452526bbf8dfd2 with result from task 3852 parsl.dataflow.dflow DEBUG: Task 3852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/8d610b44-6a6f-4dc2-ae76-a419a8eef38d_isrForCrosstalkSources_855413_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/8d610b44-6a6f-4dc2-ae76-a419a8eef38d_isrForCrosstalkSources_855413_15.stderr 2024-03-09 14:55:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964295 proc_lsst.multi INFO: [multi] job_id 16964295 2024-03-09 14:55:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 205 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 205 2024-03-09 14:55:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964295 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964295 parsl.executors.status_handling DEBUG: Launched block 260 on executor multi with job ID 16964295 parsl.executors.status_handling INFO: Allocated block ID 261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-261 2024-03-09 14:55:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 205 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 205 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f47293dce13a8733d2ee2f42c6a4cd7 with result from task 3377 parsl.dataflow.dflow DEBUG: Task 3377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4dbddeb2-d65b-4527-98e6-d6228f90ac64_isrForCrosstalkSources_855389_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4dbddeb2-d65b-4527-98e6-d6228f90ac64_isrForCrosstalkSources_855389_5.stderr parsl.dataflow.dflow INFO: Task 3701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328ad4b63958eb2023054a57a7a9a8c9 with result from task 3701 parsl.dataflow.dflow DEBUG: Task 3701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24d447f6-88fd-41c9-b28e-d07a26db4fb0_isrForCrosstalkSources_855390_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/24d447f6-88fd-41c9-b28e-d07a26db4fb0_isrForCrosstalkSources_855390_40.stderr parsl.dataflow.dflow INFO: Task 3644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0c660db504a12acd9b19aca64207e77 with result from task 3644 parsl.dataflow.dflow DEBUG: Task 3644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0e4ac518-dbd7-48a7-8729-8601cc121784_isrForCrosstalkSources_855452_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0e4ac518-dbd7-48a7-8729-8601cc121784_isrForCrosstalkSources_855452_20.stderr parsl.dataflow.dflow INFO: Task 3759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26e52240903da6b31dfae881b54933b with result from task 3759 parsl.dataflow.dflow DEBUG: Task 3759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f658c58d-e18e-451c-9859-20543777b603_isrForCrosstalkSources_855431_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f658c58d-e18e-451c-9859-20543777b603_isrForCrosstalkSources_855431_39.stderr parsl.dataflow.dflow INFO: Task 3747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31295d33864bae17e716ca05c996ddf7 with result from task 3747 parsl.dataflow.dflow DEBUG: Task 3747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a569e825-fa61-4f88-85fb-7316bb89e279_isrForCrosstalkSources_855450_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a569e825-fa61-4f88-85fb-7316bb89e279_isrForCrosstalkSources_855450_7.stderr parsl.dataflow.dflow INFO: Task 3899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e57872d292d59d87bd3089cb5e85180f with result from task 3899 parsl.dataflow.dflow DEBUG: Task 3899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1d967d13-6b18-48ee-ad9d-7a2be81a9b9c_isrForCrosstalkSources_855382_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/1d967d13-6b18-48ee-ad9d-7a2be81a9b9c_isrForCrosstalkSources_855382_8.stderr 2024-03-09 14:55:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964296 proc_lsst.multi INFO: [multi] job_id 16964296 2024-03-09 14:55:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 206 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 206 2024-03-09 14:55:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964296 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964296 parsl.executors.status_handling DEBUG: Launched block 261 on executor multi with job ID 16964296 parsl.executors.status_handling INFO: Allocated block ID 262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-262 2024-03-09 14:55:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 206 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 206 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 630e9ad84a78a83b4e8c5f2673a2e179 with result from task 3850 parsl.dataflow.dflow DEBUG: Task 3850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1550e032-8527-45b6-b5a2-19c905163406_isrForCrosstalkSources_855392_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/1550e032-8527-45b6-b5a2-19c905163406_isrForCrosstalkSources_855392_34.stderr parsl.dataflow.dflow INFO: Task 3835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 167ceaca2180f7da608b763758342034 with result from task 3835 parsl.dataflow.dflow DEBUG: Task 3835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fddd5fe9-ba87-4aed-81a7-735e201909a3_isrForCrosstalkSources_855430_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/fddd5fe9-ba87-4aed-81a7-735e201909a3_isrForCrosstalkSources_855430_56.stderr parsl.dataflow.dflow INFO: Task 3653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea1b0df1e4de73b84c7f0df62049bd2 with result from task 3653 parsl.dataflow.dflow DEBUG: Task 3653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/287b80a6-0a20-43c1-9574-2df5fa453ecd_isrForCrosstalkSources_855423_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/287b80a6-0a20-43c1-9574-2df5fa453ecd_isrForCrosstalkSources_855423_53.stderr parsl.dataflow.dflow INFO: Task 3647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae7de885fb4002041f75b7d6f00e6adb with result from task 3647 parsl.dataflow.dflow DEBUG: Task 3647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f2ca3ed1-55b0-4bb2-8111-11ab11d27ef9_isrForCrosstalkSources_855400_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/f2ca3ed1-55b0-4bb2-8111-11ab11d27ef9_isrForCrosstalkSources_855400_14.stderr parsl.dataflow.dflow INFO: Task 3722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d99b3de7b2be3d940e73c189c9993a1 with result from task 3722 parsl.dataflow.dflow DEBUG: Task 3722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6682204e-c77b-400b-9493-94da19955df6_isrForCrosstalkSources_855397_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6682204e-c77b-400b-9493-94da19955df6_isrForCrosstalkSources_855397_52.stderr parsl.dataflow.dflow INFO: Task 3680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eed0d4777dbec5fb1898ec6e31bf094 with result from task 3680 parsl.dataflow.dflow DEBUG: Task 3680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fd26399e-a084-4376-97b0-38b019b24183_isrForCrosstalkSources_855423_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/fd26399e-a084-4376-97b0-38b019b24183_isrForCrosstalkSources_855423_61.stderr parsl.dataflow.dflow INFO: Task 3650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf23c1bd1263327ac35662204d808c3b with result from task 3650 parsl.dataflow.dflow DEBUG: Task 3650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a6c2e588-65ba-429a-aa72-8324be19c7da_isrForCrosstalkSources_855391_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/a6c2e588-65ba-429a-aa72-8324be19c7da_isrForCrosstalkSources_855391_48.stderr parsl.dataflow.dflow INFO: Task 3678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23bec19bef123b74a3c7762052f525ea with result from task 3678 parsl.dataflow.dflow DEBUG: Task 3678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/111b17ea-ca2b-401b-809a-d30dafcb5c89_isrForCrosstalkSources_855442_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/111b17ea-ca2b-401b-809a-d30dafcb5c89_isrForCrosstalkSources_855442_30.stderr parsl.dataflow.dflow INFO: Task 3673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d863b42bcb0408b39bf6f1c256c8dc72 with result from task 3673 parsl.dataflow.dflow DEBUG: Task 3673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f4c1e969-338d-42ac-ab38-64ef5a6e4796_isrForCrosstalkSources_855404_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/f4c1e969-338d-42ac-ab38-64ef5a6e4796_isrForCrosstalkSources_855404_32.stderr 2024-03-09 14:55:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964297 proc_lsst.multi INFO: [multi] job_id 16964297 2024-03-09 14:55:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 207 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 207 2024-03-09 14:55:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964297 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964297 parsl.executors.status_handling DEBUG: Launched block 262 on executor multi with job ID 16964297 parsl.executors.status_handling INFO: Allocated block ID 263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-263 2024-03-09 14:55:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 207 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 207 2024-03-09 14:55:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee223419cebe908b715ccc6e5e7c4888 with result from task 3716 parsl.dataflow.dflow DEBUG: Task 3716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9676fe8c-31eb-4aa4-b319-e946100f08fb_isrForCrosstalkSources_855382_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/9676fe8c-31eb-4aa4-b319-e946100f08fb_isrForCrosstalkSources_855382_31.stderr parsl.dataflow.dflow INFO: Task 3717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f7908931096277309f752456913aa71 with result from task 3717 parsl.dataflow.dflow DEBUG: Task 3717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e5b7a2f0-5d66-4286-8ce4-a2f6867ec9da_isrForCrosstalkSources_855416_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e5b7a2f0-5d66-4286-8ce4-a2f6867ec9da_isrForCrosstalkSources_855416_46.stderr parsl.dataflow.dflow INFO: Task 3824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcd45c34b4ae53c7ffa09735a829ae3a with result from task 3824 parsl.dataflow.dflow DEBUG: Task 3824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/88a3c085-308d-4f9d-b67f-b6120e96e3ec_isrForCrosstalkSources_855420_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/88a3c085-308d-4f9d-b67f-b6120e96e3ec_isrForCrosstalkSources_855420_16.stderr parsl.dataflow.dflow INFO: Task 3880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a7be1b28a8e9c3ed4f3dd30ae25fa3c with result from task 3880 parsl.dataflow.dflow DEBUG: Task 3880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/4f4202a2-4193-45ed-8051-25567e5fb333_isrForCrosstalkSources_855446_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/4f4202a2-4193-45ed-8051-25567e5fb333_isrForCrosstalkSources_855446_24.stderr parsl.dataflow.dflow INFO: Task 3721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a8917b5a019fa9897c6e7d271a95f06 with result from task 3721 parsl.dataflow.dflow DEBUG: Task 3721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/303570a2-2ad2-447f-8747-626876a23a62_isrForCrosstalkSources_855403_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/303570a2-2ad2-447f-8747-626876a23a62_isrForCrosstalkSources_855403_6.stderr parsl.dataflow.dflow INFO: Task 3832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f168c71766e97e7b86a9876cb6c2ac5 with result from task 3832 parsl.dataflow.dflow DEBUG: Task 3832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/01ee78d2-3b72-4676-9ca8-86f8eb0dad85_isrForCrosstalkSources_855404_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/01ee78d2-3b72-4676-9ca8-86f8eb0dad85_isrForCrosstalkSources_855404_17.stderr parsl.dataflow.dflow INFO: Task 3726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39306a411c439cded272ecd9c76a0c5a with result from task 3726 parsl.dataflow.dflow DEBUG: Task 3726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/cd2e5160-3def-4d85-84dc-b86cd8bbf21f_isrForCrosstalkSources_855388_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/cd2e5160-3def-4d85-84dc-b86cd8bbf21f_isrForCrosstalkSources_855388_28.stderr parsl.dataflow.dflow INFO: Task 3703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8a8bd3b1c772a7c1ac1572fb0c14e33 with result from task 3703 parsl.dataflow.dflow DEBUG: Task 3703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7bc3ca23-8093-4078-8f74-887287bada26_isrForCrosstalkSources_855413_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/7bc3ca23-8093-4078-8f74-887287bada26_isrForCrosstalkSources_855413_25.stderr parsl.dataflow.dflow INFO: Task 3779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 256f70a23d06502bc0aabd0234d2a14d with result from task 3779 parsl.dataflow.dflow DEBUG: Task 3779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5aab5506-fe4e-48cc-aeb9-6165b6fd194f_isrForCrosstalkSources_855434_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/5aab5506-fe4e-48cc-aeb9-6165b6fd194f_isrForCrosstalkSources_855434_45.stderr parsl.dataflow.dflow INFO: Task 3316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ee45eb9fe84596ea728e72c55479df4 with result from task 3316 parsl.dataflow.dflow DEBUG: Task 3316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/24d7cb09-956f-4c08-9491-7d728c52f3b6_isrForCrosstalkSources_855426_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/24d7cb09-956f-4c08-9491-7d728c52f3b6_isrForCrosstalkSources_855426_44.stderr parsl.dataflow.dflow INFO: Task 3885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 900467fdcc995ba5fdda89ee1ebd8835 with result from task 3885 parsl.dataflow.dflow DEBUG: Task 3885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/d654b0ad-9b79-40c7-a6e9-5c13091d5c4a_isrForCrosstalkSources_855418_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/d654b0ad-9b79-40c7-a6e9-5c13091d5c4a_isrForCrosstalkSources_855418_28.stderr 2024-03-09 14:55:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964298 proc_lsst.multi INFO: [multi] job_id 16964298 2024-03-09 14:55:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 208 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 208 2024-03-09 14:55:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964298 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964298 parsl.dataflow.dflow INFO: Task 3777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27fab50465900de36aa0579af59b2af9 with result from task 3777 parsl.executors.status_handling DEBUG: Launched block 263 on executor multi with job ID 16964298 parsl.executors.status_handling INFO: Allocated block ID 264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-264 2024-03-09 14:55:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 2024-03-09 14:55:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 208 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 208 2024-03-09 14:55:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0ccba4cc-6240-4e55-9098-830f3c94ed16_isrForCrosstalkSources_855452_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/0ccba4cc-6240-4e55-9098-830f3c94ed16_isrForCrosstalkSources_855452_5.stderr parsl.dataflow.dflow INFO: Task 3480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27036107d697c2c2c92b447cf8fda4f7 with result from task 3480 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 3480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4226e1f4-21ea-462d-80ff-906cab3626b3_isrForCrosstalkSources_855440_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/4226e1f4-21ea-462d-80ff-906cab3626b3_isrForCrosstalkSources_855440_54.stderr parsl.dataflow.dflow INFO: Task 3564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5950776301b7e3ad5fc16129721ac462 with result from task 3564 parsl.dataflow.dflow DEBUG: Task 3564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/208529ab-15fa-47b2-bb59-119ca1e81cf3_isrForCrosstalkSources_855441_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/208529ab-15fa-47b2-bb59-119ca1e81cf3_isrForCrosstalkSources_855441_59.stderr parsl.dataflow.dflow INFO: Task 3366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55f7e2de965ba70ad7ea4e8ded84184b with result from task 3366 parsl.dataflow.dflow DEBUG: Task 3366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/28b08aae-ed3c-4b0e-b078-44b872797cde_isrForCrosstalkSources_855418_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/28b08aae-ed3c-4b0e-b078-44b872797cde_isrForCrosstalkSources_855418_49.stderr parsl.dataflow.dflow INFO: Task 3604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eaaf6efa5feabe380bd983e8cad212f with result from task 3604 parsl.dataflow.dflow DEBUG: Task 3604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/d8c9831b-620b-4c42-8a21-2af2dfdf7961_isrForCrosstalkSources_855405_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/d8c9831b-620b-4c42-8a21-2af2dfdf7961_isrForCrosstalkSources_855405_5.stderr parsl.dataflow.dflow INFO: Task 3675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792935a6f32144ca3653cb99d196273f with result from task 3675 parsl.dataflow.dflow DEBUG: Task 3675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/e7950baf-0b3c-413c-b50c-872095305331_isrForCrosstalkSources_855387_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/e7950baf-0b3c-413c-b50c-872095305331_isrForCrosstalkSources_855387_21.stderr parsl.dataflow.dflow INFO: Task 3485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a252abe31bda9ed3fa235539b94b847 with result from task 3485 parsl.dataflow.dflow DEBUG: Task 3485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/27c5674b-b6d3-4df4-8545-dfd0a3f068af_isrForCrosstalkSources_855416_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/27c5674b-b6d3-4df4-8545-dfd0a3f068af_isrForCrosstalkSources_855416_2.stderr parsl.dataflow.dflow INFO: Task 3810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e0b37b0ddbe5d0ed366df77aa29bea with result from task 3810 parsl.dataflow.dflow DEBUG: Task 3810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8e1c73f4-cd8b-4b4f-b5d0-b636c1bf85e9_isrForCrosstalkSources_855381_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/8e1c73f4-cd8b-4b4f-b5d0-b636c1bf85e9_isrForCrosstalkSources_855381_12.stderr parsl.dataflow.dflow INFO: Task 3689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 044529d537f36712c4582133f5cbf6f6 with result from task 3689 parsl.dataflow.dflow DEBUG: Task 3689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d09a11e2-7715-4992-953b-9cfe38f890f8_isrForCrosstalkSources_855432_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d09a11e2-7715-4992-953b-9cfe38f890f8_isrForCrosstalkSources_855432_6.stderr parsl.dataflow.dflow INFO: Task 3540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da9269b5c3eb446024e91be60d7af5cd with result from task 3540 parsl.dataflow.dflow DEBUG: Task 3540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ce9d494-2458-4e13-9242-cf909220e914_isrForCrosstalkSources_855396_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/9ce9d494-2458-4e13-9242-cf909220e914_isrForCrosstalkSources_855396_41.stderr parsl.dataflow.dflow INFO: Task 3552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45a34999839717c48d585c977317934a with result from task 3552 parsl.dataflow.dflow DEBUG: Task 3552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f73e6ad1-cc06-4254-99d2-1bd4a07f4750_isrForCrosstalkSources_855420_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/f73e6ad1-cc06-4254-99d2-1bd4a07f4750_isrForCrosstalkSources_855420_5.stderr parsl.dataflow.dflow INFO: Task 3778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c986802c4f03f72e88d11de88c07e2 with result from task 3778 parsl.dataflow.dflow DEBUG: Task 3778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a952f553-4488-4dbb-b39d-96c6303871c0_isrForCrosstalkSources_855455_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a952f553-4488-4dbb-b39d-96c6303871c0_isrForCrosstalkSources_855455_33.stderr parsl.dataflow.dflow INFO: Task 3532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0d408b854cfa3af84327fe8d46dbfde with result from task 3532 parsl.dataflow.dflow DEBUG: Task 3532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f4659f73-be9c-4973-9f14-499010998dc8_isrForCrosstalkSources_855397_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f4659f73-be9c-4973-9f14-499010998dc8_isrForCrosstalkSources_855397_48.stderr parsl.dataflow.dflow INFO: Task 3817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee3849e858497b46139274fba5d5ff0 with result from task 3817 parsl.dataflow.dflow DEBUG: Task 3817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ebd4b793-2eea-4270-8543-f5119d83ccf9_isrForCrosstalkSources_855381_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/ebd4b793-2eea-4270-8543-f5119d83ccf9_isrForCrosstalkSources_855381_25.stderr parsl.dataflow.dflow INFO: Task 3874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f5531909e61eca032ccc9d376b17845 with result from task 3874 parsl.dataflow.dflow DEBUG: Task 3874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0af23013-2ea9-4499-8343-27996a76bdaf_isrForCrosstalkSources_855456_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0af23013-2ea9-4499-8343-27996a76bdaf_isrForCrosstalkSources_855456_45.stderr parsl.dataflow.dflow INFO: Task 3619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9797786dc18bee3511a338d35cf04e2 with result from task 3619 parsl.dataflow.dflow DEBUG: Task 3619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6481ce8d-a74d-456d-bc24-292b583eb08d_isrForCrosstalkSources_855385_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6481ce8d-a74d-456d-bc24-292b583eb08d_isrForCrosstalkSources_855385_34.stderr parsl.dataflow.dflow INFO: Task 3684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89254b701d80b51aa616bda927249103 with result from task 3684 parsl.dataflow.dflow DEBUG: Task 3684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6b0fb617-d215-4ebb-9f78-9637d0d1679f_isrForCrosstalkSources_855456_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6b0fb617-d215-4ebb-9f78-9637d0d1679f_isrForCrosstalkSources_855456_23.stderr parsl.dataflow.dflow INFO: Task 3820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e339d8a636ebe1ec6f4dcae35e8c6cad with result from task 3820 parsl.dataflow.dflow DEBUG: Task 3820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/6c9840a7-9939-4dda-ba02-96def72af1dc_isrForCrosstalkSources_855430_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/6c9840a7-9939-4dda-ba02-96def72af1dc_isrForCrosstalkSources_855430_51.stderr parsl.dataflow.dflow INFO: Task 3870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae69e59d08a9d812144e8c0d9296d45a with result from task 3870 parsl.dataflow.dflow DEBUG: Task 3870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/63ce581c-9188-4bef-b044-c07bf49802da_isrForCrosstalkSources_855445_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/63ce581c-9188-4bef-b044-c07bf49802da_isrForCrosstalkSources_855445_3.stderr parsl.dataflow.dflow INFO: Task 3857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e14a7eebbcdec4a734d4637e746f05b2 with result from task 3857 parsl.dataflow.dflow DEBUG: Task 3857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb070a53-0b29-4dc5-bd78-89e54b89737a_isrForCrosstalkSources_855386_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb070a53-0b29-4dc5-bd78-89e54b89737a_isrForCrosstalkSources_855386_8.stderr parsl.dataflow.dflow INFO: Task 3748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 586281e4b55be6cbe340c256cc15cd50 with result from task 3748 parsl.dataflow.dflow DEBUG: Task 3748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5e490d57-bd2d-4210-a2af-03a5f9f24e57_isrForCrosstalkSources_855405_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5e490d57-bd2d-4210-a2af-03a5f9f24e57_isrForCrosstalkSources_855405_21.stderr parsl.dataflow.dflow INFO: Task 3621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f755bd5f3384e9afafe662715fa804dc with result from task 3621 parsl.dataflow.dflow DEBUG: Task 3621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/85c96fc0-b25e-42de-bac7-3512a7624511_isrForCrosstalkSources_855389_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/85c96fc0-b25e-42de-bac7-3512a7624511_isrForCrosstalkSources_855389_40.stderr parsl.dataflow.dflow INFO: Task 3663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb26f3867b79c4bd0ac4495002d4459a with result from task 3663 parsl.dataflow.dflow DEBUG: Task 3663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bc803bc7-51f8-4cfd-b956-03d6277874f3_isrForCrosstalkSources_855443_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bc803bc7-51f8-4cfd-b956-03d6277874f3_isrForCrosstalkSources_855443_7.stderr parsl.dataflow.dflow INFO: Task 3811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86744de8acc3b2fa5f229b1c657419a1 with result from task 3811 parsl.dataflow.dflow DEBUG: Task 3811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8820a789-a76e-4d3b-9cee-fb4321b37691_isrForCrosstalkSources_855441_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8820a789-a76e-4d3b-9cee-fb4321b37691_isrForCrosstalkSources_855441_56.stderr parsl.dataflow.dflow INFO: Task 3371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c810dc501424d27e40b2af65e9d74333 with result from task 3371 parsl.dataflow.dflow DEBUG: Task 3371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/31dace39-4684-443c-99c5-5b869aa5b392_isrForCrosstalkSources_855391_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/31dace39-4684-443c-99c5-5b869aa5b392_isrForCrosstalkSources_855391_11.stderr parsl.dataflow.dflow INFO: Task 3329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe6b13b0ed146c74ae4240627cb99623 with result from task 3329 parsl.dataflow.dflow DEBUG: Task 3329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f94bbcb-eaf2-439d-990c-0151cf42b738_isrForCrosstalkSources_855415_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/2f94bbcb-eaf2-439d-990c-0151cf42b738_isrForCrosstalkSources_855415_41.stderr parsl.dataflow.dflow INFO: Task 3772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b182cc80b20244d0779fc05d2a7744b with result from task 3772 parsl.dataflow.dflow DEBUG: Task 3772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/48977c39-e3d3-483a-bb6c-3255ab139dc6_isrForCrosstalkSources_855450_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/48977c39-e3d3-483a-bb6c-3255ab139dc6_isrForCrosstalkSources_855450_14.stderr parsl.dataflow.dflow INFO: Task 3369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b5a15abb7569e0512ba05ad034b4964 with result from task 3369 parsl.dataflow.dflow DEBUG: Task 3369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aa3af8ba-2f35-43e4-b933-aa585a4c3621_isrForCrosstalkSources_855449_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/aa3af8ba-2f35-43e4-b933-aa585a4c3621_isrForCrosstalkSources_855449_59.stderr parsl.dataflow.dflow INFO: Task 3691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d5b0b2e9a2c92d15097f7590d72a83f with result from task 3691 parsl.dataflow.dflow DEBUG: Task 3691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2023b467-1887-40cd-88b0-d78af183575c_isrForCrosstalkSources_855392_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2023b467-1887-40cd-88b0-d78af183575c_isrForCrosstalkSources_855392_41.stderr parsl.dataflow.dflow INFO: Task 3750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 132f737c1a8e139a74ad1a63ba2d8227 with result from task 3750 parsl.dataflow.dflow DEBUG: Task 3750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f043d470-8efd-4c2e-b4fe-e93cda53fd49_isrForCrosstalkSources_855417_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/f043d470-8efd-4c2e-b4fe-e93cda53fd49_isrForCrosstalkSources_855417_48.stderr parsl.dataflow.dflow INFO: Task 3887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a05adf27e7df6cff136d6339264a9d2d with result from task 3887 parsl.dataflow.dflow DEBUG: Task 3887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/76935a02-32d9-4b77-889c-d2a7f7a106f7_isrForCrosstalkSources_855424_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/76935a02-32d9-4b77-889c-d2a7f7a106f7_isrForCrosstalkSources_855424_31.stderr parsl.dataflow.dflow INFO: Task 3826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a92cfc811edf01676da2dda3bc70b396 with result from task 3826 parsl.dataflow.dflow DEBUG: Task 3826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ed0cb72d-ccec-4dd5-8338-09253664f229_isrForCrosstalkSources_855403_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ed0cb72d-ccec-4dd5-8338-09253664f229_isrForCrosstalkSources_855403_36.stderr parsl.dataflow.dflow INFO: Task 3487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fdf80dfe8de7d54a4a8c3c62d281a26 with result from task 3487 parsl.dataflow.dflow DEBUG: Task 3487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/55de4cc2-73b9-48f2-90a2-d5d5d23a10c3_isrForCrosstalkSources_855425_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/55de4cc2-73b9-48f2-90a2-d5d5d23a10c3_isrForCrosstalkSources_855425_37.stderr parsl.dataflow.dflow INFO: Task 3504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 521797f947f49d058c7772c1576ecb4d with result from task 3504 parsl.dataflow.dflow DEBUG: Task 3504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6dd07147-7195-40ea-b90c-41544a8dd14f_isrForCrosstalkSources_855397_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6dd07147-7195-40ea-b90c-41544a8dd14f_isrForCrosstalkSources_855397_6.stderr parsl.dataflow.dflow INFO: Task 3597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8226cacd90c708f3fb569bfe5d032c08 with result from task 3597 parsl.dataflow.dflow DEBUG: Task 3597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/8aa592c5-09e8-40e7-9ed8-507a0cd55416_isrForCrosstalkSources_855451_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/8aa592c5-09e8-40e7-9ed8-507a0cd55416_isrForCrosstalkSources_855451_56.stderr parsl.dataflow.dflow INFO: Task 3741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89ecd28768fe3e6fc2f0b9ff6a160e01 with result from task 3741 parsl.dataflow.dflow DEBUG: Task 3741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/eef2184e-84ae-44b0-9f8a-1d660abd38f8_isrForCrosstalkSources_855455_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/eef2184e-84ae-44b0-9f8a-1d660abd38f8_isrForCrosstalkSources_855455_4.stderr parsl.dataflow.dflow INFO: Task 3578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 639c25de9362d342994e0c87218ac0ec with result from task 3578 parsl.dataflow.dflow DEBUG: Task 3578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/310f8fda-cfe4-4389-90f6-a642e22d3e8d_isrForCrosstalkSources_855459_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/310f8fda-cfe4-4389-90f6-a642e22d3e8d_isrForCrosstalkSources_855459_8.stderr parsl.dataflow.dflow INFO: Task 3421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfa8b6c6a7d9ba8a971471e8b2bee495 with result from task 3421 parsl.dataflow.dflow DEBUG: Task 3421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/22420c90-cd35-4d88-b178-2d0d7fcc6a76_isrForCrosstalkSources_855391_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/22420c90-cd35-4d88-b178-2d0d7fcc6a76_isrForCrosstalkSources_855391_47.stderr parsl.dataflow.dflow INFO: Task 3813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f863efc6c47143923cd81962d08caa12 with result from task 3813 parsl.dataflow.dflow DEBUG: Task 3813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/47dbfa3d-0b3f-42e5-a845-4a36b9634961_isrForCrosstalkSources_855454_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/47dbfa3d-0b3f-42e5-a845-4a36b9634961_isrForCrosstalkSources_855454_46.stderr parsl.dataflow.dflow INFO: Task 3934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c429e9b4a1e26e1462005d059646cea5 with result from task 3934 parsl.dataflow.dflow DEBUG: Task 3934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a8eb784d-bb22-4b03-899b-64db91c78bb8_isrForCrosstalkSources_855410_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a8eb784d-bb22-4b03-899b-64db91c78bb8_isrForCrosstalkSources_855410_27.stderr parsl.dataflow.dflow INFO: Task 3594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0d95035a7d76d37f080f2e73e618d6b with result from task 3594 parsl.dataflow.dflow DEBUG: Task 3594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/3e25e837-931b-40f9-aae7-4147063e5388_isrForCrosstalkSources_855441_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/3e25e837-931b-40f9-aae7-4147063e5388_isrForCrosstalkSources_855441_11.stderr parsl.dataflow.dflow INFO: Task 3729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dd29172854b6ddbe03fd6bfab2a5500 with result from task 3729 parsl.dataflow.dflow DEBUG: Task 3729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e1fa3ac0-e44a-4b97-8b7b-9a30e8f90198_isrForCrosstalkSources_855432_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e1fa3ac0-e44a-4b97-8b7b-9a30e8f90198_isrForCrosstalkSources_855432_31.stderr parsl.dataflow.dflow INFO: Task 3897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70f4a6d502607e3c2acea71c8b7d714 with result from task 3897 parsl.dataflow.dflow DEBUG: Task 3897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f0a8532c-9235-4370-96da-c8c4851dc746_isrForCrosstalkSources_855419_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f0a8532c-9235-4370-96da-c8c4851dc746_isrForCrosstalkSources_855419_42.stderr parsl.dataflow.dflow INFO: Task 3725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87558b40b4afab03ec75c2be071385b6 with result from task 3725 parsl.dataflow.dflow DEBUG: Task 3725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f6e01602-c6fc-4aa7-8323-570b7bca8900_isrForCrosstalkSources_855453_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/f6e01602-c6fc-4aa7-8323-570b7bca8900_isrForCrosstalkSources_855453_1.stderr parsl.dataflow.dflow INFO: Task 3572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5273292c3d6009aee9f0e646ade35166 with result from task 3572 parsl.dataflow.dflow DEBUG: Task 3572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4371254d-56b5-418a-aa9e-a4f2d9f8ce5c_isrForCrosstalkSources_855403_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4371254d-56b5-418a-aa9e-a4f2d9f8ce5c_isrForCrosstalkSources_855403_25.stderr parsl.dataflow.dflow INFO: Task 3602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bef1ea68c2f799b3cbc769e2023cc63d with result from task 3602 parsl.dataflow.dflow DEBUG: Task 3602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c61c0b86-4ece-4ed4-bed0-5c7aa4f364bc_isrForCrosstalkSources_855403_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/c61c0b86-4ece-4ed4-bed0-5c7aa4f364bc_isrForCrosstalkSources_855403_49.stderr parsl.dataflow.dflow INFO: Task 3599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea287947c7261157783b2879a4048dc6 with result from task 3599 parsl.dataflow.dflow DEBUG: Task 3599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c5ec8882-563a-442e-8750-dfe7c440b9e6_isrForCrosstalkSources_855441_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/c5ec8882-563a-442e-8750-dfe7c440b9e6_isrForCrosstalkSources_855441_54.stderr parsl.dataflow.dflow INFO: Task 3838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a362e06a53cc9bde2b3425c8d595aa77 with result from task 3838 parsl.dataflow.dflow DEBUG: Task 3838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2a588564-a765-4c51-a437-05ed67187ecd_isrForCrosstalkSources_855455_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/2a588564-a765-4c51-a437-05ed67187ecd_isrForCrosstalkSources_855455_6.stderr parsl.dataflow.dflow INFO: Task 3613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44b4f863821c2c88aa46ed9e681acad2 with result from task 3613 2024-03-09 14:55:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964299 proc_lsst.multi INFO: [multi] job_id 16964299 2024-03-09 14:55:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 209 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 209 2024-03-09 14:55:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964299 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964299 parsl.executors.status_handling DEBUG: Launched block 264 on executor multi with job ID 16964299 parsl.executors.status_handling INFO: Allocated block ID 265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-265 2024-03-09 14:55:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 2024-03-09 14:55:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 209 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 209 2024-03-09 14:55:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/d97144e7-8dad-4bf4-8b62-8e53dc047c1f_isrForCrosstalkSources_855441_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/d97144e7-8dad-4bf4-8b62-8e53dc047c1f_isrForCrosstalkSources_855441_44.stderr parsl.dataflow.dflow INFO: Task 3590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7dfc0b9d9c20cb7292bd20fb967bdd0 with result from task 3590 parsl.dataflow.dflow DEBUG: Task 3590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e0ea03d3-ad1c-4dc9-ab47-cd0a14766c9b_isrForCrosstalkSources_855385_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e0ea03d3-ad1c-4dc9-ab47-cd0a14766c9b_isrForCrosstalkSources_855385_52.stderr parsl.dataflow.dflow INFO: Task 3567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7045c8084aa9f84da97a1491256be9c0 with result from task 3567 parsl.dataflow.dflow DEBUG: Task 3567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/008a57b0-f81f-4fbc-afd7-cebc141fdcd1_isrForCrosstalkSources_855436_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/008a57b0-f81f-4fbc-afd7-cebc141fdcd1_isrForCrosstalkSources_855436_14.stderr parsl.dataflow.dflow INFO: Task 3773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ca4359238bb02f1504203b456fa162a with result from task 3773 parsl.dataflow.dflow DEBUG: Task 3773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68c66a67-ea5f-4f13-83cc-2817db5d66f9_isrForCrosstalkSources_855399_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68c66a67-ea5f-4f13-83cc-2817db5d66f9_isrForCrosstalkSources_855399_49.stderr parsl.dataflow.dflow INFO: Task 3855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e880f25d3cb371c88b17257111e9cdb1 with result from task 3855 parsl.dataflow.dflow DEBUG: Task 3855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9236d008-0735-459d-898e-88723bbdb4cd_isrForCrosstalkSources_855457_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9236d008-0735-459d-898e-88723bbdb4cd_isrForCrosstalkSources_855457_30.stderr parsl.dataflow.dflow INFO: Task 3815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca76ee64b542beb08d24d99ebb005915 with result from task 3815 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 3815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf370c1b-c292-4f48-aa6d-9456af94e630_isrForCrosstalkSources_855432_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/bf370c1b-c292-4f48-aa6d-9456af94e630_isrForCrosstalkSources_855432_34.stderr parsl.dataflow.dflow INFO: Task 3769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ec19610f0da8aac5dbe7e6919229f0 with result from task 3769 parsl.dataflow.dflow DEBUG: Task 3769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/64b8e42c-9657-4d76-9915-879f701b73cc_isrForCrosstalkSources_855388_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/64b8e42c-9657-4d76-9915-879f701b73cc_isrForCrosstalkSources_855388_52.stderr parsl.dataflow.dflow INFO: Task 3575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd11729cbb66ab182b18de9a7d43b064 with result from task 3575 parsl.dataflow.dflow DEBUG: Task 3575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cc4e4b61-4e6b-4f25-af64-f6aa263b6174_isrForCrosstalkSources_855407_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cc4e4b61-4e6b-4f25-af64-f6aa263b6174_isrForCrosstalkSources_855407_58.stderr parsl.dataflow.dflow INFO: Task 3796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6843b24ebcf7511a85e88528d6a587e3 with result from task 3796 parsl.dataflow.dflow DEBUG: Task 3796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4872d037-545d-425c-8516-cc49679bcd37_isrForCrosstalkSources_855403_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4872d037-545d-425c-8516-cc49679bcd37_isrForCrosstalkSources_855403_34.stderr parsl.dataflow.dflow INFO: Task 3833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66da9028cc6859f6c46d538df029112d with result from task 3833 parsl.dataflow.dflow DEBUG: Task 3833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ef72c06f-44cc-46b4-8942-333f84e32b6a_isrForCrosstalkSources_855390_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/ef72c06f-44cc-46b4-8942-333f84e32b6a_isrForCrosstalkSources_855390_38.stderr parsl.dataflow.dflow INFO: Task 3539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2a75320714998ca176b7c5331a2248d with result from task 3539 parsl.dataflow.dflow DEBUG: Task 3539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58678676-eb43-4140-8bf3-169a63924fba_isrForCrosstalkSources_855407_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/58678676-eb43-4140-8bf3-169a63924fba_isrForCrosstalkSources_855407_49.stderr parsl.dataflow.dflow INFO: Task 3878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2d006c7f66a9521989ed86df62e6a62 with result from task 3878 parsl.dataflow.dflow DEBUG: Task 3878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/4a473633-5257-4127-a5cd-5f1914ce594b_isrForCrosstalkSources_855442_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/4a473633-5257-4127-a5cd-5f1914ce594b_isrForCrosstalkSources_855442_48.stderr parsl.dataflow.dflow INFO: Task 3586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1464b14b7b03dc216b988caa02939feb with result from task 3586 parsl.dataflow.dflow DEBUG: Task 3586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/6d359c1d-22f3-4381-82ad-a09a5fafbeee_isrForCrosstalkSources_855399_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/6d359c1d-22f3-4381-82ad-a09a5fafbeee_isrForCrosstalkSources_855399_53.stderr parsl.dataflow.dflow INFO: Task 3728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c61d2ee20d4bafcbb1d84a3eef9abfa6 with result from task 3728 parsl.dataflow.dflow DEBUG: Task 3728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6b3420bc-2a5a-4417-9a46-ad60a6ba3da9_isrForCrosstalkSources_855394_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6b3420bc-2a5a-4417-9a46-ad60a6ba3da9_isrForCrosstalkSources_855394_24.stderr parsl.dataflow.dflow INFO: Task 3554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab8c29f980dd63e7e5a81326ece4fa14 with result from task 3554 parsl.dataflow.dflow DEBUG: Task 3554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/423859e9-0d1d-49f9-97f4-e4c05fb38a6c_isrForCrosstalkSources_855393_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/423859e9-0d1d-49f9-97f4-e4c05fb38a6c_isrForCrosstalkSources_855393_55.stderr parsl.dataflow.dflow INFO: Task 3562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4401830838d6cb6b2351d6dde4136ac3 with result from task 3562 parsl.dataflow.dflow DEBUG: Task 3562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/84925ca2-d0c0-4191-8b0e-20ef9d9dcddf_isrForCrosstalkSources_855398_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/84925ca2-d0c0-4191-8b0e-20ef9d9dcddf_isrForCrosstalkSources_855398_30.stderr parsl.dataflow.dflow INFO: Task 3762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcbd8b1e14a8d31c928140ff0c2e713c with result from task 3762 parsl.dataflow.dflow DEBUG: Task 3762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/93dfa094-786e-49ef-8dd9-ff1b28428fa6_isrForCrosstalkSources_855403_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/93dfa094-786e-49ef-8dd9-ff1b28428fa6_isrForCrosstalkSources_855403_11.stderr parsl.dataflow.dflow INFO: Task 3849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0681203de59dcf28c75807e58b10aa83 with result from task 3849 parsl.dataflow.dflow DEBUG: Task 3849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/acaa8651-5041-4a5a-b4c6-532be4f40ebd_isrForCrosstalkSources_855384_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/acaa8651-5041-4a5a-b4c6-532be4f40ebd_isrForCrosstalkSources_855384_7.stderr parsl.dataflow.dflow INFO: Task 3865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a544f97a22aa6b19b2a49ffb4edbc7b with result from task 3865 parsl.dataflow.dflow DEBUG: Task 3865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f027c974-951a-4a82-8e94-617a3f898de8_isrForCrosstalkSources_855384_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f027c974-951a-4a82-8e94-617a3f898de8_isrForCrosstalkSources_855384_41.stderr parsl.dataflow.dflow INFO: Task 3609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f96c346c02d23a4643f22e7fd811ffdd with result from task 3609 parsl.dataflow.dflow DEBUG: Task 3609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d3e78d10-6db5-4cd8-a155-28484e26f4e1_isrForCrosstalkSources_855457_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d3e78d10-6db5-4cd8-a155-28484e26f4e1_isrForCrosstalkSources_855457_31.stderr parsl.dataflow.dflow INFO: Task 3617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3600a8495f65cf7b7c2c47d3b1ffe72a with result from task 3617 parsl.dataflow.dflow DEBUG: Task 3617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e2e0fedc-9f2c-4304-ab20-fa65b91343aa_isrForCrosstalkSources_855404_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e2e0fedc-9f2c-4304-ab20-fa65b91343aa_isrForCrosstalkSources_855404_1.stderr parsl.dataflow.dflow INFO: Task 3496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddfa39f884035c92d8e744966e73d6ac with result from task 3496 parsl.dataflow.dflow DEBUG: Task 3496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7cfb1d5c-50ef-4557-a1bd-f7222b462dc3_isrForCrosstalkSources_855458_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/7cfb1d5c-50ef-4557-a1bd-f7222b462dc3_isrForCrosstalkSources_855458_59.stderr parsl.dataflow.dflow INFO: Task 3661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b96e040fddf015e0d30dbf2dbfabda0b with result from task 3661 parsl.dataflow.dflow DEBUG: Task 3661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6ecde8cb-b10e-46b4-b0ae-83e35798a744_isrForCrosstalkSources_855394_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/6ecde8cb-b10e-46b4-b0ae-83e35798a744_isrForCrosstalkSources_855394_50.stderr parsl.dataflow.dflow INFO: Task 3867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b71a03d808dc2f81b7a818bceebaa84 with result from task 3867 parsl.dataflow.dflow DEBUG: Task 3867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/99cde655-876e-4875-bb67-ebab618c9e81_isrForCrosstalkSources_855459_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/99cde655-876e-4875-bb67-ebab618c9e81_isrForCrosstalkSources_855459_2.stderr parsl.dataflow.dflow INFO: Task 3600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15bc0a003993be204f9fc196b10cdfe with result from task 3600 parsl.dataflow.dflow DEBUG: Task 3600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9db97005-ecae-42c7-baf8-3e623ce2c52b_isrForCrosstalkSources_855406_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9db97005-ecae-42c7-baf8-3e623ce2c52b_isrForCrosstalkSources_855406_22.stderr parsl.dataflow.dflow INFO: Task 3836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df7fdd0a18c6027dcdaa38d4e04a0c64 with result from task 3836 parsl.dataflow.dflow DEBUG: Task 3836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5aee3072-2901-4329-b832-cec8a30747ea_isrForCrosstalkSources_855436_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5aee3072-2901-4329-b832-cec8a30747ea_isrForCrosstalkSources_855436_56.stderr parsl.dataflow.dflow INFO: Task 3541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb4668c77cd90236f1d6b79a00b335d8 with result from task 3541 parsl.dataflow.dflow DEBUG: Task 3541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/5f1dbe3c-d355-421c-83cb-f94940d348de_isrForCrosstalkSources_855459_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/5f1dbe3c-d355-421c-83cb-f94940d348de_isrForCrosstalkSources_855459_27.stderr parsl.dataflow.dflow INFO: Task 3688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 834d7a2c8971f3db807b005f6b542476 with result from task 3688 parsl.dataflow.dflow DEBUG: Task 3688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ba7535b-d66f-4604-8883-3a40613fad69_isrForCrosstalkSources_855437_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/9ba7535b-d66f-4604-8883-3a40613fad69_isrForCrosstalkSources_855437_24.stderr parsl.dataflow.dflow INFO: Task 3592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 314732506efc4825ed01fb23ff0b6220 with result from task 3592 parsl.dataflow.dflow DEBUG: Task 3592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3df8bfa5-68d0-4b6b-87b3-73bc4846ffc5_isrForCrosstalkSources_855383_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3df8bfa5-68d0-4b6b-87b3-73bc4846ffc5_isrForCrosstalkSources_855383_41.stderr parsl.dataflow.dflow INFO: Task 3834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f861dbc54a9c9768bb31d0911e896044 with result from task 3834 parsl.dataflow.dflow DEBUG: Task 3834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/8fc7c475-e057-4d59-93e2-4ffe8c10f65d_isrForCrosstalkSources_855424_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/8fc7c475-e057-4d59-93e2-4ffe8c10f65d_isrForCrosstalkSources_855424_57.stderr parsl.dataflow.dflow INFO: Task 3782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be8ccf0b6dc85abbd9535660f0d84702 with result from task 3782 parsl.dataflow.dflow DEBUG: Task 3782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6f28eb13-32ba-4b2c-8bff-34ed32550661_isrForCrosstalkSources_855400_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6f28eb13-32ba-4b2c-8bff-34ed32550661_isrForCrosstalkSources_855400_12.stderr parsl.dataflow.dflow INFO: Task 3818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f256dcd0b2e774485886d893e72bb739 with result from task 3818 parsl.dataflow.dflow DEBUG: Task 3818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e609f697-9c6d-4ffb-afc6-ac044324e0aa_isrForCrosstalkSources_855413_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e609f697-9c6d-4ffb-afc6-ac044324e0aa_isrForCrosstalkSources_855413_21.stderr parsl.dataflow.dflow INFO: Task 3573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51f307bbb1a418a30c8dc56540fbc923 with result from task 3573 parsl.dataflow.dflow DEBUG: Task 3573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0486ccb4-6288-482b-a93b-c56becdb1963_isrForCrosstalkSources_855437_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0486ccb4-6288-482b-a93b-c56becdb1963_isrForCrosstalkSources_855437_60.stderr parsl.dataflow.dflow INFO: Task 3830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dbe7a156e43f84a4fae4f13f0ace8b5 with result from task 3830 parsl.dataflow.dflow DEBUG: Task 3830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/34116808-5bb7-4637-bad6-15e92009af6c_isrForCrosstalkSources_855454_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/34116808-5bb7-4637-bad6-15e92009af6c_isrForCrosstalkSources_855454_55.stderr parsl.dataflow.dflow INFO: Task 3938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61f00b52c097bc5996217db8787be3c0 with result from task 3938 parsl.dataflow.dflow DEBUG: Task 3938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/41f7834c-acd0-48d8-9d11-e57df0aa1d46_isrForCrosstalkSources_855435_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/41f7834c-acd0-48d8-9d11-e57df0aa1d46_isrForCrosstalkSources_855435_20.stderr parsl.dataflow.dflow INFO: Task 3360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0377e7e1ee8245b2548ff3c70aee926b with result from task 3360 parsl.dataflow.dflow DEBUG: Task 3360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ac10cf85-57e9-4f91-ba40-3f0527b45860_isrForCrosstalkSources_855404_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/ac10cf85-57e9-4f91-ba40-3f0527b45860_isrForCrosstalkSources_855404_6.stderr parsl.dataflow.dflow INFO: Task 3699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4437d9b8a99161c34ab9403fe0a9da with result from task 3699 parsl.dataflow.dflow DEBUG: Task 3699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2d7c856f-a111-4e6e-bc69-085301876a75_isrForCrosstalkSources_855429_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2d7c856f-a111-4e6e-bc69-085301876a75_isrForCrosstalkSources_855429_20.stderr parsl.dataflow.dflow INFO: Task 3393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca2b4824b3887da67e5e2a1275227371 with result from task 3393 parsl.dataflow.dflow DEBUG: Task 3393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bef7bc5e-93d3-4d33-9776-643a60d86c46_isrForCrosstalkSources_855381_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/bef7bc5e-93d3-4d33-9776-643a60d86c46_isrForCrosstalkSources_855381_31.stderr parsl.dataflow.dflow INFO: Task 3774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642029ad24556e07e61013cdb42c1edb with result from task 3774 parsl.dataflow.dflow DEBUG: Task 3774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/12fb37dd-f6a6-43af-b23c-cc9df7d54232_isrForCrosstalkSources_855386_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/12fb37dd-f6a6-43af-b23c-cc9df7d54232_isrForCrosstalkSources_855386_33.stderr parsl.dataflow.dflow INFO: Task 3763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154ba167c547dcf3248d0507d9c453ef with result from task 3763 parsl.dataflow.dflow DEBUG: Task 3763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/215fd0b5-7cbd-45f5-bcfe-ddf06d192f94_isrForCrosstalkSources_855426_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/215fd0b5-7cbd-45f5-bcfe-ddf06d192f94_isrForCrosstalkSources_855426_61.stderr parsl.dataflow.dflow INFO: Task 3979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a49717835f3a5fa1e30565382b3c5537 with result from task 3979 parsl.dataflow.dflow DEBUG: Task 3979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c792a03-eac7-4db0-8b01-5fcfb5ce778a_isrForCrosstalkSources_855435_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/1c792a03-eac7-4db0-8b01-5fcfb5ce778a_isrForCrosstalkSources_855435_19.stderr parsl.dataflow.dflow INFO: Task 3601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 802944041e834a515f4bb7b21c42826a with result from task 3601 parsl.dataflow.dflow DEBUG: Task 3601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/07ba5436-0aaf-4711-bda9-7ef9717d2302_isrForCrosstalkSources_855430_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/07ba5436-0aaf-4711-bda9-7ef9717d2302_isrForCrosstalkSources_855430_42.stderr parsl.dataflow.dflow INFO: Task 3854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93d0b7c7dd662711e009d4c6d5898cc8 with result from task 3854 parsl.dataflow.dflow DEBUG: Task 3854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b8c49923-e362-49f2-8f68-f61af9e89214_isrForCrosstalkSources_855393_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/b8c49923-e362-49f2-8f68-f61af9e89214_isrForCrosstalkSources_855393_34.stderr parsl.dataflow.dflow INFO: Task 3489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 716d129426fa0275371f905d1de7e1eb with result from task 3489 parsl.dataflow.dflow DEBUG: Task 3489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fbbf4bd7-8410-4125-951d-5810f14bcf0a_isrForCrosstalkSources_855382_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/fbbf4bd7-8410-4125-951d-5810f14bcf0a_isrForCrosstalkSources_855382_12.stderr parsl.dataflow.dflow INFO: Task 3513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b97704d3822803174cb7d90f4b149c08 with result from task 3513 parsl.dataflow.dflow DEBUG: Task 3513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/eac1223b-e185-44b6-b58b-3a104bbbb303_isrForCrosstalkSources_855390_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/eac1223b-e185-44b6-b58b-3a104bbbb303_isrForCrosstalkSources_855390_8.stderr parsl.dataflow.dflow INFO: Task 3527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d11211f495609cd0eeb11639939681bf with result from task 3527 parsl.dataflow.dflow DEBUG: Task 3527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/01e190a5-8570-4d91-b388-d3e6f2e5ecff_isrForCrosstalkSources_855393_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/01e190a5-8570-4d91-b388-d3e6f2e5ecff_isrForCrosstalkSources_855393_53.stderr parsl.dataflow.dflow INFO: Task 3764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c088da5c38c8c6d718913d85e5d1a78b with result from task 3764 parsl.dataflow.dflow DEBUG: Task 3764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2028ab19-a252-4c76-b389-18de0df5fb8f_isrForCrosstalkSources_855442_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/2028ab19-a252-4c76-b389-18de0df5fb8f_isrForCrosstalkSources_855442_37.stderr parsl.dataflow.dflow INFO: Task 3798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379a68878cb9c0d54dbb220878ec20ed with result from task 3798 parsl.dataflow.dflow DEBUG: Task 3798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4c18d2ab-1d33-4b07-917a-16bf3ce487e2_isrForCrosstalkSources_855388_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/4c18d2ab-1d33-4b07-917a-16bf3ce487e2_isrForCrosstalkSources_855388_31.stderr parsl.dataflow.dflow INFO: Task 3582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6e948ae8b9440f6022e07e5135af956 with result from task 3582 parsl.dataflow.dflow DEBUG: Task 3582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3588ec91-e32a-45fe-a350-76619c1a2af7_isrForCrosstalkSources_855391_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/3588ec91-e32a-45fe-a350-76619c1a2af7_isrForCrosstalkSources_855391_21.stderr parsl.dataflow.dflow INFO: Task 3579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc05e5c0cc8f2c507b26b982c8ffa238 with result from task 3579 parsl.dataflow.dflow DEBUG: Task 3579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8818c5cb-3daf-4b77-865e-cd6ce9817f08_isrForCrosstalkSources_855393_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/8818c5cb-3daf-4b77-865e-cd6ce9817f08_isrForCrosstalkSources_855393_54.stderr parsl.dataflow.dflow INFO: Task 3523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a9667e3bfc6759a5c914b6dd6f131fd with result from task 3523 parsl.dataflow.dflow DEBUG: Task 3523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f77b44e1-8895-47d7-b579-a4a6d9611a6a_isrForCrosstalkSources_855384_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f77b44e1-8895-47d7-b579-a4a6d9611a6a_isrForCrosstalkSources_855384_15.stderr parsl.dataflow.dflow INFO: Task 3520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 876ff2bfbecab3c276b25396b939fb95 with result from task 3520 parsl.dataflow.dflow DEBUG: Task 3520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0e60acd6-10ad-42c3-91eb-19969e15c325_isrForCrosstalkSources_855435_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0e60acd6-10ad-42c3-91eb-19969e15c325_isrForCrosstalkSources_855435_43.stderr parsl.dataflow.dflow INFO: Task 3622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d16d028d1f88ef2ac6115fd08ed18c12 with result from task 3622 parsl.dataflow.dflow DEBUG: Task 3622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5bdf7576-c584-4bbd-81f1-c195b6eafe2f_isrForCrosstalkSources_855458_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/5bdf7576-c584-4bbd-81f1-c195b6eafe2f_isrForCrosstalkSources_855458_2.stderr parsl.dataflow.dflow INFO: Task 3381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19862d6e8e55d6d1536f976ef0c1a812 with result from task 3381 parsl.dataflow.dflow DEBUG: Task 3381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/617889d0-910d-4899-a81e-0e8e5f2310de_isrForCrosstalkSources_855442_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/617889d0-910d-4899-a81e-0e8e5f2310de_isrForCrosstalkSources_855442_6.stderr parsl.dataflow.dflow INFO: Task 3568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 158598cf6f7e60e4f838e186e653d794 with result from task 3568 parsl.dataflow.dflow DEBUG: Task 3568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/07b204cc-c189-4076-a9af-9bc46e10d701_isrForCrosstalkSources_855445_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/07b204cc-c189-4076-a9af-9bc46e10d701_isrForCrosstalkSources_855445_17.stderr parsl.dataflow.dflow INFO: Task 3615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b741b9774a9316b9d990d040daf8340d with result from task 3615 parsl.dataflow.dflow DEBUG: Task 3615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/94a0f5da-c10c-4885-8355-0684ae6217ac_isrForCrosstalkSources_855441_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/94a0f5da-c10c-4885-8355-0684ae6217ac_isrForCrosstalkSources_855441_17.stderr parsl.dataflow.dflow INFO: Task 3940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35676ba5548a5c65e9f2584d7cf884a8 with result from task 3940 parsl.dataflow.dflow DEBUG: Task 3940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f16b4c6a-4668-4702-bd61-4e41f477abd2_isrForCrosstalkSources_855379_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f16b4c6a-4668-4702-bd61-4e41f477abd2_isrForCrosstalkSources_855379_12.stderr parsl.dataflow.dflow INFO: Task 3560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4511ede14faf5dd7ef8b53138facfc0 with result from task 3560 parsl.dataflow.dflow DEBUG: Task 3560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6aa510fc-88d0-4821-9e9d-06a8e3b8195c_isrForCrosstalkSources_855401_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/6aa510fc-88d0-4821-9e9d-06a8e3b8195c_isrForCrosstalkSources_855401_7.stderr parsl.dataflow.dflow INFO: Task 3610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fad977d3240861e7b67316c20a69b9d1 with result from task 3610 parsl.dataflow.dflow DEBUG: Task 3610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/951ff6d7-ed44-4970-aa25-8a99e4269034_isrForCrosstalkSources_855451_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/951ff6d7-ed44-4970-aa25-8a99e4269034_isrForCrosstalkSources_855451_6.stderr parsl.dataflow.dflow INFO: Task 3829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b3b914d5d8f366a42e2eebc93954684 with result from task 3829 parsl.dataflow.dflow DEBUG: Task 3829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/09350d9d-37d0-461a-b799-44ddd6e2b05d_isrForCrosstalkSources_855447_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/09350d9d-37d0-461a-b799-44ddd6e2b05d_isrForCrosstalkSources_855447_4.stderr parsl.dataflow.dflow INFO: Task 3939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d830a6e092ff73f3950d82f7a1db011 with result from task 3939 parsl.dataflow.dflow DEBUG: Task 3939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/788b57e3-8262-401c-a52f-565dbac8e31d_isrForCrosstalkSources_855402_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/788b57e3-8262-401c-a52f-565dbac8e31d_isrForCrosstalkSources_855402_52.stderr parsl.dataflow.dflow INFO: Task 3696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac64945621ad8b06b2a40d02acd05fa with result from task 3696 parsl.dataflow.dflow DEBUG: Task 3696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/16489366-a461-4e72-90b8-cb749b8dea48_isrForCrosstalkSources_855416_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/16489366-a461-4e72-90b8-cb749b8dea48_isrForCrosstalkSources_855416_49.stderr parsl.dataflow.dflow INFO: Task 3780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5f37b16c77515de5fa35488bac0bf2a with result from task 3780 parsl.dataflow.dflow DEBUG: Task 3780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/30a69108-c795-4c15-ac81-88c4d7887f23_isrForCrosstalkSources_855406_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/30a69108-c795-4c15-ac81-88c4d7887f23_isrForCrosstalkSources_855406_51.stderr parsl.dataflow.dflow INFO: Task 3387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b3764023a10efc609b6a382e2a96869 with result from task 3387 parsl.dataflow.dflow DEBUG: Task 3387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/7c123490-d2b9-4c0b-b5f6-e3699becf56e_isrForCrosstalkSources_855407_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/7c123490-d2b9-4c0b-b5f6-e3699becf56e_isrForCrosstalkSources_855407_29.stderr parsl.dataflow.dflow INFO: Task 3566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7145ebc81e428ae3913ba23c219ccb with result from task 3566 parsl.dataflow.dflow DEBUG: Task 3566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7ad0a6e7-1c1e-48f6-97cf-95162de592c3_isrForCrosstalkSources_855379_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/7ad0a6e7-1c1e-48f6-97cf-95162de592c3_isrForCrosstalkSources_855379_28.stderr parsl.dataflow.dflow INFO: Task 3843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83cc91075b1dde6b2f1708a1e26a485 with result from task 3843 parsl.dataflow.dflow DEBUG: Task 3843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0320d206-926c-40ff-8f57-17585b21d9a0_isrForCrosstalkSources_855387_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0320d206-926c-40ff-8f57-17585b21d9a0_isrForCrosstalkSources_855387_42.stderr parsl.dataflow.dflow INFO: Task 3911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdc43cfba34c1d824841bb4548b08bc7 with result from task 3911 parsl.dataflow.dflow DEBUG: Task 3911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7e68914f-1918-4eff-befe-e6b76fd46b4f_isrForCrosstalkSources_855387_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7e68914f-1918-4eff-befe-e6b76fd46b4f_isrForCrosstalkSources_855387_33.stderr parsl.dataflow.dflow INFO: Task 3786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b86fd16817e133ce9527bca53e2b0ac with result from task 3786 parsl.dataflow.dflow DEBUG: Task 3786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/3a640860-b691-44dc-a624-4073f4b23904_isrForCrosstalkSources_855459_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/3a640860-b691-44dc-a624-4073f4b23904_isrForCrosstalkSources_855459_18.stderr parsl.dataflow.dflow INFO: Task 3973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09fc10bef0b34c8581e726db5f46db46 with result from task 3973 parsl.dataflow.dflow DEBUG: Task 3973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/96978879-0f53-43cd-b2a8-ebe50541465d_isrForCrosstalkSources_855387_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/96978879-0f53-43cd-b2a8-ebe50541465d_isrForCrosstalkSources_855387_34.stderr parsl.dataflow.dflow INFO: Task 3657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6171cdedae81e629649faa91b5475f6 with result from task 3657 parsl.dataflow.dflow DEBUG: Task 3657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9bec570f-aa8d-4332-b226-ed7c4208c339_isrForCrosstalkSources_855441_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/9bec570f-aa8d-4332-b226-ed7c4208c339_isrForCrosstalkSources_855441_15.stderr parsl.dataflow.dflow INFO: Task 3705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 430520341d01ab757f4b6ef98a32e598 with result from task 3705 parsl.dataflow.dflow DEBUG: Task 3705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3bfd0259-b5e5-4d82-98e6-3b0b40fdf1f5_isrForCrosstalkSources_855433_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/3bfd0259-b5e5-4d82-98e6-3b0b40fdf1f5_isrForCrosstalkSources_855433_24.stderr parsl.dataflow.dflow INFO: Task 3702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c5f190815c9a3620f672f11a86e21cd with result from task 3702 parsl.dataflow.dflow DEBUG: Task 3702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/0eca8f25-9839-45e9-ab70-4c817da2aeb8_isrForCrosstalkSources_855392_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/0eca8f25-9839-45e9-ab70-4c817da2aeb8_isrForCrosstalkSources_855392_35.stderr parsl.dataflow.dflow INFO: Task 3758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0288e9d240cee722634af1a486c3b0a with result from task 3758 parsl.dataflow.dflow DEBUG: Task 3758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f5a2027a-54e5-47fa-b725-b7d1af199f11_isrForCrosstalkSources_855379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f5a2027a-54e5-47fa-b725-b7d1af199f11_isrForCrosstalkSources_855379_2.stderr parsl.dataflow.dflow INFO: Task 3577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9df794f2a47b4011d555471dff04e934 with result from task 3577 parsl.dataflow.dflow DEBUG: Task 3577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b2582232-2a09-48f6-8d8e-f7910c951302_isrForCrosstalkSources_855433_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b2582232-2a09-48f6-8d8e-f7910c951302_isrForCrosstalkSources_855433_29.stderr parsl.dataflow.dflow INFO: Task 3581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90c1b1e39887b3a60f55a8519e0b0992 with result from task 3581 parsl.dataflow.dflow DEBUG: Task 3581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c3b3cf12-5cfa-4b9f-9515-9a94fcc57dc2_isrForCrosstalkSources_855435_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/c3b3cf12-5cfa-4b9f-9515-9a94fcc57dc2_isrForCrosstalkSources_855435_10.stderr parsl.dataflow.dflow INFO: Task 3848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8553b5ddaa0eae0f378b5640de34634 with result from task 3848 parsl.dataflow.dflow DEBUG: Task 3848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a1e38164-97d9-43ab-b233-953365124433_isrForCrosstalkSources_855448_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a1e38164-97d9-43ab-b233-953365124433_isrForCrosstalkSources_855448_9.stderr parsl.dataflow.dflow INFO: Task 3738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d075057fd3a58c595bf41ac3fea70ab3 with result from task 3738 parsl.dataflow.dflow DEBUG: Task 3738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c2b9261e-6131-4b40-b8fe-aa91a383047f_isrForCrosstalkSources_855420_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/c2b9261e-6131-4b40-b8fe-aa91a383047f_isrForCrosstalkSources_855420_15.stderr parsl.dataflow.dflow INFO: Task 3616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af9762e2e3f02e2ed267a90f5fb1b431 with result from task 3616 parsl.dataflow.dflow DEBUG: Task 3616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/964117aa-3d34-4831-9b72-95f633bd9a50_isrForCrosstalkSources_855380_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/964117aa-3d34-4831-9b72-95f633bd9a50_isrForCrosstalkSources_855380_49.stderr parsl.dataflow.dflow INFO: Task 3821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8192d8ebbfb23601dd77b8de75bd5505 with result from task 3821 parsl.dataflow.dflow DEBUG: Task 3821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b714aea0-5899-4c5d-a664-bcedf0f37dc5_isrForCrosstalkSources_855453_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b714aea0-5899-4c5d-a664-bcedf0f37dc5_isrForCrosstalkSources_855453_52.stderr parsl.dataflow.dflow INFO: Task 3623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa4352be8382c982086bcc12e048868a with result from task 3623 parsl.dataflow.dflow DEBUG: Task 3623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a8455573-1f91-4524-a812-21d0da8532ea_isrForCrosstalkSources_855397_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a8455573-1f91-4524-a812-21d0da8532ea_isrForCrosstalkSources_855397_11.stderr parsl.dataflow.dflow INFO: Task 3486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1037cc1fc6203348f490a1c4897a8862 with result from task 3486 parsl.dataflow.dflow DEBUG: Task 3486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/632c8718-46a1-4022-ad56-653d8f835739_isrForCrosstalkSources_855447_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/632c8718-46a1-4022-ad56-653d8f835739_isrForCrosstalkSources_855447_55.stderr parsl.dataflow.dflow INFO: Task 3873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9a43a8751765defbbcaf65c0b6601c with result from task 3873 parsl.dataflow.dflow DEBUG: Task 3873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/32cb854d-a9e0-4d2d-bf2c-7e770c3b7902_isrForCrosstalkSources_855422_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/32cb854d-a9e0-4d2d-bf2c-7e770c3b7902_isrForCrosstalkSources_855422_50.stderr parsl.dataflow.dflow INFO: Task 3605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fd3f675c654b701074b35162ab8d938 with result from task 3605 parsl.dataflow.dflow DEBUG: Task 3605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/29a856ae-d1d3-4412-8fd0-95c7eb513b0e_isrForCrosstalkSources_855439_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/29a856ae-d1d3-4412-8fd0-95c7eb513b0e_isrForCrosstalkSources_855439_42.stderr parsl.dataflow.dflow INFO: Task 3685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43a8441356bd13f5c97377fe756e52fc with result from task 3685 parsl.dataflow.dflow DEBUG: Task 3685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/809d1d59-be25-4cd3-a4bd-b136451b76d9_isrForCrosstalkSources_855418_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/809d1d59-be25-4cd3-a4bd-b136451b76d9_isrForCrosstalkSources_855418_61.stderr parsl.dataflow.dflow INFO: Task 3555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf4dbba610dd8e08b5a0890766ba3e52 with result from task 3555 parsl.dataflow.dflow DEBUG: Task 3555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cb313bb2-983a-4ad2-99b6-17d8b25f838e_isrForCrosstalkSources_855447_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cb313bb2-983a-4ad2-99b6-17d8b25f838e_isrForCrosstalkSources_855447_17.stderr parsl.dataflow.dflow INFO: Task 3915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1c9e0810b078e953a831a3acf839a95 with result from task 3915 parsl.dataflow.dflow DEBUG: Task 3915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/43d716fd-9df4-4810-8bf6-b73a1293a2ef_isrForCrosstalkSources_855458_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/43d716fd-9df4-4810-8bf6-b73a1293a2ef_isrForCrosstalkSources_855458_29.stderr parsl.dataflow.dflow INFO: Task 3883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4d4e8c1d92e70ec8d3d1846a6fd72dc with result from task 3883 parsl.dataflow.dflow DEBUG: Task 3883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/f86d8be2-be64-4d63-b3c0-135c97c3f75e_isrForCrosstalkSources_855413_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/f86d8be2-be64-4d63-b3c0-135c97c3f75e_isrForCrosstalkSources_855413_4.stderr parsl.dataflow.dflow INFO: Task 3866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb82e0f55e525f64d9bb76e67ccdb0b4 with result from task 3866 parsl.dataflow.dflow DEBUG: Task 3866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/def4886c-cd72-4832-9981-7e1746232089_isrForCrosstalkSources_855428_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/def4886c-cd72-4832-9981-7e1746232089_isrForCrosstalkSources_855428_46.stderr parsl.dataflow.dflow INFO: Task 3927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75c058ffd60e52b721ab66b546303317 with result from task 3927 parsl.dataflow.dflow DEBUG: Task 3927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/5370f7a8-26c5-4695-8375-10377cc9af19_isrForCrosstalkSources_855426_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/5370f7a8-26c5-4695-8375-10377cc9af19_isrForCrosstalkSources_855426_51.stderr parsl.dataflow.dflow INFO: Task 3972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 805186fc033bba8f5a24c1d4666b4822 with result from task 3972 parsl.dataflow.dflow DEBUG: Task 3972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a4bf2cda-7e05-4861-9e9b-aabcf20121b2_isrForCrosstalkSources_855422_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/a4bf2cda-7e05-4861-9e9b-aabcf20121b2_isrForCrosstalkSources_855422_51.stderr parsl.dataflow.dflow INFO: Task 3665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9714be20e5c68587687cdbf9f5955b6b with result from task 3665 parsl.dataflow.dflow DEBUG: Task 3665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/100a21d6-8233-43e4-8bd5-4178022cbdee_isrForCrosstalkSources_855427_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/100a21d6-8233-43e4-8bd5-4178022cbdee_isrForCrosstalkSources_855427_4.stderr 2024-03-09 14:55:10 proc_lsst.multi:170 [INFO] [multi] job_id 16964300 proc_lsst.multi INFO: [multi] job_id 16964300 2024-03-09 14:55:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 210 2024-03-09 14:55:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964300 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964300 parsl.executors.status_handling DEBUG: Launched block 265 on executor multi with job ID 16964300 parsl.executors.status_handling INFO: Allocated block ID 266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-266 2024-03-09 14:55:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 2024-03-09 14:55:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 210 2024-03-09 14:55:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02477c60fc3ec478012f4534c8f3cafc with result from task 3460 parsl.dataflow.dflow DEBUG: Task 3460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2deb78d0-1f66-4fcc-bbf8-b44cb7e5d958_isrForCrosstalkSources_855401_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2deb78d0-1f66-4fcc-bbf8-b44cb7e5d958_isrForCrosstalkSources_855401_35.stderr parsl.dataflow.dflow INFO: Task 3907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a66197f5903db430ca32a8072a85d2 with result from task 3907 parsl.dataflow.dflow DEBUG: Task 3907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/acb8c892-79f5-4071-931a-45503b8fdfa1_isrForCrosstalkSources_855445_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/acb8c892-79f5-4071-931a-45503b8fdfa1_isrForCrosstalkSources_855445_33.stderr parsl.dataflow.dflow INFO: Task 3912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32320e7dc8e516bf26cfdb72313771e9 with result from task 3912 parsl.dataflow.dflow DEBUG: Task 3912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c78fa6ae-5ad2-4e19-896a-0a4a156b726f_isrForCrosstalkSources_855427_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c78fa6ae-5ad2-4e19-896a-0a4a156b726f_isrForCrosstalkSources_855427_38.stderr parsl.dataflow.dflow INFO: Task 3837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e10ca46770b0388c532cbcec171d8e39 with result from task 3837 parsl.dataflow.dflow DEBUG: Task 3837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f44f06bf-54bb-4591-ac43-d5ab8585b48c_isrForCrosstalkSources_855379_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f44f06bf-54bb-4591-ac43-d5ab8585b48c_isrForCrosstalkSources_855379_40.stderr parsl.dataflow.dflow INFO: Task 3975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99a837bddf62243a8e45db64c90ff1eb with result from task 3975 parsl.dataflow.dflow DEBUG: Task 3975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4cb62395-ab44-4e85-8e83-100b7604f0b9_isrForCrosstalkSources_855392_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4cb62395-ab44-4e85-8e83-100b7604f0b9_isrForCrosstalkSources_855392_4.stderr parsl.dataflow.dflow INFO: Task 3909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bab9c9b6a1ddea47976a736da14de9b2 with result from task 3909 parsl.dataflow.dflow DEBUG: Task 3909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fffe5ddf-302d-481a-9cab-f4babc8c2249_isrForCrosstalkSources_855439_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/fffe5ddf-302d-481a-9cab-f4babc8c2249_isrForCrosstalkSources_855439_39.stderr parsl.dataflow.dflow INFO: Task 3872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be0957f412dd5e30dac292aac1c7369c with result from task 3872 parsl.dataflow.dflow DEBUG: Task 3872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/eb132a83-42cf-43d7-afeb-63d9ede0c37e_isrForCrosstalkSources_855417_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/eb132a83-42cf-43d7-afeb-63d9ede0c37e_isrForCrosstalkSources_855417_34.stderr parsl.dataflow.dflow INFO: Task 3787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd7e8a052b9dd04502989c6be111e695 with result from task 3787 parsl.dataflow.dflow DEBUG: Task 3787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bebd6491-40ac-4b9f-acce-9984648602cd_isrForCrosstalkSources_855427_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/bebd6491-40ac-4b9f-acce-9984648602cd_isrForCrosstalkSources_855427_55.stderr parsl.dataflow.dflow INFO: Task 3966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5539970ccf2653011dd1b74bd25fa6cf with result from task 3966 parsl.dataflow.dflow DEBUG: Task 3966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c2ccc096-3b4c-4874-9a07-c288bd5ea0f7_isrForCrosstalkSources_855408_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c2ccc096-3b4c-4874-9a07-c288bd5ea0f7_isrForCrosstalkSources_855408_21.stderr parsl.dataflow.dflow INFO: Task 3545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1d521f40a1bede78b30a318288d4b3a with result from task 3545 parsl.dataflow.dflow DEBUG: Task 3545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/95870a49-d685-4d7c-9156-d19edc8fede3_isrForCrosstalkSources_855435_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/95870a49-d685-4d7c-9156-d19edc8fede3_isrForCrosstalkSources_855435_14.stderr parsl.dataflow.dflow INFO: Task 3793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ec34e7dfefb9ceaa99fade11d6800c4 with result from task 3793 parsl.dataflow.dflow DEBUG: Task 3793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ce39d21-6640-40ac-b73e-d98592f25aab_isrForCrosstalkSources_855383_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ce39d21-6640-40ac-b73e-d98592f25aab_isrForCrosstalkSources_855383_61.stderr parsl.dataflow.dflow INFO: Task 3744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 047323f6a756cc69a3b0c321c4d53717 with result from task 3744 parsl.dataflow.dflow DEBUG: Task 3744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f9fd9eec-a6a5-49e8-aba5-786025ea8f55_isrForCrosstalkSources_855387_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/f9fd9eec-a6a5-49e8-aba5-786025ea8f55_isrForCrosstalkSources_855387_54.stderr parsl.dataflow.dflow INFO: Task 3732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f713da4a9e811d5e1e87fa99392ed95 with result from task 3732 parsl.dataflow.dflow DEBUG: Task 3732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e0ee264b-610f-472f-8f1a-abcb8c8ba4dd_isrForCrosstalkSources_855430_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/e0ee264b-610f-472f-8f1a-abcb8c8ba4dd_isrForCrosstalkSources_855430_6.stderr parsl.dataflow.dflow INFO: Task 3944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a671a8e132d5955ee51f91346d606ab with result from task 3944 parsl.dataflow.dflow DEBUG: Task 3944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/fb3ea7e2-084e-4824-a061-bfa61b0b22d1_isrForCrosstalkSources_855429_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/fb3ea7e2-084e-4824-a061-bfa61b0b22d1_isrForCrosstalkSources_855429_23.stderr parsl.dataflow.dflow INFO: Task 3503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd828b32b13731f0411b840c28964edf with result from task 3503 parsl.dataflow.dflow DEBUG: Task 3503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d236dcdb-dade-4bcb-a89b-f5235cd57a2d_isrForCrosstalkSources_855430_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d236dcdb-dade-4bcb-a89b-f5235cd57a2d_isrForCrosstalkSources_855430_43.stderr parsl.dataflow.dflow INFO: Task 3606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb2e81080cb2963e56a097af71084a16 with result from task 3606 parsl.dataflow.dflow DEBUG: Task 3606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2dd6bafe-8a81-4a50-a436-e3f31f3c0929_isrForCrosstalkSources_855410_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2dd6bafe-8a81-4a50-a436-e3f31f3c0929_isrForCrosstalkSources_855410_58.stderr parsl.dataflow.dflow INFO: Task 3583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 182914a35ac0c7540a78c39d12a2c326 with result from task 3583 parsl.dataflow.dflow DEBUG: Task 3583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/53f2857f-37f0-4cd3-8982-11012bd07448_isrForCrosstalkSources_855436_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/53f2857f-37f0-4cd3-8982-11012bd07448_isrForCrosstalkSources_855436_44.stderr parsl.dataflow.dflow INFO: Task 3658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5851a016220bb78daa373ec75e32bf67 with result from task 3658 parsl.dataflow.dflow DEBUG: Task 3658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9e7e049b-5ab0-4aee-a353-879b8874055e_isrForCrosstalkSources_855401_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9e7e049b-5ab0-4aee-a353-879b8874055e_isrForCrosstalkSources_855401_52.stderr parsl.dataflow.dflow INFO: Task 3775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76f4bf6899817b23c991356f3f3a5a21 with result from task 3775 parsl.dataflow.dflow DEBUG: Task 3775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b91eb5fc-cb97-4b05-b39b-95fece5d075f_isrForCrosstalkSources_855442_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/b91eb5fc-cb97-4b05-b39b-95fece5d075f_isrForCrosstalkSources_855442_42.stderr parsl.dataflow.dflow INFO: Task 3563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a31e57451dec7b7705180a17f1bba377 with result from task 3563 parsl.dataflow.dflow DEBUG: Task 3563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c252f74f-be97-44fa-8d34-238a6ed5b288_isrForCrosstalkSources_855427_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/c252f74f-be97-44fa-8d34-238a6ed5b288_isrForCrosstalkSources_855427_6.stderr parsl.dataflow.dflow INFO: Task 3660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5f32f2176e7e7b1ac6e392cc9face9 with result from task 3660 parsl.dataflow.dflow DEBUG: Task 3660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/52e6869f-45a3-4b32-b15e-4db63f7d0353_isrForCrosstalkSources_855384_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/52e6869f-45a3-4b32-b15e-4db63f7d0353_isrForCrosstalkSources_855384_38.stderr parsl.dataflow.dflow INFO: Task 2132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8916a6c5ab3b4113b9daeb3b626b6a9 with result from task 2132 parsl.dataflow.dflow DEBUG: Task 2132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/162f738d-3a9b-44bf-8978-e08f62aa10c8_isrForCrosstalkSources_855424_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/162f738d-3a9b-44bf-8978-e08f62aa10c8_isrForCrosstalkSources_855424_54.stderr parsl.dataflow.dflow INFO: Task 3598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d73a6c4e34f9140287c1ee41652ef11 with result from task 3598 parsl.dataflow.dflow DEBUG: Task 3598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d367339f-46e3-4cdb-95c9-ea5bf38bf46c_isrForCrosstalkSources_855432_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d367339f-46e3-4cdb-95c9-ea5bf38bf46c_isrForCrosstalkSources_855432_37.stderr parsl.dataflow.dflow INFO: Task 3860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b00a89b9b89e4a4826b4a6f758c188ec with result from task 3860 parsl.dataflow.dflow DEBUG: Task 3860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/86637b6a-d7d0-420a-a4c6-48d810b3518a_isrForCrosstalkSources_855421_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/86637b6a-d7d0-420a-a4c6-48d810b3518a_isrForCrosstalkSources_855421_9.stderr parsl.dataflow.dflow INFO: Task 3630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7df1e4d7967ea05caf81bf7cf113d6b9 with result from task 3630 parsl.dataflow.dflow DEBUG: Task 3630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c8e393dc-da15-4b63-a472-287586098114_isrForCrosstalkSources_855418_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/c8e393dc-da15-4b63-a472-287586098114_isrForCrosstalkSources_855418_38.stderr parsl.dataflow.dflow INFO: Task 3724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc456167cf2dfbabc77909a854e29808 with result from task 3724 parsl.dataflow.dflow DEBUG: Task 3724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/769e8593-9407-4164-8975-bdd9838dd333_isrForCrosstalkSources_855397_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/769e8593-9407-4164-8975-bdd9838dd333_isrForCrosstalkSources_855397_3.stderr parsl.dataflow.dflow INFO: Task 3845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e47531fc217d0f4e9dafed917d73ab0 with result from task 3845 parsl.dataflow.dflow DEBUG: Task 3845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbcf1dbd-f46b-4742-b5d0-b89a7ece2764_isrForCrosstalkSources_855431_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/dbcf1dbd-f46b-4742-b5d0-b89a7ece2764_isrForCrosstalkSources_855431_6.stderr parsl.dataflow.dflow INFO: Task 3971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9de25ed3198ec678d63a4b23ac252ee9 with result from task 3971 parsl.dataflow.dflow DEBUG: Task 3971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/71c037d1-acd1-4c43-b4f2-8c21e26902ae_isrForCrosstalkSources_855393_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/71c037d1-acd1-4c43-b4f2-8c21e26902ae_isrForCrosstalkSources_855393_41.stderr parsl.dataflow.dflow INFO: Task 3676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25304cbe876261fb6fb3ac263a95fa43 with result from task 3676 parsl.dataflow.dflow DEBUG: Task 3676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3ed9d75d-9f36-4b53-a46a-af579e2902a5_isrForCrosstalkSources_855414_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3ed9d75d-9f36-4b53-a46a-af579e2902a5_isrForCrosstalkSources_855414_7.stderr parsl.dataflow.dflow INFO: Task 3745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b3020f43ed89070e513407f190bf98 with result from task 3745 parsl.dataflow.dflow DEBUG: Task 3745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/242d89e7-fbf2-47cf-af04-980f457ba055_isrForCrosstalkSources_855429_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/242d89e7-fbf2-47cf-af04-980f457ba055_isrForCrosstalkSources_855429_6.stderr parsl.dataflow.dflow INFO: Task 3585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62b1d68a523d7e268d0ea01ff51f33f5 with result from task 3585 parsl.dataflow.dflow DEBUG: Task 3585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/81c7a4c0-94e1-4a09-8a92-d16545778a93_isrForCrosstalkSources_855433_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/81c7a4c0-94e1-4a09-8a92-d16545778a93_isrForCrosstalkSources_855433_3.stderr parsl.dataflow.dflow INFO: Task 3882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68fca4f7fc1a887fba73aeaf6dc1e456 with result from task 3882 parsl.dataflow.dflow DEBUG: Task 3882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/55b22387-f72a-4914-a004-f674c13399fb_isrForCrosstalkSources_855436_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/55b22387-f72a-4914-a004-f674c13399fb_isrForCrosstalkSources_855436_43.stderr parsl.dataflow.dflow INFO: Task 3914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aef50a8f2e29734ce0ae71a934ad1836 with result from task 3914 parsl.dataflow.dflow DEBUG: Task 3914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c7847335-0535-4335-95ec-c6fe2b23c99c_isrForCrosstalkSources_855431_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c7847335-0535-4335-95ec-c6fe2b23c99c_isrForCrosstalkSources_855431_21.stderr parsl.dataflow.dflow INFO: Task 3783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575b79d991849d1f6c08e62505726937 with result from task 3783 parsl.dataflow.dflow DEBUG: Task 3783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/aae7a606-71a1-4a8f-bfc2-3295d6010b43_isrForCrosstalkSources_855428_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/aae7a606-71a1-4a8f-bfc2-3295d6010b43_isrForCrosstalkSources_855428_11.stderr parsl.dataflow.dflow INFO: Task 3546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d0120bea582f55b8d2a80f856ee306f with result from task 3546 parsl.dataflow.dflow DEBUG: Task 3546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2351cf1d-98db-4152-8904-b7d13e794f4f_isrForCrosstalkSources_855417_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2351cf1d-98db-4152-8904-b7d13e794f4f_isrForCrosstalkSources_855417_23.stderr parsl.dataflow.dflow INFO: Task 3968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29099d9babb91388b180acacc0aaee46 with result from task 3968 parsl.dataflow.dflow DEBUG: Task 3968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8b3f2487-e26f-4ec2-930b-d18af10aeeaa_isrForCrosstalkSources_855442_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8b3f2487-e26f-4ec2-930b-d18af10aeeaa_isrForCrosstalkSources_855442_24.stderr parsl.dataflow.dflow INFO: Task 3816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 411377895562c073c4596d07f399c6a6 with result from task 3816 parsl.dataflow.dflow DEBUG: Task 3816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20eb1eec-5638-4781-af19-aa2038dba493_isrForCrosstalkSources_855418_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/20eb1eec-5638-4781-af19-aa2038dba493_isrForCrosstalkSources_855418_50.stderr parsl.dataflow.dflow INFO: Task 3649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbc16c87b619dd9006e1b817bf96eb98 with result from task 3649 parsl.dataflow.dflow DEBUG: Task 3649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b9857c0-c115-4896-a496-6ddc531b9808_isrForCrosstalkSources_855459_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b9857c0-c115-4896-a496-6ddc531b9808_isrForCrosstalkSources_855459_53.stderr parsl.dataflow.dflow INFO: Task 3964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 251f378a1c86266b4eb45a7ac14c878c with result from task 3964 parsl.dataflow.dflow DEBUG: Task 3964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/166cee37-11b9-4d0e-b4e4-7a2724fe7c47_isrForCrosstalkSources_855398_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/166cee37-11b9-4d0e-b4e4-7a2724fe7c47_isrForCrosstalkSources_855398_21.stderr parsl.dataflow.dflow INFO: Task 3895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b506450ad87410936ef453f25a970769 with result from task 3895 parsl.dataflow.dflow DEBUG: Task 3895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e5463549-345b-4749-9e40-94afe26c287b_isrForCrosstalkSources_855434_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/e5463549-345b-4749-9e40-94afe26c287b_isrForCrosstalkSources_855434_2.stderr parsl.dataflow.dflow INFO: Task 3980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6a6ed9969d505a1192607a1f2439f61 with result from task 3980 parsl.dataflow.dflow DEBUG: Task 3980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8f81c688-6b99-4cb1-8c53-d91f5165ff6e_isrForCrosstalkSources_855400_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/8f81c688-6b99-4cb1-8c53-d91f5165ff6e_isrForCrosstalkSources_855400_9.stderr parsl.dataflow.dflow INFO: Task 3751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 855f14479cf9a024762a8496caab9cae with result from task 3751 parsl.dataflow.dflow DEBUG: Task 3751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/36b9270b-0b00-4d7b-bf62-540e5714f30f_isrForCrosstalkSources_855431_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/36b9270b-0b00-4d7b-bf62-540e5714f30f_isrForCrosstalkSources_855431_59.stderr parsl.dataflow.dflow INFO: Task 3588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25a72272d662ffaaa59a6f73109d5a7f with result from task 3588 parsl.dataflow.dflow DEBUG: Task 3588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67ecb9fa-6d5d-43a8-a789-4c28949abdbf_isrForCrosstalkSources_855393_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/67ecb9fa-6d5d-43a8-a789-4c28949abdbf_isrForCrosstalkSources_855393_42.stderr parsl.dataflow.dflow INFO: Task 3842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa704e41ea1001ffc2bccd25f56fb608 with result from task 3842 parsl.dataflow.dflow DEBUG: Task 3842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e96774b4-c4fe-4e81-9e0d-81405906ed48_isrForCrosstalkSources_855420_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e96774b4-c4fe-4e81-9e0d-81405906ed48_isrForCrosstalkSources_855420_51.stderr parsl.dataflow.dflow INFO: Task 3427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd3e361d642925cfd602c7b018ef8b1 with result from task 3427 parsl.dataflow.dflow DEBUG: Task 3427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/52b4a8d5-4874-4169-b32a-f25652b34b9e_isrForCrosstalkSources_855411_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/52b4a8d5-4874-4169-b32a-f25652b34b9e_isrForCrosstalkSources_855411_62.stderr parsl.dataflow.dflow INFO: Task 3978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb4b45561618e05577d6cccbb5727a62 with result from task 3978 parsl.dataflow.dflow DEBUG: Task 3978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/076b6598-1f64-481f-8a27-af285d49a416_isrForCrosstalkSources_855392_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/076b6598-1f64-481f-8a27-af285d49a416_isrForCrosstalkSources_855392_26.stderr parsl.dataflow.dflow INFO: Task 3847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e689675670068cb8413142b78cebdffc with result from task 3847 parsl.dataflow.dflow DEBUG: Task 3847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/21facd6d-4ba2-46ca-9b31-9cdca7a62b5b_isrForCrosstalkSources_855436_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/21facd6d-4ba2-46ca-9b31-9cdca7a62b5b_isrForCrosstalkSources_855436_42.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6014bebea579b260b92187b7baf8efb with result from task 3864 parsl.dataflow.dflow DEBUG: Task 3864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b738ac9d-c7b0-423d-8a3c-69dead1ad002_isrForCrosstalkSources_855453_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b738ac9d-c7b0-423d-8a3c-69dead1ad002_isrForCrosstalkSources_855453_61.stderr parsl.dataflow.dflow INFO: Task 3734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ee3ccb953c4ea60224f6de4ee93a6ce with result from task 3734 parsl.dataflow.dflow DEBUG: Task 3734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a8106ae8-255d-4e0c-b4ff-7260a627ee17_isrForCrosstalkSources_855443_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a8106ae8-255d-4e0c-b4ff-7260a627ee17_isrForCrosstalkSources_855443_1.stderr parsl.dataflow.dflow INFO: Task 3840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea525f878b3d1657140beec0f95666b8 with result from task 3840 parsl.dataflow.dflow DEBUG: Task 3840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a00ae263-0f39-4371-9ab1-58d2289cc5c6_isrForCrosstalkSources_855411_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a00ae263-0f39-4371-9ab1-58d2289cc5c6_isrForCrosstalkSources_855411_43.stderr parsl.dataflow.dflow INFO: Task 3736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9a7f89dedacf4ce31ceefe1a3863057 with result from task 3736 parsl.dataflow.dflow DEBUG: Task 3736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4278d7c9-a5f5-4a98-9137-119a3b5a09de_isrForCrosstalkSources_855413_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/4278d7c9-a5f5-4a98-9137-119a3b5a09de_isrForCrosstalkSources_855413_7.stderr parsl.dataflow.dflow INFO: Task 3928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db14dd5063933dd363a4f201baaae9c4 with result from task 3928 parsl.dataflow.dflow DEBUG: Task 3928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/cfa4f8db-2648-47d6-860a-778aed1ca4f2_isrForCrosstalkSources_855416_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/cfa4f8db-2648-47d6-860a-778aed1ca4f2_isrForCrosstalkSources_855416_11.stderr parsl.dataflow.dflow INFO: Task 3956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15a3b9a67cb60bb629eaaa635c3fb90f with result from task 3956 parsl.dataflow.dflow DEBUG: Task 3956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa4289db-8470-48e5-9438-1d59b77f9d34_isrForCrosstalkSources_855408_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/aa4289db-8470-48e5-9438-1d59b77f9d34_isrForCrosstalkSources_855408_40.stderr parsl.dataflow.dflow INFO: Task 3946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdcb138758358bae543a017c3fb00ad7 with result from task 3946 parsl.dataflow.dflow DEBUG: Task 3946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e85ab6f8-53cb-4974-a7ff-9ba2c55d985e_isrForCrosstalkSources_855422_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e85ab6f8-53cb-4974-a7ff-9ba2c55d985e_isrForCrosstalkSources_855422_43.stderr parsl.dataflow.dflow INFO: Task 3905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65a6342e13bc3aba49441b1a6d9b26b8 with result from task 3905 parsl.dataflow.dflow DEBUG: Task 3905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/2eb7d098-faf3-4db3-8100-e29a2feef5a6_isrForCrosstalkSources_855440_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/2eb7d098-faf3-4db3-8100-e29a2feef5a6_isrForCrosstalkSources_855440_22.stderr parsl.dataflow.dflow INFO: Task 3932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 169eb70b4fd2c9978a86c0a8b70863ed with result from task 3932 parsl.dataflow.dflow DEBUG: Task 3932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a7bcc0a3-4ba9-4791-b3e9-0e54b292a73b_isrForCrosstalkSources_855416_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a7bcc0a3-4ba9-4791-b3e9-0e54b292a73b_isrForCrosstalkSources_855416_16.stderr parsl.dataflow.dflow INFO: Task 3884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 984b00ed3805bc462a4ff97c33b4e01f with result from task 3884 parsl.dataflow.dflow DEBUG: Task 3884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/0a2f38c1-6ab9-4c14-9d46-ea4ba7738214_isrForCrosstalkSources_855423_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/0a2f38c1-6ab9-4c14-9d46-ea4ba7738214_isrForCrosstalkSources_855423_16.stderr parsl.dataflow.dflow INFO: Task 3913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a8ad7becdc5986963b0543abdf54bf7 with result from task 3913 parsl.dataflow.dflow DEBUG: Task 3913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a9ad1ebb-315f-4508-9f7c-0a3791db3b06_isrForCrosstalkSources_855393_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/a9ad1ebb-315f-4508-9f7c-0a3791db3b06_isrForCrosstalkSources_855393_5.stderr parsl.dataflow.dflow INFO: Task 3962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a7b0560dbcfa4bf2d95702ddbc0581e with result from task 3962 parsl.dataflow.dflow DEBUG: Task 3962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/82ccee9b-d29e-4e9e-a985-d4001885ef91_isrForCrosstalkSources_855408_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/82ccee9b-d29e-4e9e-a985-d4001885ef91_isrForCrosstalkSources_855408_17.stderr parsl.dataflow.dflow INFO: Task 3935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd8966b8fb38a45e597889562975d60 with result from task 3935 parsl.dataflow.dflow DEBUG: Task 3935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7809eb10-57e8-43d2-b000-5e004ca04553_isrForCrosstalkSources_855390_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7809eb10-57e8-43d2-b000-5e004ca04553_isrForCrosstalkSources_855390_10.stderr parsl.dataflow.dflow INFO: Task 3768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f3793fb52ed984752baac9306726aa0 with result from task 3768 parsl.dataflow.dflow DEBUG: Task 3768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/01f69a39-0f20-4196-a731-bfc0a717d2db_isrForCrosstalkSources_855436_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/01f69a39-0f20-4196-a731-bfc0a717d2db_isrForCrosstalkSources_855436_55.stderr parsl.dataflow.dflow INFO: Task 3941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c669460fc86b597e112af199eda6c4f8 with result from task 3941 parsl.dataflow.dflow DEBUG: Task 3941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/af5c812d-3140-4c78-8302-e5a584408037_isrForCrosstalkSources_855407_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/af5c812d-3140-4c78-8302-e5a584408037_isrForCrosstalkSources_855407_17.stderr parsl.dataflow.dflow INFO: Task 3576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea4b0930ce5f3a935ae3c1b9927f6f2b with result from task 3576 parsl.dataflow.dflow DEBUG: Task 3576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fe837c7-635e-4a7e-a026-29ab8d6786c9_isrForCrosstalkSources_855387_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/7fe837c7-635e-4a7e-a026-29ab8d6786c9_isrForCrosstalkSources_855387_6.stderr parsl.dataflow.dflow INFO: Task 3670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d49b47fffa0bd3ee0db4c691c1dd0d3 with result from task 3670 parsl.dataflow.dflow DEBUG: Task 3670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/02809a4b-1515-4766-9312-5508a9d521e9_isrForCrosstalkSources_855455_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/02809a4b-1515-4766-9312-5508a9d521e9_isrForCrosstalkSources_855455_31.stderr parsl.dataflow.dflow INFO: Task 3955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98671d16de36b1d55e6837888fab8141 with result from task 3955 parsl.dataflow.dflow DEBUG: Task 3955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9dc0b178-7169-4255-b8c2-33486c9b51aa_isrForCrosstalkSources_855426_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9dc0b178-7169-4255-b8c2-33486c9b51aa_isrForCrosstalkSources_855426_38.stderr parsl.dataflow.dflow INFO: Task 3662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 809b252a0ae820aa768f82029f7126d6 with result from task 3662 parsl.dataflow.dflow DEBUG: Task 3662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bf2be93a-7038-4a03-9597-ae2e7d558cc0_isrForCrosstalkSources_855434_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bf2be93a-7038-4a03-9597-ae2e7d558cc0_isrForCrosstalkSources_855434_3.stderr parsl.dataflow.dflow INFO: Task 3950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77a24ec98fd0ba1c1325b04e897fcd28 with result from task 3950 parsl.dataflow.dflow DEBUG: Task 3950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/172ec7a2-0b44-47e7-8dd8-aa42af8f2512_isrForCrosstalkSources_855459_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/172ec7a2-0b44-47e7-8dd8-aa42af8f2512_isrForCrosstalkSources_855459_48.stderr parsl.dataflow.dflow INFO: Task 3846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbd94534318c8f28655731abb14a24df with result from task 3846 parsl.dataflow.dflow DEBUG: Task 3846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/368457fa-8ba9-4abb-94cf-2a06a7c20bcd_isrForCrosstalkSources_855451_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/368457fa-8ba9-4abb-94cf-2a06a7c20bcd_isrForCrosstalkSources_855451_41.stderr 2024-03-09 14:55:12 proc_lsst.multi:170 [INFO] [multi] job_id 16964301 proc_lsst.multi INFO: [multi] job_id 16964301 2024-03-09 14:55:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-09 14:55:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964301 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964301 parsl.executors.status_handling DEBUG: Launched block 266 on executor multi with job ID 16964301 parsl.executors.status_handling INFO: Allocated block ID 267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-267 2024-03-09 14:55:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 2024-03-09 14:55:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-09 14:55:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 3924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef82255a8c140a97384361f41e8c1ac6 with result from task 3924 parsl.dataflow.dflow DEBUG: Task 3924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/682c8723-8207-4ed7-bec7-f8a7151fecf7_isrForCrosstalkSources_855428_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/682c8723-8207-4ed7-bec7-f8a7151fecf7_isrForCrosstalkSources_855428_60.stderr parsl.dataflow.dflow INFO: Task 3565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b83a9f80bbe2badac854acd088089de5 with result from task 3565 parsl.dataflow.dflow DEBUG: Task 3565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/69b8195e-cbb1-4d4c-99a4-686f9943f016_isrForCrosstalkSources_855448_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/69b8195e-cbb1-4d4c-99a4-686f9943f016_isrForCrosstalkSources_855448_38.stderr parsl.dataflow.dflow INFO: Task 3910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27e4f577506544e61f4191176d259c70 with result from task 3910 parsl.dataflow.dflow DEBUG: Task 3910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/512b9b0b-1395-42b8-913f-8a00fed14d84_isrForCrosstalkSources_855442_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/512b9b0b-1395-42b8-913f-8a00fed14d84_isrForCrosstalkSources_855442_60.stderr parsl.dataflow.dflow INFO: Task 3792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d74617ee7d7fd5512035598ed3481822 with result from task 3792 parsl.dataflow.dflow DEBUG: Task 3792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0819a946-de57-4596-bdd8-4155d3023363_isrForCrosstalkSources_855427_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/0819a946-de57-4596-bdd8-4155d3023363_isrForCrosstalkSources_855427_48.stderr parsl.dataflow.dflow INFO: Task 4026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f9f576d0ea42398cab6c08022c09a21 with result from task 4026 parsl.dataflow.dflow DEBUG: Task 4026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/88e93d8a-955a-437c-93f6-0596b373ca46_isrForCrosstalkSources_855447_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/88e93d8a-955a-437c-93f6-0596b373ca46_isrForCrosstalkSources_855447_35.stderr parsl.dataflow.dflow INFO: Task 3886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f0eb0e9b0572ddce676666bc887dbaf with result from task 3886 parsl.dataflow.dflow DEBUG: Task 3886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a5d3597d-110a-4e36-9e39-07d9fdedf0ee_isrForCrosstalkSources_855416_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/a5d3597d-110a-4e36-9e39-07d9fdedf0ee_isrForCrosstalkSources_855416_43.stderr parsl.dataflow.dflow INFO: Task 3970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def8e1216416e9615783a27df8da1431 with result from task 3970 parsl.dataflow.dflow DEBUG: Task 3970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/3fc607b1-499f-4b66-82f9-8e1dc59709a0_isrForCrosstalkSources_855439_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/3fc607b1-499f-4b66-82f9-8e1dc59709a0_isrForCrosstalkSources_855439_28.stderr parsl.dataflow.dflow INFO: Task 3790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe0999efd091330cb21cd9a0a8b148f with result from task 3790 parsl.dataflow.dflow DEBUG: Task 3790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2b4c7ed7-c700-40e5-ab7f-69054c89d8d0_isrForCrosstalkSources_855408_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/2b4c7ed7-c700-40e5-ab7f-69054c89d8d0_isrForCrosstalkSources_855408_52.stderr parsl.dataflow.dflow INFO: Task 3784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50b556f0d28f8bd1ab33de64053619a5 with result from task 3784 parsl.dataflow.dflow DEBUG: Task 3784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4b22bbfb-97df-4e59-9cc5-5736aa77ed0c_isrForCrosstalkSources_855403_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/4b22bbfb-97df-4e59-9cc5-5736aa77ed0c_isrForCrosstalkSources_855403_54.stderr parsl.dataflow.dflow INFO: Task 3795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03029e4cdaf260c5a2edcaafd75c6a71 with result from task 3795 parsl.dataflow.dflow DEBUG: Task 3795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/aba3bf78-b6f3-4a93-b48e-dfff5f420b65_isrForCrosstalkSources_855409_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/aba3bf78-b6f3-4a93-b48e-dfff5f420b65_isrForCrosstalkSources_855409_59.stderr parsl.dataflow.dflow INFO: Task 3869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 927103a9d5e9782a4bad542d5f7a273e with result from task 3869 parsl.dataflow.dflow DEBUG: Task 3869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/49067eda-a1db-4013-80a8-059f115661b8_isrForCrosstalkSources_855382_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/49067eda-a1db-4013-80a8-059f115661b8_isrForCrosstalkSources_855382_50.stderr parsl.dataflow.dflow INFO: Task 3919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1416e4299de1755d702737479e2dc5a9 with result from task 3919 parsl.dataflow.dflow DEBUG: Task 3919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/fb267b70-f9b1-4dbc-bdd0-d20bcba6f595_isrForCrosstalkSources_855444_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/fb267b70-f9b1-4dbc-bdd0-d20bcba6f595_isrForCrosstalkSources_855444_49.stderr parsl.dataflow.dflow INFO: Task 3765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1bbf644f50c66d3f0dbf4b42a0989c4 with result from task 3765 parsl.dataflow.dflow DEBUG: Task 3765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b5e498ab-0fb7-483f-a7af-b77f814b9d23_isrForCrosstalkSources_855415_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b5e498ab-0fb7-483f-a7af-b77f814b9d23_isrForCrosstalkSources_855415_55.stderr parsl.dataflow.dflow INFO: Task 3733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d6904bda48ee553a7f6ee799f2729d with result from task 3733 parsl.dataflow.dflow DEBUG: Task 3733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c4f0401b-7289-4aea-80e7-b48f3b90071b_isrForCrosstalkSources_855429_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c4f0401b-7289-4aea-80e7-b48f3b90071b_isrForCrosstalkSources_855429_62.stderr parsl.dataflow.dflow INFO: Task 3877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6afff33df111d5aa5982679185f952a6 with result from task 3877 parsl.dataflow.dflow DEBUG: Task 3877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24bf88a6-111b-4b94-be28-c45169c085dc_isrForCrosstalkSources_855379_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/24bf88a6-111b-4b94-be28-c45169c085dc_isrForCrosstalkSources_855379_42.stderr parsl.dataflow.dflow INFO: Task 3707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba65ec4f967d6655fe999e6b58415d0 with result from task 3707 parsl.dataflow.dflow DEBUG: Task 3707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/22a3f58b-811c-4a2c-9610-daf3aabc0cb5_isrForCrosstalkSources_855385_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/22a3f58b-811c-4a2c-9610-daf3aabc0cb5_isrForCrosstalkSources_855385_59.stderr parsl.dataflow.dflow INFO: Task 3891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb79ddaa60cae111a2b8e390840d8178 with result from task 3891 parsl.dataflow.dflow DEBUG: Task 3891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e29e7b27-ce34-4b53-92e0-130effe98810_isrForCrosstalkSources_855405_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e29e7b27-ce34-4b53-92e0-130effe98810_isrForCrosstalkSources_855405_52.stderr parsl.dataflow.dflow INFO: Task 3825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 373ed84cfff436a6268f583518d74b8e with result from task 3825 parsl.dataflow.dflow DEBUG: Task 3825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/36652692-738e-40bb-ae44-13897b6b5b20_isrForCrosstalkSources_855437_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/36652692-738e-40bb-ae44-13897b6b5b20_isrForCrosstalkSources_855437_44.stderr parsl.dataflow.dflow INFO: Task 3823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 901c878dad4863635590f4bca11661ca with result from task 3823 parsl.dataflow.dflow DEBUG: Task 3823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/960f2db5-de65-40a3-bc8c-ed80f8ecdba9_isrForCrosstalkSources_855415_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/960f2db5-de65-40a3-bc8c-ed80f8ecdba9_isrForCrosstalkSources_855415_52.stderr parsl.dataflow.dflow INFO: Task 3933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0f8ea312db0abc315d34ec8c0806cb with result from task 3933 parsl.dataflow.dflow DEBUG: Task 3933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c7c6a5e2-7b3c-447f-886e-fc742f2cbfcb_isrForCrosstalkSources_855401_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c7c6a5e2-7b3c-447f-886e-fc742f2cbfcb_isrForCrosstalkSources_855401_43.stderr parsl.dataflow.dflow INFO: Task 3714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51ef061fa3e5f1f4b38470e3edb41e83 with result from task 3714 parsl.dataflow.dflow DEBUG: Task 3714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/faf00e26-00ba-41a5-ba11-895d293b50d2_isrForCrosstalkSources_855412_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/faf00e26-00ba-41a5-ba11-895d293b50d2_isrForCrosstalkSources_855412_42.stderr parsl.dataflow.dflow INFO: Task 3892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8748dd091297832bd042367ddad97f3c with result from task 3892 parsl.dataflow.dflow DEBUG: Task 3892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/cfeef5d3-817c-420c-94ae-8d66fb1a5252_isrForCrosstalkSources_855428_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/cfeef5d3-817c-420c-94ae-8d66fb1a5252_isrForCrosstalkSources_855428_49.stderr parsl.dataflow.dflow INFO: Task 3739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8c429f3c1d7776b69dd6028de0b08f8 with result from task 3739 parsl.dataflow.dflow DEBUG: Task 3739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2fab58c4-f35d-4a91-a3c4-3ff704e00a1e_isrForCrosstalkSources_855411_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2fab58c4-f35d-4a91-a3c4-3ff704e00a1e_isrForCrosstalkSources_855411_55.stderr parsl.dataflow.dflow INFO: Task 4036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8788a9051b9ee849290feca804fecd57 with result from task 4036 parsl.dataflow.dflow DEBUG: Task 4036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8be988df-2a29-4a24-9ee6-8df0e2131565_isrForCrosstalkSources_855410_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8be988df-2a29-4a24-9ee6-8df0e2131565_isrForCrosstalkSources_855410_61.stderr parsl.dataflow.dflow INFO: Task 3648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ea53f8fca935530e24df03bde79b34e with result from task 3648 parsl.dataflow.dflow DEBUG: Task 3648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a593f637-45aa-468a-8db6-1363e4dea858_isrForCrosstalkSources_855438_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/a593f637-45aa-468a-8db6-1363e4dea858_isrForCrosstalkSources_855438_2.stderr parsl.dataflow.dflow INFO: Task 3858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9189d207bf08622004cba5674e3716bc with result from task 3858 parsl.dataflow.dflow DEBUG: Task 3858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/7deb292d-3cc1-4199-b472-be5eb08e54f7_isrForCrosstalkSources_855444_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/7deb292d-3cc1-4199-b472-be5eb08e54f7_isrForCrosstalkSources_855444_44.stderr parsl.dataflow.dflow INFO: Task 4047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95252adcfb6391d7a8995dd258923cd with result from task 4047 parsl.dataflow.dflow DEBUG: Task 4047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1739cb96-23c3-4007-9694-e327a68966b3_isrForCrosstalkSources_855385_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1739cb96-23c3-4007-9694-e327a68966b3_isrForCrosstalkSources_855385_5.stderr parsl.dataflow.dflow INFO: Task 3879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ccf1f36157f345954bd02caeffaeab with result from task 3879 parsl.dataflow.dflow DEBUG: Task 3879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/abc22dad-f572-4cf8-a18f-56fe02d27aaf_isrForCrosstalkSources_855414_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/abc22dad-f572-4cf8-a18f-56fe02d27aaf_isrForCrosstalkSources_855414_54.stderr parsl.dataflow.dflow INFO: Task 4029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80acb71d63a5273a7cd9957c4f2991a0 with result from task 4029 parsl.dataflow.dflow DEBUG: Task 4029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/431b91d8-da30-4e33-9e6b-0ae6b79174c4_isrForCrosstalkSources_855384_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/431b91d8-da30-4e33-9e6b-0ae6b79174c4_isrForCrosstalkSources_855384_25.stderr parsl.dataflow.dflow INFO: Task 3952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ede8f1e4fd8561e4d5bd5a824e37316 with result from task 3952 parsl.dataflow.dflow DEBUG: Task 3952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/e9fe238e-263e-4718-acfc-e93316e3ab67_isrForCrosstalkSources_855417_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/e9fe238e-263e-4718-acfc-e93316e3ab67_isrForCrosstalkSources_855417_14.stderr parsl.dataflow.dflow INFO: Task 3856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9468e45c5e9c2fe8725af39a676d017d with result from task 3856 parsl.dataflow.dflow DEBUG: Task 3856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0797670f-ea2e-42fe-bfa7-c2302e7fc9eb_isrForCrosstalkSources_855435_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/0797670f-ea2e-42fe-bfa7-c2302e7fc9eb_isrForCrosstalkSources_855435_49.stderr parsl.dataflow.dflow INFO: Task 3994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46784337868b998096478a70f62d496b with result from task 3994 parsl.dataflow.dflow DEBUG: Task 3994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee114502-24bb-4025-87a7-e0cd13566410_isrForCrosstalkSources_855443_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee114502-24bb-4025-87a7-e0cd13566410_isrForCrosstalkSources_855443_19.stderr parsl.dataflow.dflow INFO: Task 3896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e637d05535f7bcb25c35014bf989334 with result from task 3896 parsl.dataflow.dflow DEBUG: Task 3896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2d37c0a8-9fb9-42b1-b0f1-0137924d4d6c_isrForCrosstalkSources_855401_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/2d37c0a8-9fb9-42b1-b0f1-0137924d4d6c_isrForCrosstalkSources_855401_13.stderr parsl.dataflow.dflow INFO: Task 4030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 448ee219aab8bad2780e100b43eeb2c8 with result from task 4030 parsl.dataflow.dflow DEBUG: Task 4030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73c542aa-0542-4e52-ad58-d5c54d9af3c6_isrForCrosstalkSources_855444_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73c542aa-0542-4e52-ad58-d5c54d9af3c6_isrForCrosstalkSources_855444_27.stderr parsl.dataflow.dflow INFO: Task 4019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaf6ef1f96eff1e900c722a5b5a8b1a8 with result from task 4019 parsl.dataflow.dflow DEBUG: Task 4019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/12156752-df13-43c5-8980-fbcf4de647f6_isrForCrosstalkSources_855410_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/12156752-df13-43c5-8980-fbcf4de647f6_isrForCrosstalkSources_855410_39.stderr parsl.dataflow.dflow INFO: Task 3901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f05d0f2120062e8b0a706741ee0f9b4f with result from task 3901 parsl.dataflow.dflow DEBUG: Task 3901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/2a3855bd-ce06-4b14-b23a-dc0e54dd0a6b_isrForCrosstalkSources_855412_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/2a3855bd-ce06-4b14-b23a-dc0e54dd0a6b_isrForCrosstalkSources_855412_4.stderr parsl.dataflow.dflow INFO: Task 4014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aec777a3f60b9d7d293c349e8bc2196 with result from task 4014 parsl.dataflow.dflow DEBUG: Task 4014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/83ebaeea-3a91-4f0a-91c4-5139fcafed75_isrForCrosstalkSources_855397_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/83ebaeea-3a91-4f0a-91c4-5139fcafed75_isrForCrosstalkSources_855397_46.stderr parsl.dataflow.dflow INFO: Task 3985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cab35d818ed127d520bc65cb6e2d9811 with result from task 3985 parsl.dataflow.dflow DEBUG: Task 3985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/97658274-4fc6-4ca2-97a1-a89f49d40002_isrForCrosstalkSources_855418_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/97658274-4fc6-4ca2-97a1-a89f49d40002_isrForCrosstalkSources_855418_21.stderr parsl.dataflow.dflow INFO: Task 4040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3f4a029d8ff1dec6f9fe4322a76c1bc with result from task 4040 parsl.dataflow.dflow DEBUG: Task 4040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/be6a3853-a0f5-4f4c-81e6-a613bb62ec3f_isrForCrosstalkSources_855436_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/be6a3853-a0f5-4f4c-81e6-a613bb62ec3f_isrForCrosstalkSources_855436_46.stderr parsl.dataflow.dflow INFO: Task 4058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c98bb9cdb8ec6ce6068bfcc5d80f201d with result from task 4058 parsl.dataflow.dflow DEBUG: Task 4058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/624eb7eb-e2e4-4837-8cef-0e2bb4332568_isrForCrosstalkSources_855408_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/624eb7eb-e2e4-4837-8cef-0e2bb4332568_isrForCrosstalkSources_855408_26.stderr parsl.dataflow.dflow INFO: Task 4056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dafe2f87afc7b43f444a80f2c51d1e13 with result from task 4056 parsl.dataflow.dflow DEBUG: Task 4056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c63ee3cc-25f2-44cc-8120-35b0374fc093_isrForCrosstalkSources_855384_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/c63ee3cc-25f2-44cc-8120-35b0374fc093_isrForCrosstalkSources_855384_13.stderr parsl.dataflow.dflow INFO: Task 4159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd3214b4269bdbc65ba81ac0086a458e with result from task 4159 parsl.dataflow.dflow DEBUG: Task 4159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6ac82cc4-d32b-4aa4-bdf3-38cf01e394ce_isrForCrosstalkSources_855400_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6ac82cc4-d32b-4aa4-bdf3-38cf01e394ce_isrForCrosstalkSources_855400_46.stderr parsl.dataflow.dflow INFO: Task 4168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00281097ac14442dc3f94a6deb2bee5c with result from task 4168 parsl.dataflow.dflow DEBUG: Task 4168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccb2b8a7-7cda-4374-94f2-713b4409ce54_isrForCrosstalkSources_855403_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ccb2b8a7-7cda-4374-94f2-713b4409ce54_isrForCrosstalkSources_855403_26.stderr parsl.dataflow.dflow INFO: Task 3936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee25ca98005d9426ee247553b18f80c3 with result from task 3936 parsl.dataflow.dflow DEBUG: Task 3936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/c5bb0709-27de-4ee5-87ef-c5947f616cb2_isrForCrosstalkSources_855424_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/c5bb0709-27de-4ee5-87ef-c5947f616cb2_isrForCrosstalkSources_855424_23.stderr parsl.dataflow.dflow INFO: Task 4189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec4159a69ef97acd976287b4575f6ba4 with result from task 4189 parsl.dataflow.dflow DEBUG: Task 4189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/49b2897d-28ce-41eb-acaf-76e9f1cddb4e_isrForCrosstalkSources_855393_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/49b2897d-28ce-41eb-acaf-76e9f1cddb4e_isrForCrosstalkSources_855393_9.stderr parsl.dataflow.dflow INFO: Task 4138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b71ff2cdefa637c001e4245ba36fe42a with result from task 4138 parsl.dataflow.dflow DEBUG: Task 4138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccdbee24-8d33-421c-9e05-7763798cd891_isrForCrosstalkSources_855433_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ccdbee24-8d33-421c-9e05-7763798cd891_isrForCrosstalkSources_855433_49.stderr parsl.dataflow.dflow INFO: Task 4197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b512b82bba2de2ba17a47c18b7f3c1c with result from task 4197 parsl.dataflow.dflow DEBUG: Task 4197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78c2ceee-d589-4537-a27a-ee99e87d410b_isrForCrosstalkSources_855433_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/78c2ceee-d589-4537-a27a-ee99e87d410b_isrForCrosstalkSources_855433_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 791 try 0 failed parsl.dataflow.dflow INFO: Task 791 marked for retry parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 1e7a458c5c77278ec905ecaa6e2adf22 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 1 launched on executor multi with executor id 5023 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 792 try 0 failed parsl.dataflow.dflow INFO: Task 792 marked for retry parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 8c3e06d0ae95866fbe420644ed949c6e parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa19f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 1 launched on executor multi with executor id 5024 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 793 try 0 failed parsl.dataflow.dflow INFO: Task 793 marked for retry parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash eee190888085a014e2496ec72d0baad1 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 1 launched on executor multi with executor id 5025 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 794 try 0 failed parsl.dataflow.dflow INFO: Task 794 marked for retry parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 05794fc12ed60142732abdaed4d25bec parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 1 launched on executor multi with executor id 5026 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 795 try 0 failed parsl.dataflow.dflow INFO: Task 795 marked for retry parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 5b206c1e89c81ea81fde7bb005171730 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 1 launched on executor multi with executor id 5027 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 796 try 0 failed parsl.dataflow.dflow INFO: Task 796 marked for retry parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash daf3d1daf138b382a49930693605b896 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2402c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 1 launched on executor multi with executor id 5028 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 797 try 0 failed parsl.dataflow.dflow INFO: Task 797 marked for retry parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 71ea44cf6efbebf6de2ba24206180614 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 1 launched on executor multi with executor id 5029 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 798 try 0 failed parsl.dataflow.dflow INFO: Task 798 marked for retry parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash b2d020d98c9f6fb6df30831818517ee0 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 1 launched on executor multi with executor id 5030 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 799 try 0 failed parsl.dataflow.dflow INFO: Task 799 marked for retry parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 415f643c16d33fb8b9172458dab79d6b parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 1 launched on executor multi with executor id 5031 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 800 try 0 failed parsl.dataflow.dflow INFO: Task 800 marked for retry parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash b3f23a67a0d40f776cdd5221d20f985a parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 1 launched on executor multi with executor id 5032 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 801 try 0 failed parsl.dataflow.dflow INFO: Task 801 marked for retry parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 9952d59c8ebfd5b0565529b1da2bcc9c parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2405e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 1 launched on executor multi with executor id 5033 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 802 try 0 failed parsl.dataflow.dflow INFO: Task 802 marked for retry parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 4d6aad113247ed5ac8eb070948cdc11f parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 1 launched on executor multi with executor id 5034 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 803 try 0 failed parsl.dataflow.dflow INFO: Task 803 marked for retry parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 972c49fedc3efbbf9f277a960b822c55 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa2407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 1 launched on executor multi with executor id 5035 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 804 try 0 failed parsl.dataflow.dflow INFO: Task 804 marked for retry parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 347d4182371e0f83516a93341e40f981 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 1 launched on executor multi with executor id 5036 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 805 try 0 failed parsl.dataflow.dflow INFO: Task 805 marked for retry parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash ec11fa2fffad88bc471b77b669a7a928 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 1 launched on executor multi with executor id 5037 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 806 try 0 failed parsl.dataflow.dflow INFO: Task 806 marked for retry parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 63334f9bd3766f10de65a7c9d6cd1930 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459aa240900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 1 launched on executor multi with executor id 5038 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stderr parsl.dataflow.dflow INFO: Task 4079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17d1cf7d27d9637bbf7c19cb6fc35f9a with result from task 4079 parsl.dataflow.dflow DEBUG: Task 4079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/eba5b252-fab5-40ad-8de2-5b6d8919e224_isrForCrosstalkSources_855418_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/eba5b252-fab5-40ad-8de2-5b6d8919e224_isrForCrosstalkSources_855418_9.stderr parsl.dataflow.dflow INFO: Task 4174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07ba00d8b2cb589029a238d992298a80 with result from task 4174 parsl.dataflow.dflow DEBUG: Task 4174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8a9a6031-92ea-4f85-ad17-cabbf82a6df5_isrForCrosstalkSources_855414_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/8a9a6031-92ea-4f85-ad17-cabbf82a6df5_isrForCrosstalkSources_855414_27.stderr parsl.dataflow.dflow INFO: Task 4313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f971958c6643d6fba6fd716c50ce233 with result from task 4313 parsl.dataflow.dflow DEBUG: Task 4313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1320388-92ad-44f6-8d1f-3d065e48c5c8_isrForCrosstalkSources_855395_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1320388-92ad-44f6-8d1f-3d065e48c5c8_isrForCrosstalkSources_855395_25.stderr parsl.dataflow.dflow INFO: Task 4280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e61cadf4195565a9887b2ac25423053 with result from task 4280 parsl.dataflow.dflow DEBUG: Task 4280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6dc09dcc-b68d-46fa-8028-3dabcb00b4af_isrForCrosstalkSources_855417_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/6dc09dcc-b68d-46fa-8028-3dabcb00b4af_isrForCrosstalkSources_855417_40.stderr parsl.dataflow.dflow INFO: Task 4092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc2cfd2a1d06bdb7faca0a19cd5775d with result from task 4092 parsl.dataflow.dflow DEBUG: Task 4092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/9c2274c9-4c12-4fd6-b32d-c1c8f194e966_isrForCrosstalkSources_855388_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/9c2274c9-4c12-4fd6-b32d-c1c8f194e966_isrForCrosstalkSources_855388_45.stderr parsl.dataflow.dflow INFO: Task 4083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dc317aaa251a2827f37a2bb1695950f with result from task 4083 parsl.dataflow.dflow DEBUG: Task 4083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b50ee4b3-780a-4eac-94cb-8c482a1a7256_isrForCrosstalkSources_855410_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b50ee4b3-780a-4eac-94cb-8c482a1a7256_isrForCrosstalkSources_855410_14.stderr parsl.dataflow.dflow INFO: Task 4281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e7141e914155b7633e713be22a12ef with result from task 4281 parsl.dataflow.dflow DEBUG: Task 4281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a91fd2fd-42ff-4cba-86af-e85409eb06a3_isrForCrosstalkSources_855420_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a91fd2fd-42ff-4cba-86af-e85409eb06a3_isrForCrosstalkSources_855420_32.stderr parsl.dataflow.dflow INFO: Task 4258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4740c7a8bf98af4a8925af732fdd9190 with result from task 4258 parsl.dataflow.dflow DEBUG: Task 4258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1ec98d01-bf9b-40bd-a3e8-144b2fba6626_isrForCrosstalkSources_855446_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1ec98d01-bf9b-40bd-a3e8-144b2fba6626_isrForCrosstalkSources_855446_25.stderr parsl.dataflow.dflow INFO: Task 4060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d267fb98bd4ddd94ee66ba3641c953c9 with result from task 4060 parsl.dataflow.dflow DEBUG: Task 4060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f667d1f9-7fda-4957-8996-71c5a39f9c48_isrForCrosstalkSources_855380_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f667d1f9-7fda-4957-8996-71c5a39f9c48_isrForCrosstalkSources_855380_51.stderr parsl.dataflow.dflow INFO: Task 3990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 371c749253c15200da39a57415c2d1ea with result from task 3990 parsl.dataflow.dflow DEBUG: Task 3990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/517c74c4-aa07-4dd9-8a0d-d93f7b4af689_isrForCrosstalkSources_855386_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/517c74c4-aa07-4dd9-8a0d-d93f7b4af689_isrForCrosstalkSources_855386_5.stderr parsl.dataflow.dflow INFO: Task 4039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a7006b4d8d6ad6c5572b92a8fcec9de with result from task 4039 parsl.dataflow.dflow DEBUG: Task 4039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/12f053dc-30e9-4352-8447-d8f0db5fcc22_isrForCrosstalkSources_855437_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/12f053dc-30e9-4352-8447-d8f0db5fcc22_isrForCrosstalkSources_855437_16.stderr parsl.dataflow.dflow INFO: Task 4172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf85fc0fbe103b7c13fdf7e5ea49df31 with result from task 4172 parsl.dataflow.dflow DEBUG: Task 4172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/d6122729-3df5-4e31-be7f-9c360016f879_isrForCrosstalkSources_855434_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/d6122729-3df5-4e31-be7f-9c360016f879_isrForCrosstalkSources_855434_55.stderr parsl.dataflow.dflow INFO: Task 4257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e8edec3a5d907ac7c6fc67f90a84392 with result from task 4257 parsl.dataflow.dflow DEBUG: Task 4257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/71771860-4808-4042-abf9-815b1203e9e9_isrForCrosstalkSources_855453_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/71771860-4808-4042-abf9-815b1203e9e9_isrForCrosstalkSources_855453_8.stderr parsl.dataflow.dflow INFO: Task 4167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33cc921b05ce02bac1ec908592cb57c3 with result from task 4167 parsl.dataflow.dflow DEBUG: Task 4167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/84020382-f54a-4bdf-8616-a5e0b9e30b0e_isrForCrosstalkSources_855446_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/84020382-f54a-4bdf-8616-a5e0b9e30b0e_isrForCrosstalkSources_855446_41.stderr parsl.dataflow.dflow INFO: Task 4349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f34b1b0a612ae9785c3af05e7acc6505 with result from task 4349 parsl.dataflow.dflow DEBUG: Task 4349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2568793-13d0-4029-9a2a-b749a655ed11_isrForCrosstalkSources_855385_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2568793-13d0-4029-9a2a-b749a655ed11_isrForCrosstalkSources_855385_25.stderr parsl.dataflow.dflow INFO: Task 4052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e4f1bf7d8252e937e9902a2a6de4982 with result from task 4052 parsl.dataflow.dflow DEBUG: Task 4052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3eccadee-5586-41f4-817b-a5011d33d756_isrForCrosstalkSources_855447_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3eccadee-5586-41f4-817b-a5011d33d756_isrForCrosstalkSources_855447_26.stderr parsl.dataflow.dflow INFO: Task 4170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d636619292385e3d6dd5f087c4af23 with result from task 4170 parsl.dataflow.dflow DEBUG: Task 4170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0fa490fb-573a-4b3a-82a1-16ee1d2d9060_isrForCrosstalkSources_855418_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/0fa490fb-573a-4b3a-82a1-16ee1d2d9060_isrForCrosstalkSources_855418_44.stderr parsl.dataflow.dflow INFO: Task 4231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b02872a0eb610e8fd7afc7c45f6dfffa with result from task 4231 parsl.dataflow.dflow DEBUG: Task 4231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a582f427-c275-4e94-8903-dbc0be2576c8_isrForCrosstalkSources_855386_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/a582f427-c275-4e94-8903-dbc0be2576c8_isrForCrosstalkSources_855386_24.stderr parsl.dataflow.dflow INFO: Task 4275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81477f2c8e05e2972ac3c0247c28ee75 with result from task 4275 parsl.dataflow.dflow DEBUG: Task 4275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/205700b1-c287-42ad-b66f-2f522dc7d6f9_isrForCrosstalkSources_855418_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/205700b1-c287-42ad-b66f-2f522dc7d6f9_isrForCrosstalkSources_855418_17.stderr parsl.dataflow.dflow INFO: Task 3986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdaa4dc3ab130c70af4c30fa78ea74b with result from task 3986 parsl.dataflow.dflow DEBUG: Task 3986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b1813780-e950-419a-812a-e2d6c378d83c_isrForCrosstalkSources_855423_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/b1813780-e950-419a-812a-e2d6c378d83c_isrForCrosstalkSources_855423_26.stderr parsl.dataflow.dflow INFO: Task 3974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b128ee80558e9819a06d08931b309535 with result from task 3974 parsl.dataflow.dflow DEBUG: Task 3974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ab4de581-290c-4b93-9830-fade76cb882d_isrForCrosstalkSources_855414_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/ab4de581-290c-4b93-9830-fade76cb882d_isrForCrosstalkSources_855414_10.stderr parsl.dataflow.dflow INFO: Task 4287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36970f888c7616b7fa8e189e7cd26843 with result from task 4287 parsl.dataflow.dflow DEBUG: Task 4287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fab6202b-21da-44a1-a265-6746e65ac78b_isrForCrosstalkSources_855379_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/fab6202b-21da-44a1-a265-6746e65ac78b_isrForCrosstalkSources_855379_30.stderr parsl.dataflow.dflow INFO: Task 4033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2594ca3832ef7c8cf17f298a757b3d4 with result from task 4033 parsl.dataflow.dflow DEBUG: Task 4033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/60036fcf-0b99-4870-bc54-18afd82022ee_isrForCrosstalkSources_855392_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/60036fcf-0b99-4870-bc54-18afd82022ee_isrForCrosstalkSources_855392_29.stderr parsl.dataflow.dflow INFO: Task 4084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fef3d919ac94a969fc6a17739391cf0 with result from task 4084 parsl.dataflow.dflow DEBUG: Task 4084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/85d95276-8375-4f2d-bd26-46e85a59ee65_isrForCrosstalkSources_855387_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/85d95276-8375-4f2d-bd26-46e85a59ee65_isrForCrosstalkSources_855387_22.stderr parsl.dataflow.dflow INFO: Task 4182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0212f337b9c0b110f44e0983ebec2068 with result from task 4182 parsl.dataflow.dflow DEBUG: Task 4182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0350d338-c59e-46be-ac66-3e464fd8b46e_isrForCrosstalkSources_855443_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0350d338-c59e-46be-ac66-3e464fd8b46e_isrForCrosstalkSources_855443_21.stderr parsl.dataflow.dflow INFO: Task 4049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 664c339ef65513716a14d0d412482508 with result from task 4049 parsl.dataflow.dflow DEBUG: Task 4049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6943df4f-a1f7-4f84-b92a-19aadb1e6abd_isrForCrosstalkSources_855379_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6943df4f-a1f7-4f84-b92a-19aadb1e6abd_isrForCrosstalkSources_855379_43.stderr parsl.dataflow.dflow INFO: Task 4051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b29c26230ff6cf40e616b41758aa0e69 with result from task 4051 parsl.dataflow.dflow DEBUG: Task 4051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/da78e0f6-e453-42ba-bee4-fe31f3c150fc_isrForCrosstalkSources_855389_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/da78e0f6-e453-42ba-bee4-fe31f3c150fc_isrForCrosstalkSources_855389_3.stderr parsl.dataflow.dflow INFO: Task 4046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e56dd6ce1f38b85361497fb42baad4 with result from task 4046 parsl.dataflow.dflow DEBUG: Task 4046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1015263-12ca-4d96-84e9-67cfd9e67a14_isrForCrosstalkSources_855395_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a1015263-12ca-4d96-84e9-67cfd9e67a14_isrForCrosstalkSources_855395_29.stderr parsl.dataflow.dflow INFO: Task 4346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad018f34916556651d1a1760129c489 with result from task 4346 parsl.dataflow.dflow DEBUG: Task 4346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2a7c569b-89a9-49f1-bc1d-b7b544989edf_isrForCrosstalkSources_855419_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2a7c569b-89a9-49f1-bc1d-b7b544989edf_isrForCrosstalkSources_855419_20.stderr parsl.dataflow.dflow INFO: Task 4194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3d4eb9dc5b843b6eca46d1a88e1554 with result from task 4194 parsl.dataflow.dflow DEBUG: Task 4194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/bd2ede90-4b5e-4fc6-9c42-a39bb40fec7a_isrForCrosstalkSources_855433_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/bd2ede90-4b5e-4fc6-9c42-a39bb40fec7a_isrForCrosstalkSources_855433_25.stderr parsl.dataflow.dflow INFO: Task 4206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa9e0457714dde83605a8959465e6bba with result from task 4206 parsl.dataflow.dflow DEBUG: Task 4206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8bfe1cd4-0e97-4ad0-829d-5bfed62a1472_isrForCrosstalkSources_855397_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8bfe1cd4-0e97-4ad0-829d-5bfed62a1472_isrForCrosstalkSources_855397_26.stderr parsl.dataflow.dflow INFO: Task 4013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 840dbadce3cd9b8b1d44408c6b564ba4 with result from task 4013 parsl.dataflow.dflow DEBUG: Task 4013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/53385777-37ab-49b6-b581-2fea9ded981c_isrForCrosstalkSources_855404_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/53385777-37ab-49b6-b581-2fea9ded981c_isrForCrosstalkSources_855404_41.stderr parsl.dataflow.dflow INFO: Task 4248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88c7068fd2fac22ef24d553e9baa85a4 with result from task 4248 parsl.dataflow.dflow DEBUG: Task 4248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032a29d0-ad81-48de-9470-92a374822501_isrForCrosstalkSources_855414_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/032a29d0-ad81-48de-9470-92a374822501_isrForCrosstalkSources_855414_9.stderr parsl.dataflow.dflow INFO: Task 4129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 383e347c920624a98873707672b1f0c6 with result from task 4129 parsl.dataflow.dflow DEBUG: Task 4129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2c2cafa-2235-413a-af0b-3f9e523fdf4b_isrForCrosstalkSources_855382_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e2c2cafa-2235-413a-af0b-3f9e523fdf4b_isrForCrosstalkSources_855382_32.stderr parsl.dataflow.dflow INFO: Task 4186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d3cf36735f8c2ada85d590ad182fbc9 with result from task 4186 parsl.dataflow.dflow DEBUG: Task 4186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d5ccbfd-feda-4f2b-8818-9a40a22df10f_isrForCrosstalkSources_855444_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8d5ccbfd-feda-4f2b-8818-9a40a22df10f_isrForCrosstalkSources_855444_24.stderr parsl.dataflow.dflow INFO: Task 4171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6025d3675caa30b1a831b62bf069de24 with result from task 4171 parsl.dataflow.dflow DEBUG: Task 4171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ad55fdb8-cec8-4dc1-a842-423e572e7bff_isrForCrosstalkSources_855410_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/ad55fdb8-cec8-4dc1-a842-423e572e7bff_isrForCrosstalkSources_855410_56.stderr parsl.dataflow.dflow INFO: Task 4055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b028e5e69f4b1473c81c0a6e937c989 with result from task 4055 parsl.dataflow.dflow DEBUG: Task 4055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b80c4d22-fe84-41a2-8830-1571a5fe59d4_isrForCrosstalkSources_855414_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/b80c4d22-fe84-41a2-8830-1571a5fe59d4_isrForCrosstalkSources_855414_36.stderr parsl.dataflow.dflow INFO: Task 4267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37540f5e42757eaa526a93c661f52cde with result from task 4267 parsl.dataflow.dflow DEBUG: Task 4267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/71e7b0c4-1fe8-425f-b453-2e0fbecb5930_isrForCrosstalkSources_855441_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/71e7b0c4-1fe8-425f-b453-2e0fbecb5930_isrForCrosstalkSources_855441_39.stderr parsl.dataflow.dflow INFO: Task 4134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fced5358a3eeb0d8994fff4ca29a713b with result from task 4134 parsl.dataflow.dflow DEBUG: Task 4134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6843b2d3-51f0-4ce4-9151-f966eec9025b_isrForCrosstalkSources_855400_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/6843b2d3-51f0-4ce4-9151-f966eec9025b_isrForCrosstalkSources_855400_5.stderr parsl.dataflow.dflow INFO: Task 4303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c92bfa071066d46180b593ab1892ef79 with result from task 4303 parsl.dataflow.dflow DEBUG: Task 4303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ae6c3658-420a-4b6b-97d7-05db0a2f91e2_isrForCrosstalkSources_855424_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/ae6c3658-420a-4b6b-97d7-05db0a2f91e2_isrForCrosstalkSources_855424_8.stderr parsl.dataflow.dflow INFO: Task 4045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9e32b6a32d2de9a9d79c956ecdaea39 with result from task 4045 parsl.dataflow.dflow DEBUG: Task 4045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/5b5e0f72-a910-422a-9adb-dfcbd51a5582_isrForCrosstalkSources_855390_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/5b5e0f72-a910-422a-9adb-dfcbd51a5582_isrForCrosstalkSources_855390_58.stderr parsl.dataflow.dflow INFO: Task 4331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 550c9b49dedd06d1d827b19031511bc9 with result from task 4331 parsl.dataflow.dflow DEBUG: Task 4331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d14fcdf0-43e7-4e8b-9b09-deca2241f98d_isrForCrosstalkSources_855420_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/d14fcdf0-43e7-4e8b-9b09-deca2241f98d_isrForCrosstalkSources_855420_46.stderr parsl.dataflow.dflow INFO: Task 4289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c908d119c9eadb9fd64012389759a89 with result from task 4289 parsl.dataflow.dflow DEBUG: Task 4289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6411571b-b776-4ac4-ae59-b14b02c796ad_isrForCrosstalkSources_855436_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/6411571b-b776-4ac4-ae59-b14b02c796ad_isrForCrosstalkSources_855436_23.stderr parsl.dataflow.dflow INFO: Task 4295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b879b051653cdb402722c70602ddb9b6 with result from task 4295 parsl.dataflow.dflow DEBUG: Task 4295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0bd859d6-8bfb-4c5f-89fa-513b1ea68a34_isrForCrosstalkSources_855387_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/0bd859d6-8bfb-4c5f-89fa-513b1ea68a34_isrForCrosstalkSources_855387_4.stderr parsl.dataflow.dflow INFO: Task 4141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73aa250660285b231a0e5fe1f49d5b67 with result from task 4141 parsl.dataflow.dflow DEBUG: Task 4141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/60fa02d1-7fdf-4120-9542-4eca5566609f_isrForCrosstalkSources_855421_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/60fa02d1-7fdf-4120-9542-4eca5566609f_isrForCrosstalkSources_855421_40.stderr parsl.dataflow.dflow INFO: Task 4260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f283a7c9e96ec8b2dc6b858131f8ad4b with result from task 4260 parsl.dataflow.dflow DEBUG: Task 4260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/56b535f2-785b-461f-b9e8-1e06f77e3b0f_isrForCrosstalkSources_855455_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/56b535f2-785b-461f-b9e8-1e06f77e3b0f_isrForCrosstalkSources_855455_15.stderr parsl.dataflow.dflow INFO: Task 4347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b621b1f55f28e8cd48d6bf3f7ff9767 with result from task 4347 parsl.dataflow.dflow DEBUG: Task 4347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20298ed1-f6cd-4c96-96a3-b4ff1f4709bd_isrForCrosstalkSources_855419_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/20298ed1-f6cd-4c96-96a3-b4ff1f4709bd_isrForCrosstalkSources_855419_41.stderr parsl.dataflow.dflow INFO: Task 4348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85820042dfa0e03b007217d06e6c17fe with result from task 4348 parsl.dataflow.dflow DEBUG: Task 4348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bf81d2c1-cf4e-4c39-921c-06066c330c7d_isrForCrosstalkSources_855430_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/bf81d2c1-cf4e-4c39-921c-06066c330c7d_isrForCrosstalkSources_855430_11.stderr parsl.dataflow.dflow INFO: Task 4353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3b68194b8f7b0f103129d8c194b7d4c with result from task 4353 parsl.dataflow.dflow DEBUG: Task 4353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cfdeaba8-05c2-40ab-9e42-50bc0e566918_isrForCrosstalkSources_855407_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/cfdeaba8-05c2-40ab-9e42-50bc0e566918_isrForCrosstalkSources_855407_51.stderr parsl.dataflow.dflow INFO: Task 4344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea905992f51731cc5bcb414ffd5a7513 with result from task 4344 parsl.dataflow.dflow DEBUG: Task 4344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/034cbe0e-7ee6-4b2b-b767-ef264a3014fd_isrForCrosstalkSources_855430_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/034cbe0e-7ee6-4b2b-b767-ef264a3014fd_isrForCrosstalkSources_855430_3.stderr parsl.dataflow.dflow INFO: Task 3960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42b4a6518a708ffefabcd528fffe9b93 with result from task 3960 parsl.dataflow.dflow DEBUG: Task 3960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e524de97-b1b2-4171-aa36-90d820d858d6_isrForCrosstalkSources_855450_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/e524de97-b1b2-4171-aa36-90d820d858d6_isrForCrosstalkSources_855450_60.stderr parsl.dataflow.dflow INFO: Task 4285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe99fc883178b4709d50c7978e392a6 with result from task 4285 parsl.dataflow.dflow DEBUG: Task 4285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/78bd9115-97a5-4eee-a7b2-d117cc10752a_isrForCrosstalkSources_855428_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/78bd9115-97a5-4eee-a7b2-d117cc10752a_isrForCrosstalkSources_855428_1.stderr parsl.dataflow.dflow INFO: Task 4268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8048e2a8d41ffda64632132597e8ca28 with result from task 4268 parsl.dataflow.dflow DEBUG: Task 4268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f3e2bf66-f251-4c1a-b796-4a0ec5570b40_isrForCrosstalkSources_855388_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/f3e2bf66-f251-4c1a-b796-4a0ec5570b40_isrForCrosstalkSources_855388_51.stderr parsl.dataflow.dflow INFO: Task 4230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df82a9a9db6bb4f9732a662190084f9f with result from task 4230 parsl.dataflow.dflow DEBUG: Task 4230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/12744a18-8301-4664-bbe0-a46a40fd3868_isrForCrosstalkSources_855385_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/12744a18-8301-4664-bbe0-a46a40fd3868_isrForCrosstalkSources_855385_40.stderr parsl.dataflow.dflow INFO: Task 4332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8185e7cdd9bebca80ff1c22cdc7c65a with result from task 4332 parsl.dataflow.dflow DEBUG: Task 4332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/56f34de0-2f67-4d1b-b801-14a5826e852b_isrForCrosstalkSources_855399_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/56f34de0-2f67-4d1b-b801-14a5826e852b_isrForCrosstalkSources_855399_13.stderr parsl.dataflow.dflow INFO: Task 4041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70775d894b1b87528f3c0edbb6468ca6 with result from task 4041 parsl.dataflow.dflow DEBUG: Task 4041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a3d9fa00-69d8-4094-ad44-c53d7097e208_isrForCrosstalkSources_855420_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/a3d9fa00-69d8-4094-ad44-c53d7097e208_isrForCrosstalkSources_855420_43.stderr parsl.dataflow.dflow INFO: Task 4304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9e1543d750fbe4b760dadcc2c78e04 with result from task 4304 parsl.dataflow.dflow DEBUG: Task 4304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e416f927-4d8c-4665-8237-05ec79c33a0b_isrForCrosstalkSources_855391_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e416f927-4d8c-4665-8237-05ec79c33a0b_isrForCrosstalkSources_855391_29.stderr parsl.dataflow.dflow INFO: Task 4274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3ecf98874364639e444ece78eb60ede with result from task 4274 parsl.dataflow.dflow DEBUG: Task 4274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ab04e1a2-cfee-4e51-a782-7abc344179b6_isrForCrosstalkSources_855434_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/ab04e1a2-cfee-4e51-a782-7abc344179b6_isrForCrosstalkSources_855434_36.stderr parsl.dataflow.dflow INFO: Task 4255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b66dcf745aa496b7a2fa8763eeb24ba9 with result from task 4255 parsl.dataflow.dflow DEBUG: Task 4255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/db97ba76-012d-482f-91e9-7efbd8202a60_isrForCrosstalkSources_855452_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/db97ba76-012d-482f-91e9-7efbd8202a60_isrForCrosstalkSources_855452_36.stderr parsl.dataflow.dflow INFO: Task 4302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67956917ca5c735047673a30ecb2fa25 with result from task 4302 parsl.dataflow.dflow DEBUG: Task 4302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4985e6b5-85b3-4b43-95fb-fc6daf8c133a_isrForCrosstalkSources_855383_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4985e6b5-85b3-4b43-95fb-fc6daf8c133a_isrForCrosstalkSources_855383_30.stderr parsl.dataflow.dflow INFO: Task 4054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73edd4c21aa100726fab1aaf053e2038 with result from task 4054 parsl.dataflow.dflow DEBUG: Task 4054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/5043290e-7317-4b88-bbe6-0788ac930ab9_isrForCrosstalkSources_855386_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/5043290e-7317-4b88-bbe6-0788ac930ab9_isrForCrosstalkSources_855386_12.stderr parsl.dataflow.dflow INFO: Task 4296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6bb33c1d6487d27b230520f5e724f5c with result from task 4296 parsl.dataflow.dflow DEBUG: Task 4296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/87cc0c1c-c4e4-4893-abec-c94a170da88c_isrForCrosstalkSources_855426_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/87cc0c1c-c4e4-4893-abec-c94a170da88c_isrForCrosstalkSources_855426_56.stderr parsl.dataflow.dflow INFO: Task 4234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d03df7eea69f3e199e7795fe1e276d6c with result from task 4234 parsl.dataflow.dflow DEBUG: Task 4234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/bf76530e-46a5-4420-9746-10ca5e5dca6a_isrForCrosstalkSources_855435_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/bf76530e-46a5-4420-9746-10ca5e5dca6a_isrForCrosstalkSources_855435_51.stderr parsl.dataflow.dflow INFO: Task 4091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 976e9432389ee9f5f05cab2acf2164d0 with result from task 4091 parsl.dataflow.dflow DEBUG: Task 4091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5d1becd9-4ea7-4561-a0cd-4b4b14423d0f_isrForCrosstalkSources_855382_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/5d1becd9-4ea7-4561-a0cd-4b4b14423d0f_isrForCrosstalkSources_855382_47.stderr parsl.dataflow.dflow INFO: Task 4048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f97da3eef8d91a5d9277ce1280bde32 with result from task 4048 parsl.dataflow.dflow DEBUG: Task 4048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b49768f6-ed97-4c85-9dc3-a186565602a2_isrForCrosstalkSources_855459_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/b49768f6-ed97-4c85-9dc3-a186565602a2_isrForCrosstalkSources_855459_58.stderr parsl.dataflow.dflow INFO: Task 4110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c4a7abcbf621b773d2b5afe3cde88c with result from task 4110 parsl.dataflow.dflow DEBUG: Task 4110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5c3c9d9a-2806-43ba-9f48-b9a9503d4aed_isrForCrosstalkSources_855444_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5c3c9d9a-2806-43ba-9f48-b9a9503d4aed_isrForCrosstalkSources_855444_36.stderr parsl.dataflow.dflow INFO: Task 4333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 005acd61eff21c7c260fd183f6a97c93 with result from task 4333 parsl.dataflow.dflow DEBUG: Task 4333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/faf62389-0918-4dc3-813f-ca0cb073ffc8_isrForCrosstalkSources_855396_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/faf62389-0918-4dc3-813f-ca0cb073ffc8_isrForCrosstalkSources_855396_24.stderr parsl.dataflow.dflow INFO: Task 4061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbc8843cd5e445bb63b8c96bb5625d08 with result from task 4061 parsl.dataflow.dflow DEBUG: Task 4061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f86a8568-2ac7-4735-8feb-a67c01abad58_isrForCrosstalkSources_855426_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/f86a8568-2ac7-4735-8feb-a67c01abad58_isrForCrosstalkSources_855426_50.stderr parsl.dataflow.dflow INFO: Task 4358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be5c91b3d30721afa48e8e3d188c9fe with result from task 4358 parsl.dataflow.dflow DEBUG: Task 4358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6cf854e3-546d-426a-b28a-01c66bb65b6c_isrForCrosstalkSources_855410_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6cf854e3-546d-426a-b28a-01c66bb65b6c_isrForCrosstalkSources_855410_40.stderr parsl.dataflow.dflow INFO: Task 4104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc7b464ebb05a7a4ddae81e8adb16b10 with result from task 4104 parsl.dataflow.dflow DEBUG: Task 4104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cc3b8fb1-0b0e-47e6-84da-5d31a109291e_isrForCrosstalkSources_855447_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/cc3b8fb1-0b0e-47e6-84da-5d31a109291e_isrForCrosstalkSources_855447_15.stderr parsl.dataflow.dflow INFO: Task 4229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9933461ce0b74a7fc30525d22b823f1 with result from task 4229 parsl.dataflow.dflow DEBUG: Task 4229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/59809dc9-c3e0-4558-abf2-8517243640b3_isrForCrosstalkSources_855435_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/59809dc9-c3e0-4558-abf2-8517243640b3_isrForCrosstalkSources_855435_53.stderr parsl.dataflow.dflow INFO: Task 4254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7edffde9cb9eeefc0ed2f39bef8a935 with result from task 4254 parsl.dataflow.dflow DEBUG: Task 4254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9bf5a57d-5bdd-4e9a-9558-258c84ce7715_isrForCrosstalkSources_855397_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/9bf5a57d-5bdd-4e9a-9558-258c84ce7715_isrForCrosstalkSources_855397_47.stderr parsl.dataflow.dflow INFO: Task 4321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a386668887e1056fa31129cca8e36c51 with result from task 4321 parsl.dataflow.dflow DEBUG: Task 4321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8f97bd81-7abf-4f1a-a8fa-6714ca40f4f9_isrForCrosstalkSources_855423_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8f97bd81-7abf-4f1a-a8fa-6714ca40f4f9_isrForCrosstalkSources_855423_56.stderr parsl.dataflow.dflow INFO: Task 4053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d12a6883d335d0aa2b6a00a4e254bd with result from task 4053 parsl.dataflow.dflow DEBUG: Task 4053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f19bdcfc-b364-4b5f-afff-2c2369f98bd9_isrForCrosstalkSources_855445_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/f19bdcfc-b364-4b5f-afff-2c2369f98bd9_isrForCrosstalkSources_855445_54.stderr parsl.dataflow.dflow INFO: Task 4345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8864c21e7279373f86c3209b0630c02 with result from task 4345 parsl.dataflow.dflow DEBUG: Task 4345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c85f8162-c9d2-4c78-8d01-5e750fc80a6a_isrForCrosstalkSources_855390_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c85f8162-c9d2-4c78-8d01-5e750fc80a6a_isrForCrosstalkSources_855390_47.stderr parsl.dataflow.dflow INFO: Task 4154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1493e0e1b7b717b84f807c9a7e8aa7be with result from task 4154 parsl.dataflow.dflow DEBUG: Task 4154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f0f3166d-808a-445c-ba01-b8273d30c6e5_isrForCrosstalkSources_855425_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f0f3166d-808a-445c-ba01-b8273d30c6e5_isrForCrosstalkSources_855425_9.stderr parsl.dataflow.dflow INFO: Task 4292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caee9518d533df6bdfd0b8110a210835 with result from task 4292 parsl.dataflow.dflow DEBUG: Task 4292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3c869758-f155-42ce-ad0e-7a0c1f34568f_isrForCrosstalkSources_855438_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/3c869758-f155-42ce-ad0e-7a0c1f34568f_isrForCrosstalkSources_855438_28.stderr parsl.dataflow.dflow INFO: Task 4351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d54f918420c7fc5ec26cc211bd036005 with result from task 4351 parsl.dataflow.dflow DEBUG: Task 4351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/be8c3e03-9c64-44bf-8995-2a09eefac86e_isrForCrosstalkSources_855388_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/be8c3e03-9c64-44bf-8995-2a09eefac86e_isrForCrosstalkSources_855388_20.stderr parsl.dataflow.dflow INFO: Task 4389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c24feea3d81f2d62370ec62d94ca91ac with result from task 4389 parsl.dataflow.dflow DEBUG: Task 4389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2dc70254-7757-4770-8dbb-8a7341a3344f_isrForCrosstalkSources_855418_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2dc70254-7757-4770-8dbb-8a7341a3344f_isrForCrosstalkSources_855418_53.stderr parsl.dataflow.dflow INFO: Task 4247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67d02025ed26b1410dce0ac655f0bfd9 with result from task 4247 parsl.dataflow.dflow DEBUG: Task 4247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a37c415c-5407-449f-bd5c-5376a3f8c6d0_isrForCrosstalkSources_855404_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a37c415c-5407-449f-bd5c-5376a3f8c6d0_isrForCrosstalkSources_855404_58.stderr parsl.dataflow.dflow INFO: Task 4377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e71cca44fd8b6059c5060334e2c7781a with result from task 4377 parsl.dataflow.dflow DEBUG: Task 4377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6ca5fd3d-6cd0-447b-b6fa-b9c1c399cbaa_isrForCrosstalkSources_855410_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6ca5fd3d-6cd0-447b-b6fa-b9c1c399cbaa_isrForCrosstalkSources_855410_46.stderr parsl.dataflow.dflow INFO: Task 4038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95022c5980f4e54096166e06d982674f with result from task 4038 parsl.dataflow.dflow DEBUG: Task 4038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0bb39b89-0cf6-462a-a707-df9e40953561_isrForCrosstalkSources_855411_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0bb39b89-0cf6-462a-a707-df9e40953561_isrForCrosstalkSources_855411_38.stderr parsl.dataflow.dflow INFO: Task 4391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d07b160ae148606c3198746df8e3e429 with result from task 4391 parsl.dataflow.dflow DEBUG: Task 4391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/495c3577-204e-4fc3-a090-555caa1a71cb_isrForCrosstalkSources_855403_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/495c3577-204e-4fc3-a090-555caa1a71cb_isrForCrosstalkSources_855403_10.stderr parsl.dataflow.dflow INFO: Task 4394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7912cb3fe2c89b2e4112d00fc3c1e998 with result from task 4394 parsl.dataflow.dflow DEBUG: Task 4394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/23be3b5e-e824-409d-b8b5-c99841d86b06_isrForCrosstalkSources_855440_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/23be3b5e-e824-409d-b8b5-c99841d86b06_isrForCrosstalkSources_855440_18.stderr parsl.dataflow.dflow INFO: Task 4330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c6670fe4da91c322a05d18a118ce2bd with result from task 4330 parsl.dataflow.dflow DEBUG: Task 4330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d7a6c32e-f06c-47c8-9b83-bd323f382fa3_isrForCrosstalkSources_855417_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d7a6c32e-f06c-47c8-9b83-bd323f382fa3_isrForCrosstalkSources_855417_2.stderr parsl.dataflow.dflow INFO: Task 4070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d7038a7ebfc2c3ce876c1ce7e2506b9 with result from task 4070 parsl.dataflow.dflow DEBUG: Task 4070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b8982267-92a5-4651-a460-ad9278fab4b9_isrForCrosstalkSources_855384_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/b8982267-92a5-4651-a460-ad9278fab4b9_isrForCrosstalkSources_855384_61.stderr parsl.dataflow.dflow INFO: Task 3893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 818865af0bff762855c6bd9909969cb7 with result from task 3893 parsl.dataflow.dflow DEBUG: Task 3893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/01229cae-cb3e-4419-8167-649dbf03422c_isrForCrosstalkSources_855421_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/01229cae-cb3e-4419-8167-649dbf03422c_isrForCrosstalkSources_855421_59.stderr parsl.dataflow.dflow INFO: Task 4311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d65f07991b3ce2bc79ddb8f2b6407c6d with result from task 4311 parsl.dataflow.dflow DEBUG: Task 4311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a1bf5ad9-0425-4cd6-84fd-0aa3f1de06c1_isrForCrosstalkSources_855439_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a1bf5ad9-0425-4cd6-84fd-0aa3f1de06c1_isrForCrosstalkSources_855439_43.stderr parsl.dataflow.dflow INFO: Task 4293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f49c5a433a195a3bc80dbe8a4b5f74b8 with result from task 4293 parsl.dataflow.dflow DEBUG: Task 4293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5b30f73c-f8a1-48d5-858a-319cc0842653_isrForCrosstalkSources_855427_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5b30f73c-f8a1-48d5-858a-319cc0842653_isrForCrosstalkSources_855427_58.stderr parsl.dataflow.dflow INFO: Task 4200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ad202e7386a19faf1ac61c498f540cb with result from task 4200 parsl.dataflow.dflow DEBUG: Task 4200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/3a6d1326-b07c-45f9-a9ef-54b14e83bb23_isrForCrosstalkSources_855415_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/3a6d1326-b07c-45f9-a9ef-54b14e83bb23_isrForCrosstalkSources_855415_10.stderr parsl.dataflow.dflow INFO: Task 4130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa4603055f121c1de66634d7123ef08a with result from task 4130 parsl.dataflow.dflow DEBUG: Task 4130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/9c444596-8be8-4514-954f-dc36a352c09b_isrForCrosstalkSources_855454_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/9c444596-8be8-4514-954f-dc36a352c09b_isrForCrosstalkSources_855454_30.stderr parsl.dataflow.dflow INFO: Task 4235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4f16e489c8b926579bd9711d581e389 with result from task 4235 parsl.dataflow.dflow DEBUG: Task 4235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4769bf72-6e44-4dea-a79e-ea56f2ae7c38_isrForCrosstalkSources_855434_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/4769bf72-6e44-4dea-a79e-ea56f2ae7c38_isrForCrosstalkSources_855434_7.stderr parsl.dataflow.dflow INFO: Task 4376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1c8956e3ac13dfdea208e36d840ba4a with result from task 4376 parsl.dataflow.dflow DEBUG: Task 4376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7cfc70d-46cc-42bd-ab00-e366d4755953_isrForCrosstalkSources_855382_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7cfc70d-46cc-42bd-ab00-e366d4755953_isrForCrosstalkSources_855382_24.stderr parsl.dataflow.dflow INFO: Task 4373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ec64000a4368cffc36f5c7ac2c92a07 with result from task 4373 parsl.dataflow.dflow DEBUG: Task 4373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/9868d12e-0c2c-49f1-8ea6-7c644d319fa0_isrForCrosstalkSources_855453_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/9868d12e-0c2c-49f1-8ea6-7c644d319fa0_isrForCrosstalkSources_855453_24.stderr parsl.dataflow.dflow INFO: Task 4372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 628704ef831effcc9e272980d5d430c7 with result from task 4372 parsl.dataflow.dflow DEBUG: Task 4372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/92090860-1d56-4d54-8946-6f56d2970484_isrForCrosstalkSources_855379_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/92090860-1d56-4d54-8946-6f56d2970484_isrForCrosstalkSources_855379_10.stderr parsl.dataflow.dflow INFO: Task 4375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d70dd8f88b4e074e432105025ecbabe6 with result from task 4375 parsl.dataflow.dflow DEBUG: Task 4375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/71c041a9-0c97-4ca6-ae4e-2e5022970a1e_isrForCrosstalkSources_855428_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/71c041a9-0c97-4ca6-ae4e-2e5022970a1e_isrForCrosstalkSources_855428_36.stderr parsl.dataflow.dflow INFO: Task 4336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8aec7fbf2996844587e13e2f90a0149 with result from task 4336 parsl.dataflow.dflow DEBUG: Task 4336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/280a5be6-1799-4df5-91f2-5605b3aefe84_isrForCrosstalkSources_855380_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/280a5be6-1799-4df5-91f2-5605b3aefe84_isrForCrosstalkSources_855380_50.stderr parsl.dataflow.dflow INFO: Task 4294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ccf1874b07b1f8be5e463c5e4820aa with result from task 4294 parsl.dataflow.dflow DEBUG: Task 4294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/e6d112da-6181-429b-9ff5-1b0c8ccc5815_isrForCrosstalkSources_855410_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/e6d112da-6181-429b-9ff5-1b0c8ccc5815_isrForCrosstalkSources_855410_11.stderr parsl.dataflow.dflow INFO: Task 4342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f012e7cf5a48aa0e68bb8fc7e371b963 with result from task 4342 parsl.dataflow.dflow DEBUG: Task 4342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a1140311-00d8-4046-becb-6aab00b1ebfc_isrForCrosstalkSources_855418_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a1140311-00d8-4046-becb-6aab00b1ebfc_isrForCrosstalkSources_855418_58.stderr parsl.dataflow.dflow INFO: Task 4290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d6669c2d3cf0145c03c48190216bdbb with result from task 4290 parsl.dataflow.dflow DEBUG: Task 4290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8c8e81f3-de92-45db-bedc-b8bdfbe0bdff_isrForCrosstalkSources_855435_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/8c8e81f3-de92-45db-bedc-b8bdfbe0bdff_isrForCrosstalkSources_855435_38.stderr parsl.dataflow.dflow INFO: Task 4209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db795f9d6045217cdea8dcfc82dd24ee with result from task 4209 parsl.dataflow.dflow DEBUG: Task 4209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2a7048c2-89fb-4855-ad89-3ef5d3b6286f_isrForCrosstalkSources_855393_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/2a7048c2-89fb-4855-ad89-3ef5d3b6286f_isrForCrosstalkSources_855393_25.stderr parsl.dataflow.dflow INFO: Task 4299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f60a80562405fc96ae446f84c9b2cce with result from task 4299 parsl.dataflow.dflow DEBUG: Task 4299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/1914859f-9cba-4d0e-846f-9b4a4c6684dc_isrForCrosstalkSources_855415_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/1914859f-9cba-4d0e-846f-9b4a4c6684dc_isrForCrosstalkSources_855415_59.stderr parsl.dataflow.dflow INFO: Task 4142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 171aab325b6205d514f0af1bc3ac495b with result from task 4142 parsl.dataflow.dflow DEBUG: Task 4142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a51b7543-d2ce-45ab-b0c6-124392513c36_isrForCrosstalkSources_855397_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/a51b7543-d2ce-45ab-b0c6-124392513c36_isrForCrosstalkSources_855397_37.stderr parsl.dataflow.dflow INFO: Task 4361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e3f6a72d08bc2ea7d30c61c037396c with result from task 4361 parsl.dataflow.dflow DEBUG: Task 4361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9d3edea4-2c68-455d-8f45-71cc0e8543e5_isrForCrosstalkSources_855403_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9d3edea4-2c68-455d-8f45-71cc0e8543e5_isrForCrosstalkSources_855403_3.stderr parsl.dataflow.dflow INFO: Task 4283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 169b1f9341eb5a1514bf9c40e27e1dac with result from task 4283 parsl.dataflow.dflow DEBUG: Task 4283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a652c21-eaca-40a0-87a0-185f80c1e7ab_isrForCrosstalkSources_855396_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/0a652c21-eaca-40a0-87a0-185f80c1e7ab_isrForCrosstalkSources_855396_60.stderr parsl.dataflow.dflow INFO: Task 4340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 959a9cdb0fb5c482c667018eacb91596 with result from task 4340 parsl.dataflow.dflow DEBUG: Task 4340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e7afe286-d946-4c62-a24d-5b81e59c8afe_isrForCrosstalkSources_855399_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/e7afe286-d946-4c62-a24d-5b81e59c8afe_isrForCrosstalkSources_855399_43.stderr parsl.dataflow.dflow INFO: Task 4133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b1a2073d4213a7c8cb382c9633f15c5 with result from task 4133 parsl.dataflow.dflow DEBUG: Task 4133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8d227e92-3232-452d-9aee-71b4072a3ffd_isrForCrosstalkSources_855450_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/8d227e92-3232-452d-9aee-71b4072a3ffd_isrForCrosstalkSources_855450_9.stderr parsl.dataflow.dflow INFO: Task 4310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d995a816153c8165d51dfaaec7d59e35 with result from task 4310 parsl.dataflow.dflow DEBUG: Task 4310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a4ee7da2-f2ef-413c-af27-d51219a36388_isrForCrosstalkSources_855381_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a4ee7da2-f2ef-413c-af27-d51219a36388_isrForCrosstalkSources_855381_55.stderr parsl.dataflow.dflow INFO: Task 4150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8918071289b3616d803d97414369eedb with result from task 4150 parsl.dataflow.dflow DEBUG: Task 4150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c4863a77-8f04-4bf8-9276-63c01e2809ff_isrForCrosstalkSources_855443_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/c4863a77-8f04-4bf8-9276-63c01e2809ff_isrForCrosstalkSources_855443_56.stderr parsl.dataflow.dflow INFO: Task 4212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dbcc22d2d367e852cc6669a2454fa88 with result from task 4212 parsl.dataflow.dflow DEBUG: Task 4212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8d4e034e-d8f7-414c-b457-ba0be886e0b2_isrForCrosstalkSources_855410_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/8d4e034e-d8f7-414c-b457-ba0be886e0b2_isrForCrosstalkSources_855410_9.stderr parsl.dataflow.dflow INFO: Task 4320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 435649ce3ed2fe5508f536146cd56947 with result from task 4320 parsl.dataflow.dflow DEBUG: Task 4320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3298eae9-9c6e-4462-9c57-b5d8d5624ac2_isrForCrosstalkSources_855457_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/3298eae9-9c6e-4462-9c57-b5d8d5624ac2_isrForCrosstalkSources_855457_37.stderr parsl.dataflow.dflow INFO: Task 3920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18de3aef85dd2e16a0f49f51783621dd with result from task 3920 parsl.dataflow.dflow DEBUG: Task 3920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f7366e35-adf0-4597-8ad1-ef7cfd07b6c2_isrForCrosstalkSources_855432_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/f7366e35-adf0-4597-8ad1-ef7cfd07b6c2_isrForCrosstalkSources_855432_36.stderr parsl.dataflow.dflow INFO: Task 4305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a11c7214358e6948cf8b02160bfd4d6 with result from task 4305 parsl.dataflow.dflow DEBUG: Task 4305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/47808243-0548-4a62-9e1b-a04fbc2f9a4a_isrForCrosstalkSources_855452_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/47808243-0548-4a62-9e1b-a04fbc2f9a4a_isrForCrosstalkSources_855452_55.stderr parsl.dataflow.dflow INFO: Task 4223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edc164ca17d3bdff07b17da64966a66b with result from task 4223 parsl.dataflow.dflow DEBUG: Task 4223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/8cc7944a-e517-49a4-9c76-006994169e22_isrForCrosstalkSources_855394_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/8cc7944a-e517-49a4-9c76-006994169e22_isrForCrosstalkSources_855394_6.stderr parsl.dataflow.dflow INFO: Task 4286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e4f24cbb7394c6c780879ccfc48a57e with result from task 4286 parsl.dataflow.dflow DEBUG: Task 4286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9a5ea6ae-e895-460a-9951-263a3985bc45_isrForCrosstalkSources_855392_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/9a5ea6ae-e895-460a-9951-263a3985bc45_isrForCrosstalkSources_855392_58.stderr parsl.dataflow.dflow INFO: Task 4259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00dcf7ca3b339b22fa50319f65a513d9 with result from task 4259 parsl.dataflow.dflow DEBUG: Task 4259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/805f8f68-cc2a-4970-ae48-daff7ff4a079_isrForCrosstalkSources_855384_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/805f8f68-cc2a-4970-ae48-daff7ff4a079_isrForCrosstalkSources_855384_55.stderr parsl.dataflow.dflow INFO: Task 4222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1885695f7f360c072e1bd780e4196526 with result from task 4222 parsl.dataflow.dflow DEBUG: Task 4222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30745fe3-957e-46d4-9c79-7301e7c034b6_isrForCrosstalkSources_855436_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/30745fe3-957e-46d4-9c79-7301e7c034b6_isrForCrosstalkSources_855436_58.stderr parsl.dataflow.dflow INFO: Task 4350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1e9f023bf27175d07583dabd91e811f with result from task 4350 parsl.dataflow.dflow DEBUG: Task 4350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e73b2c18-1ac7-4fca-b1ec-c5f4b948f000_isrForCrosstalkSources_855416_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/e73b2c18-1ac7-4fca-b1ec-c5f4b948f000_isrForCrosstalkSources_855416_30.stderr parsl.dataflow.dflow INFO: Task 4288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d6aecd97d51d5a76c19ddc230167d8e with result from task 4288 parsl.dataflow.dflow DEBUG: Task 4288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1554b1c4-0a1a-4401-93b6-e56378f94cd4_isrForCrosstalkSources_855387_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1554b1c4-0a1a-4401-93b6-e56378f94cd4_isrForCrosstalkSources_855387_62.stderr parsl.dataflow.dflow INFO: Task 4306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc61ee90f9894092b81db31a3e10e5aa with result from task 4306 parsl.dataflow.dflow DEBUG: Task 4306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d1f4fae9-8085-45f8-b79e-9b6c0e1e3d88_isrForCrosstalkSources_855430_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d1f4fae9-8085-45f8-b79e-9b6c0e1e3d88_isrForCrosstalkSources_855430_29.stderr parsl.dataflow.dflow INFO: Task 4226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8233cec7b040000bda175b7b672ce9c with result from task 4226 parsl.dataflow.dflow DEBUG: Task 4226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9cac7f3d-4def-4ee5-96f4-97a7ed91742d_isrForCrosstalkSources_855380_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/9cac7f3d-4def-4ee5-96f4-97a7ed91742d_isrForCrosstalkSources_855380_54.stderr parsl.dataflow.dflow INFO: Task 4135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a198a00bee7f2d218fda8f197e406eed with result from task 4135 parsl.dataflow.dflow DEBUG: Task 4135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/517c4b3b-87cc-42c0-b970-d86164a9d381_isrForCrosstalkSources_855424_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/517c4b3b-87cc-42c0-b970-d86164a9d381_isrForCrosstalkSources_855424_52.stderr parsl.dataflow.dflow INFO: Task 3937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed185e8f8de74b2a2170c4e6247d649a with result from task 3937 parsl.dataflow.dflow DEBUG: Task 3937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0278b029-7e91-4a10-b6b3-1d76b61c938a_isrForCrosstalkSources_855411_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/0278b029-7e91-4a10-b6b3-1d76b61c938a_isrForCrosstalkSources_855411_27.stderr parsl.dataflow.dflow INFO: Task 4319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574025372894836882beb8ef1489062a with result from task 4319 parsl.dataflow.dflow DEBUG: Task 4319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/016f279b-dd4d-4798-9264-06fe125ab113_isrForCrosstalkSources_855411_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/016f279b-dd4d-4798-9264-06fe125ab113_isrForCrosstalkSources_855411_19.stderr parsl.dataflow.dflow INFO: Task 4341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23c5ffc7a7b29d5e65ee91f0965aa062 with result from task 4341 parsl.dataflow.dflow DEBUG: Task 4341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c53b3e1d-81e2-4fac-9c8f-1e6b7509f0ba_isrForCrosstalkSources_855408_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/c53b3e1d-81e2-4fac-9c8f-1e6b7509f0ba_isrForCrosstalkSources_855408_22.stderr parsl.dataflow.dflow INFO: Task 4155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1c5cbd71cdc7a1087903cfa4266da08 with result from task 4155 parsl.dataflow.dflow DEBUG: Task 4155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec1642b2-0fdd-456e-aec2-a9e79033be57_isrForCrosstalkSources_855413_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/ec1642b2-0fdd-456e-aec2-a9e79033be57_isrForCrosstalkSources_855413_58.stderr parsl.dataflow.dflow INFO: Task 4458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7527f1d7d694d8520d3bc58a74fa05e with result from task 4458 parsl.dataflow.dflow DEBUG: Task 4458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b854d41-fd2d-4b61-98fd-49038971167b_isrForCrosstalkSources_855459_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b854d41-fd2d-4b61-98fd-49038971167b_isrForCrosstalkSources_855459_4.stderr parsl.dataflow.dflow INFO: Task 4423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd9329e11273c339b1a3e4c3efa05f0 with result from task 4423 parsl.dataflow.dflow DEBUG: Task 4423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e016955a-906c-4e1b-894a-088f835dffa5_isrForCrosstalkSources_855413_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e016955a-906c-4e1b-894a-088f835dffa5_isrForCrosstalkSources_855413_26.stderr parsl.dataflow.dflow INFO: Task 4324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c556d8d0efaae0cf04b52fdc97cb0e01 with result from task 4324 parsl.dataflow.dflow DEBUG: Task 4324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04b89321-1378-4bbb-a4bf-a8cca6969019_isrForCrosstalkSources_855412_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/04b89321-1378-4bbb-a4bf-a8cca6969019_isrForCrosstalkSources_855412_21.stderr parsl.dataflow.dflow INFO: Task 4460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a3f3a329486062e1402629d354e5dbf with result from task 4460 parsl.dataflow.dflow DEBUG: Task 4460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73a28890-b65f-429e-be36-190d29235c92_isrForCrosstalkSources_855444_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/73a28890-b65f-429e-be36-190d29235c92_isrForCrosstalkSources_855444_40.stderr parsl.dataflow.dflow INFO: Task 4415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55430cb38b8e33aeaad7c3f11795eb4c with result from task 4415 parsl.dataflow.dflow DEBUG: Task 4415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4017f72c-931c-42bb-95d3-73f341feda8c_isrForCrosstalkSources_855384_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/4017f72c-931c-42bb-95d3-73f341feda8c_isrForCrosstalkSources_855384_8.stderr parsl.dataflow.dflow INFO: Task 4444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d75c8fb3d23acc1c908b33c022fcbcd with result from task 4444 parsl.dataflow.dflow DEBUG: Task 4444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ace41b0-9a4d-4563-9770-a75fa37d4620_isrForCrosstalkSources_855429_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/3ace41b0-9a4d-4563-9770-a75fa37d4620_isrForCrosstalkSources_855429_14.stderr parsl.dataflow.dflow INFO: Task 4419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bb141d4ba10020b59a8d21a50546873 with result from task 4419 parsl.dataflow.dflow DEBUG: Task 4419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e5682352-1ad8-4ee4-bc1e-08578ae07939_isrForCrosstalkSources_855455_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/e5682352-1ad8-4ee4-bc1e-08578ae07939_isrForCrosstalkSources_855455_8.stderr parsl.dataflow.dflow INFO: Task 4364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04c5a2070f9e0c43cac371a8ff4bb439 with result from task 4364 parsl.dataflow.dflow DEBUG: Task 4364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/260908fd-91e3-4816-b5d3-1050e1112e56_isrForCrosstalkSources_855435_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/260908fd-91e3-4816-b5d3-1050e1112e56_isrForCrosstalkSources_855435_13.stderr parsl.dataflow.dflow INFO: Task 4442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e9e27187cefc797e5a957a06740c6e with result from task 4442 parsl.dataflow.dflow DEBUG: Task 4442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/94288910-0a97-4915-901a-c940fda1a86a_isrForCrosstalkSources_855389_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/94288910-0a97-4915-901a-c940fda1a86a_isrForCrosstalkSources_855389_19.stderr parsl.dataflow.dflow INFO: Task 4314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ee7a1112583c6fdc44a35f6745ce42b with result from task 4314 parsl.dataflow.dflow DEBUG: Task 4314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ba2ab040-6fae-45db-9f0a-1717891bbeab_isrForCrosstalkSources_855419_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ba2ab040-6fae-45db-9f0a-1717891bbeab_isrForCrosstalkSources_855419_46.stderr parsl.dataflow.dflow INFO: Task 4380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bcd1837d4239523a59b336ad8e3f738 with result from task 4380 parsl.dataflow.dflow DEBUG: Task 4380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6d72d1d2-e407-4db5-857d-4039541801f6_isrForCrosstalkSources_855431_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6d72d1d2-e407-4db5-857d-4039541801f6_isrForCrosstalkSources_855431_53.stderr parsl.dataflow.dflow INFO: Task 4269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3a254b575c57b0504ac9bc0746876e7 with result from task 4269 parsl.dataflow.dflow DEBUG: Task 4269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d8229bc-8dae-4087-8b71-d14fb7fae80d_isrForCrosstalkSources_855380_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/3d8229bc-8dae-4087-8b71-d14fb7fae80d_isrForCrosstalkSources_855380_5.stderr parsl.dataflow.dflow INFO: Task 4434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b059be8e5e647d1a7c20281efd8e91f with result from task 4434 parsl.dataflow.dflow DEBUG: Task 4434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/002a4cb0-e615-4c63-8a84-e8dbf1b70dd7_isrForCrosstalkSources_855418_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/002a4cb0-e615-4c63-8a84-e8dbf1b70dd7_isrForCrosstalkSources_855418_10.stderr parsl.dataflow.dflow INFO: Task 4371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81ae21f1b2893a4495d36522ee0363fc with result from task 4371 parsl.dataflow.dflow DEBUG: Task 4371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1500d13c-29b9-4fec-81c0-184ddde0efdc_isrForCrosstalkSources_855397_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1500d13c-29b9-4fec-81c0-184ddde0efdc_isrForCrosstalkSources_855397_39.stderr parsl.dataflow.dflow INFO: Task 4441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a052bf8e7e3e28db0aa0bc42c1fd55 with result from task 4441 parsl.dataflow.dflow DEBUG: Task 4441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4a4b74fd-ee3e-4e0e-b669-435d838e8091_isrForCrosstalkSources_855433_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/4a4b74fd-ee3e-4e0e-b669-435d838e8091_isrForCrosstalkSources_855433_16.stderr parsl.dataflow.dflow INFO: Task 4396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 250e9bdd62c5c6847b1558216de655a1 with result from task 4396 parsl.dataflow.dflow DEBUG: Task 4396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a9305ad1-e2d3-4592-97e4-4e254f2d435d_isrForCrosstalkSources_855398_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/a9305ad1-e2d3-4592-97e4-4e254f2d435d_isrForCrosstalkSources_855398_16.stderr parsl.dataflow.dflow INFO: Task 4428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ebcad6dc582b5e84cafba170d2542f8 with result from task 4428 parsl.dataflow.dflow DEBUG: Task 4428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/43cd52a0-13f3-459a-b649-9b44ee95c3e4_isrForCrosstalkSources_855398_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/43cd52a0-13f3-459a-b649-9b44ee95c3e4_isrForCrosstalkSources_855398_39.stderr parsl.dataflow.dflow INFO: Task 4464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e2f2ca9dc00ab4cdea76806cc0d6239 with result from task 4464 parsl.dataflow.dflow DEBUG: Task 4464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8f247bcc-d81b-4552-a96d-40ee46084706_isrForCrosstalkSources_855404_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8f247bcc-d81b-4552-a96d-40ee46084706_isrForCrosstalkSources_855404_40.stderr parsl.dataflow.dflow INFO: Task 4388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b6e9eb42be07b121c1d6e0be86cc55 with result from task 4388 parsl.dataflow.dflow DEBUG: Task 4388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/055575f9-03a0-4ea5-915e-5212e42c7ab0_isrForCrosstalkSources_855425_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/055575f9-03a0-4ea5-915e-5212e42c7ab0_isrForCrosstalkSources_855425_4.stderr parsl.dataflow.dflow INFO: Task 4433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9af54a221ad9e946e90d798b5060ef90 with result from task 4433 parsl.dataflow.dflow DEBUG: Task 4433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7c9809a2-0d40-4e39-ba39-de55d191ff02_isrForCrosstalkSources_855453_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/7c9809a2-0d40-4e39-ba39-de55d191ff02_isrForCrosstalkSources_855453_22.stderr parsl.dataflow.dflow INFO: Task 4264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9851635817bc2a1fae4f9f442674dcef with result from task 4264 parsl.dataflow.dflow DEBUG: Task 4264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b9da4ce2-e1d3-4acc-a981-5499f10115a0_isrForCrosstalkSources_855391_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/b9da4ce2-e1d3-4acc-a981-5499f10115a0_isrForCrosstalkSources_855391_35.stderr parsl.dataflow.dflow INFO: Task 4085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 242edfd70a1d4ea0c2d3fc421a44018b with result from task 4085 parsl.dataflow.dflow DEBUG: Task 4085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/11b7eb84-148a-48ff-aa47-6659a7e31ff9_isrForCrosstalkSources_855398_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/11b7eb84-148a-48ff-aa47-6659a7e31ff9_isrForCrosstalkSources_855398_11.stderr parsl.dataflow.dflow INFO: Task 4454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c105598ce7d3052aaae337e76382b84c with result from task 4454 parsl.dataflow.dflow DEBUG: Task 4454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fa9852de-f4ae-4023-8762-8c9713a27039_isrForCrosstalkSources_855425_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/fa9852de-f4ae-4023-8762-8c9713a27039_isrForCrosstalkSources_855425_36.stderr parsl.dataflow.dflow INFO: Task 4466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 471018880cbaf33acf90c412d15e0901 with result from task 4466 parsl.dataflow.dflow DEBUG: Task 4466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14113fc8-7660-43e6-80e7-90dfe46fd2ac_isrForCrosstalkSources_855442_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14113fc8-7660-43e6-80e7-90dfe46fd2ac_isrForCrosstalkSources_855442_32.stderr parsl.dataflow.dflow INFO: Task 4327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bfcfddf484ce1ac7bf4097ec387daf1 with result from task 4327 parsl.dataflow.dflow DEBUG: Task 4327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e26f81aa-4f8a-435c-ab48-a4c07446d2f2_isrForCrosstalkSources_855457_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e26f81aa-4f8a-435c-ab48-a4c07446d2f2_isrForCrosstalkSources_855457_28.stderr parsl.dataflow.dflow INFO: Task 4421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d853315941d709e1158ce4585b8e72bd with result from task 4421 parsl.dataflow.dflow DEBUG: Task 4421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/dccd7e6c-766a-449e-bd9c-c5ad72854b75_isrForCrosstalkSources_855379_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/dccd7e6c-766a-449e-bd9c-c5ad72854b75_isrForCrosstalkSources_855379_1.stderr parsl.dataflow.dflow INFO: Task 4403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddf6d147c26ae49c6d888d3bc6625845 with result from task 4403 parsl.dataflow.dflow DEBUG: Task 4403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/60783d28-d268-4aeb-9474-9703d9cf18c4_isrForCrosstalkSources_855381_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/60783d28-d268-4aeb-9474-9703d9cf18c4_isrForCrosstalkSources_855381_8.stderr parsl.dataflow.dflow INFO: Task 4429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce0d92a3eef8bde4954f5958026ee473 with result from task 4429 parsl.dataflow.dflow DEBUG: Task 4429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c8209a57-ef40-4de5-be89-a35638a88170_isrForCrosstalkSources_855400_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/c8209a57-ef40-4de5-be89-a35638a88170_isrForCrosstalkSources_855400_40.stderr parsl.dataflow.dflow INFO: Task 4381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36595643a29f5f30795c6edd17f3c9c8 with result from task 4381 parsl.dataflow.dflow DEBUG: Task 4381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/47674b6d-e83a-4abb-90b3-8f5bad1719d1_isrForCrosstalkSources_855456_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/47674b6d-e83a-4abb-90b3-8f5bad1719d1_isrForCrosstalkSources_855456_14.stderr parsl.dataflow.dflow INFO: Task 4365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d589356c17738528f01097a510d9bb7 with result from task 4365 parsl.dataflow.dflow DEBUG: Task 4365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0b3c2fc-616f-409a-8a4b-4e8be36fea1f_isrForCrosstalkSources_855380_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/a0b3c2fc-616f-409a-8a4b-4e8be36fea1f_isrForCrosstalkSources_855380_15.stderr parsl.dataflow.dflow INFO: Task 4432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574bf2238b67b9b31572dcca75eab076 with result from task 4432 parsl.dataflow.dflow DEBUG: Task 4432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/2ee420dd-8d3d-48f5-8047-0615a201292c_isrForCrosstalkSources_855430_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/2ee420dd-8d3d-48f5-8047-0615a201292c_isrForCrosstalkSources_855430_15.stderr parsl.dataflow.dflow INFO: Task 4436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b481ffc588974150a7c49a57982e7e9a with result from task 4436 parsl.dataflow.dflow DEBUG: Task 4436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/db9ff831-3340-4348-8786-53c56050d689_isrForCrosstalkSources_855457_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/db9ff831-3340-4348-8786-53c56050d689_isrForCrosstalkSources_855457_61.stderr parsl.dataflow.dflow INFO: Task 4445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fbacfe7d532e1fbdd365b90bac207c2 with result from task 4445 parsl.dataflow.dflow DEBUG: Task 4445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b868ba9e-24e7-478c-817e-99fb5dd4b4bc_isrForCrosstalkSources_855415_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/b868ba9e-24e7-478c-817e-99fb5dd4b4bc_isrForCrosstalkSources_855415_16.stderr parsl.dataflow.dflow INFO: Task 4113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4fb752bcd949fc6500ceab915ed2e1a with result from task 4113 parsl.dataflow.dflow DEBUG: Task 4113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/852416bc-87c3-4c33-99d2-bac582a98731_isrForCrosstalkSources_855394_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/852416bc-87c3-4c33-99d2-bac582a98731_isrForCrosstalkSources_855394_28.stderr parsl.dataflow.dflow INFO: Task 4449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a215f900bec2bf6827b579d4032b267 with result from task 4449 parsl.dataflow.dflow DEBUG: Task 4449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ea2e2923-f1f3-4c12-8840-bde6c5a69092_isrForCrosstalkSources_855397_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ea2e2923-f1f3-4c12-8840-bde6c5a69092_isrForCrosstalkSources_855397_45.stderr parsl.dataflow.dflow INFO: Task 4418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eae4c250589b6eaa3eefd40310db0389 with result from task 4418 parsl.dataflow.dflow DEBUG: Task 4418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bc40a678-c62a-4593-bbdd-8188bf6cc3c7_isrForCrosstalkSources_855436_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bc40a678-c62a-4593-bbdd-8188bf6cc3c7_isrForCrosstalkSources_855436_30.stderr parsl.dataflow.dflow INFO: Task 4437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5c02922c6489407472d1d20977e289c with result from task 4437 parsl.dataflow.dflow DEBUG: Task 4437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/36abda30-4635-4ffa-a893-7b529fd1f123_isrForCrosstalkSources_855383_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/36abda30-4635-4ffa-a893-7b529fd1f123_isrForCrosstalkSources_855383_36.stderr parsl.dataflow.dflow INFO: Task 4404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66586e4830694bce0590fb2e0f310608 with result from task 4404 parsl.dataflow.dflow DEBUG: Task 4404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b443d601-f43c-4b0d-859b-c86996e417b1_isrForCrosstalkSources_855436_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/b443d601-f43c-4b0d-859b-c86996e417b1_isrForCrosstalkSources_855436_22.stderr parsl.dataflow.dflow INFO: Task 4368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1004d97258f4177212c085eb712df3b8 with result from task 4368 parsl.dataflow.dflow DEBUG: Task 4368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/ebe34178-d732-45f3-8181-56a0f2e43f55_isrForCrosstalkSources_855428_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/ebe34178-d732-45f3-8181-56a0f2e43f55_isrForCrosstalkSources_855428_18.stderr parsl.dataflow.dflow INFO: Task 4266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d14263569a823548a4c8484e9a5b9755 with result from task 4266 parsl.dataflow.dflow DEBUG: Task 4266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3c07092c-b8ed-4908-8541-80e7c7dda3ec_isrForCrosstalkSources_855442_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/3c07092c-b8ed-4908-8541-80e7c7dda3ec_isrForCrosstalkSources_855442_49.stderr parsl.dataflow.dflow INFO: Task 4438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 168672c8010d1021b4554c9a1ec9699b with result from task 4438 parsl.dataflow.dflow DEBUG: Task 4438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/7e3d3aad-7bd6-4374-b37f-beb02cd2d543_isrForCrosstalkSources_855425_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/7e3d3aad-7bd6-4374-b37f-beb02cd2d543_isrForCrosstalkSources_855425_57.stderr parsl.dataflow.dflow INFO: Task 4312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d37f36fa9d2ea9460fdf22d5a68e9948 with result from task 4312 parsl.dataflow.dflow DEBUG: Task 4312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/22581263-b603-437b-8f8e-99ce46ba51bc_isrForCrosstalkSources_855393_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/22581263-b603-437b-8f8e-99ce46ba51bc_isrForCrosstalkSources_855393_57.stderr parsl.dataflow.dflow INFO: Task 4482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ca77d45af73edad21894d3eccb6234 with result from task 4482 parsl.dataflow.dflow DEBUG: Task 4482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e0abd165-1fee-44e0-8027-85d183b94b13_isrForCrosstalkSources_855400_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/e0abd165-1fee-44e0-8027-85d183b94b13_isrForCrosstalkSources_855400_44.stderr parsl.dataflow.dflow INFO: Task 4383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e93f38797e81c06c6f8931fe9678287 with result from task 4383 parsl.dataflow.dflow DEBUG: Task 4383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4393b30e-1934-4044-b920-bdbe31fb6d94_isrForCrosstalkSources_855415_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4393b30e-1934-4044-b920-bdbe31fb6d94_isrForCrosstalkSources_855415_36.stderr parsl.dataflow.dflow INFO: Task 4401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ec2ed0d028b0cd3e84cff8146dc3b53 with result from task 4401 parsl.dataflow.dflow DEBUG: Task 4401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7c46f8bb-cee0-46c1-9345-fd9bf3f53a0b_isrForCrosstalkSources_855391_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/7c46f8bb-cee0-46c1-9345-fd9bf3f53a0b_isrForCrosstalkSources_855391_3.stderr parsl.dataflow.dflow INFO: Task 4405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47167d30940208add098550c0200a0dc with result from task 4405 parsl.dataflow.dflow DEBUG: Task 4405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ec87c72f-117f-4eb9-9ba5-d290194dfa44_isrForCrosstalkSources_855430_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/ec87c72f-117f-4eb9-9ba5-d290194dfa44_isrForCrosstalkSources_855430_2.stderr parsl.dataflow.dflow INFO: Task 4414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5336b4b2bac76b5b0d82e9a7769d909b with result from task 4414 parsl.dataflow.dflow DEBUG: Task 4414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/625a8c04-2085-4f56-b9bc-458d53bff5a6_isrForCrosstalkSources_855380_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/625a8c04-2085-4f56-b9bc-458d53bff5a6_isrForCrosstalkSources_855380_12.stderr parsl.dataflow.dflow INFO: Task 4265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e056e35d69ac526d2142d2af0c215c97 with result from task 4265 parsl.dataflow.dflow DEBUG: Task 4265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7876e18-3e28-4dbc-8fa2-62b2d7c2fe19_isrForCrosstalkSources_855382_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/d7876e18-3e28-4dbc-8fa2-62b2d7c2fe19_isrForCrosstalkSources_855382_49.stderr parsl.dataflow.dflow INFO: Task 4485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c431942dc402d10041716c14a5a3434a with result from task 4485 parsl.dataflow.dflow DEBUG: Task 4485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/effa8f22-eb80-40ce-8746-77c9a6116ba9_isrForCrosstalkSources_855409_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/effa8f22-eb80-40ce-8746-77c9a6116ba9_isrForCrosstalkSources_855409_15.stderr parsl.dataflow.dflow INFO: Task 4338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8c6c0036395fdfe0040bf487c6154ca with result from task 4338 parsl.dataflow.dflow DEBUG: Task 4338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d609c42f-2e4c-4211-885e-842f4d545d7f_isrForCrosstalkSources_855430_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/d609c42f-2e4c-4211-885e-842f4d545d7f_isrForCrosstalkSources_855430_48.stderr parsl.dataflow.dflow INFO: Task 4477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb3ee60c98249b3f86628b52b579e0cf with result from task 4477 parsl.dataflow.dflow DEBUG: Task 4477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca06a1de-5591-4005-94d4-c5a320c5f1d5_isrForCrosstalkSources_855400_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ca06a1de-5591-4005-94d4-c5a320c5f1d5_isrForCrosstalkSources_855400_3.stderr parsl.dataflow.dflow INFO: Task 4378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b583b6e29b0cae8483666540d297e17 with result from task 4378 parsl.dataflow.dflow DEBUG: Task 4378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2559b0f2-abf9-4d5a-a2c8-23e8197484c5_isrForCrosstalkSources_855424_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/2559b0f2-abf9-4d5a-a2c8-23e8197484c5_isrForCrosstalkSources_855424_58.stderr parsl.dataflow.dflow INFO: Task 4399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63bcac0318828eaa09721c704ca78041 with result from task 4399 parsl.dataflow.dflow DEBUG: Task 4399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3c6ebad6-4ac6-4b3d-b298-c56c767e962d_isrForCrosstalkSources_855401_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/3c6ebad6-4ac6-4b3d-b298-c56c767e962d_isrForCrosstalkSources_855401_53.stderr parsl.dataflow.dflow INFO: Task 4424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 235f3c56e2403d04093ce76768f5325a with result from task 4424 parsl.dataflow.dflow DEBUG: Task 4424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8cc45309-f079-4483-ac33-dd64d08d4c7d_isrForCrosstalkSources_855423_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/8cc45309-f079-4483-ac33-dd64d08d4c7d_isrForCrosstalkSources_855423_43.stderr parsl.dataflow.dflow INFO: Task 4474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9944fc4e5c2d05fe9a7a46a35205dc6c with result from task 4474 parsl.dataflow.dflow DEBUG: Task 4474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/06dba6f5-2d2a-4a88-b61d-0f8e62e89de2_isrForCrosstalkSources_855408_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/06dba6f5-2d2a-4a88-b61d-0f8e62e89de2_isrForCrosstalkSources_855408_29.stderr parsl.dataflow.dflow INFO: Task 4270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5cd65bf5fa75c15f67f68d6296ec345 with result from task 4270 parsl.dataflow.dflow DEBUG: Task 4270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/41c6fb97-d1ca-48ff-86a3-ba04ee6f2454_isrForCrosstalkSources_855427_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/41c6fb97-d1ca-48ff-86a3-ba04ee6f2454_isrForCrosstalkSources_855427_62.stderr parsl.dataflow.dflow INFO: Task 4407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b15d4ed8ee8371e67f293934c855fc with result from task 4407 parsl.dataflow.dflow DEBUG: Task 4407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6864b0cf-8312-48e8-88d8-fd52acef28fb_isrForCrosstalkSources_855391_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/6864b0cf-8312-48e8-88d8-fd52acef28fb_isrForCrosstalkSources_855391_19.stderr parsl.dataflow.dflow INFO: Task 4463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda6c9df288a2baf0ad6faa1169a2500 with result from task 4463 parsl.dataflow.dflow DEBUG: Task 4463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ed4f14de-d393-4282-8c52-5d6be4fddd5e_isrForCrosstalkSources_855429_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ed4f14de-d393-4282-8c52-5d6be4fddd5e_isrForCrosstalkSources_855429_55.stderr parsl.dataflow.dflow INFO: Task 4317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3493a50b360b77df2ac15152fa266f2 with result from task 4317 parsl.dataflow.dflow DEBUG: Task 4317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/605a2946-06cb-4841-ba0e-7ade089fabab_isrForCrosstalkSources_855392_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/605a2946-06cb-4841-ba0e-7ade089fabab_isrForCrosstalkSources_855392_50.stderr parsl.dataflow.dflow INFO: Task 4382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6be5ddbf02c65c8aa9c7513abfe43e7c with result from task 4382 parsl.dataflow.dflow DEBUG: Task 4382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8feb2d1b-61c7-4046-a24a-b585563d6d6e_isrForCrosstalkSources_855444_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/8feb2d1b-61c7-4046-a24a-b585563d6d6e_isrForCrosstalkSources_855444_61.stderr parsl.dataflow.dflow INFO: Task 4453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04c6d143d13ed2a05d64788be9c95a46 with result from task 4453 parsl.dataflow.dflow DEBUG: Task 4453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/9a622bef-6fd7-4e59-8f44-e79d2169fe9e_isrForCrosstalkSources_855455_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/9a622bef-6fd7-4e59-8f44-e79d2169fe9e_isrForCrosstalkSources_855455_20.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:55:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964304 proc_lsst.multi INFO: [multi] job_id 16964304 2024-03-09 14:55:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 212 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 212 2024-03-09 14:55:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964304 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964304 parsl.executors.status_handling DEBUG: Launched block 267 on executor multi with job ID 16964304 parsl.executors.status_handling INFO: Allocated block ID 268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-268 2024-03-09 14:55:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 212 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 212 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02f5c2bd849f5f9970f922b91ae741a6 with result from task 3930 parsl.dataflow.dflow DEBUG: Task 3930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/90d47d33-8974-44db-9a23-16d52d75bcdd_isrForCrosstalkSources_855428_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/90d47d33-8974-44db-9a23-16d52d75bcdd_isrForCrosstalkSources_855428_62.stderr parsl.dataflow.dflow INFO: Task 3983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2a69a3dd76aecf5b4ffd90cb5e801d with result from task 3983 parsl.dataflow.dflow DEBUG: Task 3983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/89379901-8f88-41dc-807f-e54899c35632_isrForCrosstalkSources_855458_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/89379901-8f88-41dc-807f-e54899c35632_isrForCrosstalkSources_855458_34.stderr parsl.dataflow.dflow INFO: Task 3888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6173397a8642c1bc8cb1157d12a9fe with result from task 3888 parsl.dataflow.dflow DEBUG: Task 3888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/dd5caa20-2449-46c8-b1e3-d8409cbeac58_isrForCrosstalkSources_855399_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/dd5caa20-2449-46c8-b1e3-d8409cbeac58_isrForCrosstalkSources_855399_57.stderr parsl.dataflow.dflow INFO: Task 3961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4023e07a7eda2b191c09eede3576bb80 with result from task 3961 parsl.dataflow.dflow DEBUG: Task 3961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9b1c87d3-bcca-43f9-be5a-802a006ac4a9_isrForCrosstalkSources_855403_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/9b1c87d3-bcca-43f9-be5a-802a006ac4a9_isrForCrosstalkSources_855403_57.stderr parsl.dataflow.dflow INFO: Task 3929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 153647177b9acfc41e43f6a8ac87ed49 with result from task 3929 parsl.dataflow.dflow DEBUG: Task 3929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/bf2dc53c-6ed0-4b57-bc03-a19f9b1d08ed_isrForCrosstalkSources_855406_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/bf2dc53c-6ed0-4b57-bc03-a19f9b1d08ed_isrForCrosstalkSources_855406_57.stderr parsl.dataflow.dflow INFO: Task 4043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ad00a669067cba7e4bd35df0dadcd72 with result from task 4043 parsl.dataflow.dflow DEBUG: Task 4043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d53ef0ef-ae96-4d8d-ae9a-386c9ef17a3d_isrForCrosstalkSources_855431_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d53ef0ef-ae96-4d8d-ae9a-386c9ef17a3d_isrForCrosstalkSources_855431_26.stderr parsl.dataflow.dflow INFO: Task 3948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ff75c90244b3aca1b24ffb355ad6654 with result from task 3948 parsl.dataflow.dflow DEBUG: Task 3948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f9388783-84cb-41c3-804b-72532829d5eb_isrForCrosstalkSources_855455_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/f9388783-84cb-41c3-804b-72532829d5eb_isrForCrosstalkSources_855455_24.stderr parsl.dataflow.dflow INFO: Task 3730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77137b5ff0adb5fdb63972b806f6501a with result from task 3730 parsl.dataflow.dflow DEBUG: Task 3730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8d7cef6a-99bb-444d-8c0d-d5db03543bff_isrForCrosstalkSources_855449_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8d7cef6a-99bb-444d-8c0d-d5db03543bff_isrForCrosstalkSources_855449_52.stderr parsl.dataflow.dflow INFO: Task 3987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 505c861711ce97b4ec871b8938e3b07d with result from task 3987 parsl.dataflow.dflow DEBUG: Task 3987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b56d1927-06df-4dc5-bad0-dc2fb1c876f1_isrForCrosstalkSources_855438_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b56d1927-06df-4dc5-bad0-dc2fb1c876f1_isrForCrosstalkSources_855438_45.stderr parsl.dataflow.dflow INFO: Task 4016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 264e49444ae25af0fc3f5cbea935e872 with result from task 4016 parsl.dataflow.dflow DEBUG: Task 4016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ef40cbe9-28d7-43c0-acca-861a510ec7c1_isrForCrosstalkSources_855459_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ef40cbe9-28d7-43c0-acca-861a510ec7c1_isrForCrosstalkSources_855459_22.stderr parsl.dataflow.dflow INFO: Task 4000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70a255a254e825739fc6871cc2ca3dec with result from task 4000 parsl.dataflow.dflow DEBUG: Task 4000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d368e4b6-77f0-4ac0-b2f2-3d57d54b8f5f_isrForCrosstalkSources_855448_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d368e4b6-77f0-4ac0-b2f2-3d57d54b8f5f_isrForCrosstalkSources_855448_4.stderr parsl.dataflow.dflow INFO: Task 4471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 733380dea5a8d8f2a535a6d87c079575 with result from task 4471 parsl.dataflow.dflow DEBUG: Task 4471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a270fc16-02d1-4c8b-b724-96a953998d97_isrForCrosstalkSources_855382_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/a270fc16-02d1-4c8b-b724-96a953998d97_isrForCrosstalkSources_855382_29.stderr parsl.dataflow.dflow INFO: Task 3959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47abb9d3abbfb62a58058527f3332b11 with result from task 3959 parsl.dataflow.dflow DEBUG: Task 3959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/80dbe48d-8d77-45aa-aab7-812798163726_isrForCrosstalkSources_855453_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/80dbe48d-8d77-45aa-aab7-812798163726_isrForCrosstalkSources_855453_55.stderr parsl.dataflow.dflow INFO: Task 4140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd0d7dc1d3d88aa929ed5ee39d3dcf5a with result from task 4140 parsl.dataflow.dflow DEBUG: Task 4140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/90d8a7d0-6457-40f0-8bb7-6d7a7a8b6210_isrForCrosstalkSources_855400_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/90d8a7d0-6457-40f0-8bb7-6d7a7a8b6210_isrForCrosstalkSources_855400_45.stderr parsl.dataflow.dflow INFO: Task 4011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 207be426b078f85d24d1296fed089a42 with result from task 4011 parsl.dataflow.dflow DEBUG: Task 4011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e0aa0099-1dda-4afa-976f-25a9ebb38dc9_isrForCrosstalkSources_855422_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/e0aa0099-1dda-4afa-976f-25a9ebb38dc9_isrForCrosstalkSources_855422_23.stderr parsl.dataflow.dflow INFO: Task 4472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27fec0bb3a8ce97c0324d2efc82dae7d with result from task 4472 parsl.dataflow.dflow DEBUG: Task 4472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/70061a86-416a-4240-b250-ce3f18d16000_isrForCrosstalkSources_855428_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/70061a86-416a-4240-b250-ce3f18d16000_isrForCrosstalkSources_855428_12.stderr parsl.dataflow.dflow INFO: Task 4473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e2f60936f9def38263a6149fe7d9d80 with result from task 4473 parsl.dataflow.dflow DEBUG: Task 4473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/bf5168fa-a09d-4fc1-9706-e413d35ef3eb_isrForCrosstalkSources_855457_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/bf5168fa-a09d-4fc1-9706-e413d35ef3eb_isrForCrosstalkSources_855457_4.stderr parsl.dataflow.dflow INFO: Task 3719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e598bbf23e2b62cce3067e6e6cab31c with result from task 3719 parsl.dataflow.dflow DEBUG: Task 3719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1f49fccc-4475-4eee-a32c-d8eb0f8166e0_isrForCrosstalkSources_855442_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/1f49fccc-4475-4eee-a32c-d8eb0f8166e0_isrForCrosstalkSources_855442_55.stderr parsl.dataflow.dflow INFO: Task 3557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b91873525f64f7f3064239a4ec5d2e5d with result from task 3557 parsl.dataflow.dflow DEBUG: Task 3557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e1877135-a346-405a-a1cc-b0b28b0fd76d_isrForCrosstalkSources_855409_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e1877135-a346-405a-a1cc-b0b28b0fd76d_isrForCrosstalkSources_855409_55.stderr parsl.dataflow.dflow INFO: Task 3749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddcc8eeb7a1f0cc37643d44e3a4acd1b with result from task 3749 parsl.dataflow.dflow DEBUG: Task 3749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6475bc93-494c-4cc4-b274-4a08b542f8b9_isrForCrosstalkSources_855387_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/6475bc93-494c-4cc4-b274-4a08b542f8b9_isrForCrosstalkSources_855387_58.stderr parsl.dataflow.dflow INFO: Task 3631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c224cdfb51333b99cc592afbf6ab6768 with result from task 3631 parsl.dataflow.dflow DEBUG: Task 3631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/51c4dca3-02b4-4162-8080-65655e44d13c_isrForCrosstalkSources_855451_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/51c4dca3-02b4-4162-8080-65655e44d13c_isrForCrosstalkSources_855451_49.stderr parsl.dataflow.dflow INFO: Task 3757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f718345f41f85ffc6d40888716e3c849 with result from task 3757 parsl.dataflow.dflow DEBUG: Task 3757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/cdfe4257-f6d3-4856-b654-0f0d92699c34_isrForCrosstalkSources_855395_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/cdfe4257-f6d3-4856-b654-0f0d92699c34_isrForCrosstalkSources_855395_59.stderr parsl.dataflow.dflow INFO: Task 4177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a7211939bf6d78f9faef0d6522dc22 with result from task 4177 parsl.dataflow.dflow DEBUG: Task 4177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf54b091-92b1-4059-b987-c077ed566754_isrForCrosstalkSources_855401_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/cf54b091-92b1-4059-b987-c077ed566754_isrForCrosstalkSources_855401_56.stderr parsl.dataflow.dflow INFO: Task 4181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44bcf52ff91433f22a7b4e3877f2019d with result from task 4181 parsl.dataflow.dflow DEBUG: Task 4181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f1648c5e-8999-43bc-b06e-5310b4a14c43_isrForCrosstalkSources_855379_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/f1648c5e-8999-43bc-b06e-5310b4a14c43_isrForCrosstalkSources_855379_37.stderr parsl.dataflow.dflow INFO: Task 3547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08f9b3af4712f5e2ccb72791895ce1d6 with result from task 3547 parsl.dataflow.dflow DEBUG: Task 3547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c1b2f402-584f-489d-8f7d-9f9ef07d94d8_isrForCrosstalkSources_855417_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c1b2f402-584f-489d-8f7d-9f9ef07d94d8_isrForCrosstalkSources_855417_57.stderr parsl.dataflow.dflow INFO: Task 3720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89bc0fd8cb681efb6b9213b6b93f13f with result from task 3720 parsl.dataflow.dflow DEBUG: Task 3720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f607274c-ad8c-43e0-b2c6-3ee34b1f3f22_isrForCrosstalkSources_855392_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/f607274c-ad8c-43e0-b2c6-3ee34b1f3f22_isrForCrosstalkSources_855392_60.stderr parsl.dataflow.dflow INFO: Task 3903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da6ede873488d3476093688fc186dc5a with result from task 3903 parsl.dataflow.dflow DEBUG: Task 3903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/002578a0-a7fa-4356-8817-6d5060634888_isrForCrosstalkSources_855408_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/002578a0-a7fa-4356-8817-6d5060634888_isrForCrosstalkSources_855408_4.stderr parsl.dataflow.dflow INFO: Task 3797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d00c88276396933bf07e4218d6bbbf with result from task 3797 parsl.dataflow.dflow DEBUG: Task 3797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/baacdf0f-d9e5-4855-a171-1812ecb59d55_isrForCrosstalkSources_855432_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/baacdf0f-d9e5-4855-a171-1812ecb59d55_isrForCrosstalkSources_855432_49.stderr parsl.dataflow.dflow INFO: Task 3958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff5f00299c1e923f149281ff155bc3b3 with result from task 3958 parsl.dataflow.dflow DEBUG: Task 3958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bfa080b1-bd6c-4c62-a648-50444f07014e_isrForCrosstalkSources_855458_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/bfa080b1-bd6c-4c62-a648-50444f07014e_isrForCrosstalkSources_855458_12.stderr parsl.dataflow.dflow INFO: Task 3746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a02f2658e29229d7456d5f7e49e82cbd with result from task 3746 2024-03-09 14:55:56 proc_lsst.multi:170 [INFO] [multi] job_id 16964305 proc_lsst.multi INFO: [multi] job_id 16964305 2024-03-09 14:55:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 213 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 213 2024-03-09 14:55:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964305 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964305 parsl.executors.status_handling DEBUG: Launched block 268 on executor multi with job ID 16964305 parsl.executors.status_handling INFO: Allocated block ID 269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-269 2024-03-09 14:55:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 213 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 213 2024-03-09 14:55:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fe684390-86b3-4ce8-a576-f6506e97f75c_isrForCrosstalkSources_855417_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/fe684390-86b3-4ce8-a576-f6506e97f75c_isrForCrosstalkSources_855417_55.stderr parsl.dataflow.dflow INFO: Task 4107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38f218537d39e2ec23c65736c3a303c9 with result from task 4107 parsl.dataflow.dflow DEBUG: Task 4107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052c61d6-bbcd-4c00-8470-b5b0f1ff88c0_isrForCrosstalkSources_855451_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052c61d6-bbcd-4c00-8470-b5b0f1ff88c0_isrForCrosstalkSources_855451_45.stderr parsl.dataflow.dflow INFO: Task 3890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1aa3bb61fced47fdb6a0dd52f42b9d6 with result from task 3890 parsl.dataflow.dflow DEBUG: Task 3890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/138acd79-482a-4aa1-b62d-988bc6348dc6_isrForCrosstalkSources_855440_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/138acd79-482a-4aa1-b62d-988bc6348dc6_isrForCrosstalkSources_855440_46.stderr parsl.dataflow.dflow INFO: Task 3570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b79c3e3e292bb19b73ba2e7be422f4d0 with result from task 3570 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 3570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0f0900d6-bd6a-486b-919f-a20d0c421761_isrForCrosstalkSources_855446_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/0f0900d6-bd6a-486b-919f-a20d0c421761_isrForCrosstalkSources_855446_48.stderr parsl.dataflow.dflow INFO: Task 3916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e425a361c391de14b7a473539a96154 with result from task 3916 parsl.dataflow.dflow DEBUG: Task 3916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/32863892-a6b5-47d7-aef2-2c9b553c24d1_isrForCrosstalkSources_855379_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/32863892-a6b5-47d7-aef2-2c9b553c24d1_isrForCrosstalkSources_855379_39.stderr parsl.dataflow.dflow INFO: Task 3654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ede7c766173553d562fa2494aef3769 with result from task 3654 parsl.dataflow.dflow DEBUG: Task 3654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3c0eb17a-e61b-4502-a15d-aa553523b4c0_isrForCrosstalkSources_855414_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/3c0eb17a-e61b-4502-a15d-aa553523b4c0_isrForCrosstalkSources_855414_37.stderr parsl.dataflow.dflow INFO: Task 4031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d59ee4dd848deb61a049f2cdb612826 with result from task 4031 parsl.dataflow.dflow DEBUG: Task 4031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ba2cc7-f2ae-481d-9a7b-32c70a702556_isrForCrosstalkSources_855396_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e3ba2cc7-f2ae-481d-9a7b-32c70a702556_isrForCrosstalkSources_855396_26.stderr parsl.dataflow.dflow INFO: Task 4221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 009b68f0b7e5ebcd72e0290fcc3a8e6e with result from task 4221 parsl.dataflow.dflow DEBUG: Task 4221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/a3ed68a1-a281-4616-b1ab-115f1676ba83_isrForCrosstalkSources_855430_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/a3ed68a1-a281-4616-b1ab-115f1676ba83_isrForCrosstalkSources_855430_17.stderr parsl.dataflow.dflow INFO: Task 3853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0af29039568694a0036a3533c30f398 with result from task 3853 parsl.dataflow.dflow DEBUG: Task 3853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/bfb54fda-f97c-44c0-880f-d2d9d57abb9c_isrForCrosstalkSources_855447_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/bfb54fda-f97c-44c0-880f-d2d9d57abb9c_isrForCrosstalkSources_855447_22.stderr parsl.dataflow.dflow INFO: Task 4018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b6a6c05f3b4e524a484041266879915 with result from task 4018 parsl.dataflow.dflow DEBUG: Task 4018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/55332c2d-537a-4892-ac4c-1711c06e2178_isrForCrosstalkSources_855445_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/55332c2d-537a-4892-ac4c-1711c06e2178_isrForCrosstalkSources_855445_16.stderr parsl.dataflow.dflow INFO: Task 3906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0147dd4b84b1ec586ac4290234d23cf with result from task 3906 parsl.dataflow.dflow DEBUG: Task 3906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/46360f89-66df-48a0-979c-7e7512269e3b_isrForCrosstalkSources_855457_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/46360f89-66df-48a0-979c-7e7512269e3b_isrForCrosstalkSources_855457_26.stderr parsl.dataflow.dflow INFO: Task 3839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 952fbfa41c17a98145cc3f9942b83c8c with result from task 3839 parsl.dataflow.dflow DEBUG: Task 3839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ba469a5b-2f9d-41d9-962e-bce4c2b2526f_isrForCrosstalkSources_855379_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/ba469a5b-2f9d-41d9-962e-bce4c2b2526f_isrForCrosstalkSources_855379_13.stderr parsl.dataflow.dflow INFO: Task 3851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d88cf456a4d5406fe02ea20815977137 with result from task 3851 parsl.dataflow.dflow DEBUG: Task 3851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d924536b-caa3-412b-ade6-e79348ad582f_isrForCrosstalkSources_855422_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/d924536b-caa3-412b-ade6-e79348ad582f_isrForCrosstalkSources_855422_24.stderr parsl.dataflow.dflow INFO: Task 3711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ff14a87bcc5aa20181a3ed7d7a073a2 with result from task 3711 parsl.dataflow.dflow DEBUG: Task 3711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/77832b1e-1cdf-444c-bbdf-964668af3bb7_isrForCrosstalkSources_855459_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/77832b1e-1cdf-444c-bbdf-964668af3bb7_isrForCrosstalkSources_855459_42.stderr 2024-03-09 14:55:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964306 proc_lsst.multi INFO: [multi] job_id 16964306 2024-03-09 14:55:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 214 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 214 2024-03-09 14:55:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964306 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964306 parsl.executors.status_handling DEBUG: Launched block 269 on executor multi with job ID 16964306 parsl.executors.status_handling INFO: Allocated block ID 270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-270 2024-03-09 14:55:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 214 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 214 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43c104265d99e1cd0225c05efb8b48f with result from task 4149 parsl.dataflow.dflow DEBUG: Task 4149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/77ef7869-4cf8-44f3-907b-b2152900c221_isrForCrosstalkSources_855447_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/77ef7869-4cf8-44f3-907b-b2152900c221_isrForCrosstalkSources_855447_13.stderr parsl.dataflow.dflow INFO: Task 3995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3337740f1afba9b32c99b163ee677d07 with result from task 3995 parsl.dataflow.dflow DEBUG: Task 3995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fbe00247-6794-4f96-9f78-cb39e16be885_isrForCrosstalkSources_855458_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/fbe00247-6794-4f96-9f78-cb39e16be885_isrForCrosstalkSources_855458_28.stderr parsl.dataflow.dflow INFO: Task 4430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e635cdf22ebe3086119508915aed12 with result from task 4430 parsl.dataflow.dflow DEBUG: Task 4430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/79f9398d-762c-49f1-82a0-187dba53a126_isrForCrosstalkSources_855456_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/79f9398d-762c-49f1-82a0-187dba53a126_isrForCrosstalkSources_855456_51.stderr parsl.dataflow.dflow INFO: Task 3991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d14f20e1801729bfb3e5c749b7b0a7f with result from task 3991 parsl.dataflow.dflow DEBUG: Task 3991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/76fadd4b-0e29-46ae-852b-279f8748afda_isrForCrosstalkSources_855408_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/76fadd4b-0e29-46ae-852b-279f8748afda_isrForCrosstalkSources_855408_31.stderr parsl.dataflow.dflow INFO: Task 3997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3054501da97d685350de31c1512c5ea2 with result from task 3997 parsl.dataflow.dflow DEBUG: Task 3997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6ea8a8b8-3371-481b-af4b-3bfc920fa484_isrForCrosstalkSources_855452_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/6ea8a8b8-3371-481b-af4b-3bfc920fa484_isrForCrosstalkSources_855452_57.stderr parsl.dataflow.dflow INFO: Task 4002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef08c6630c19b77e266fb2ae543f346 with result from task 4002 parsl.dataflow.dflow DEBUG: Task 4002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6536cba7-e5f8-4c99-8d9e-2730fb2000b7_isrForCrosstalkSources_855431_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/6536cba7-e5f8-4c99-8d9e-2730fb2000b7_isrForCrosstalkSources_855431_49.stderr parsl.dataflow.dflow INFO: Task 3942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3420af97c1a719e6d8fb9537ba486180 with result from task 3942 parsl.dataflow.dflow DEBUG: Task 3942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7dd81437-5405-43cd-a533-e65b0920459d_isrForCrosstalkSources_855410_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/7dd81437-5405-43cd-a533-e65b0920459d_isrForCrosstalkSources_855410_28.stderr parsl.dataflow.dflow INFO: Task 4003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc373bf92e40c76423c33364ffad94d with result from task 4003 parsl.dataflow.dflow DEBUG: Task 4003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918a470-adad-4cd3-bd3c-fc7950ad7ec7_isrForCrosstalkSources_855404_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/e918a470-adad-4cd3-bd3c-fc7950ad7ec7_isrForCrosstalkSources_855404_10.stderr parsl.dataflow.dflow INFO: Task 4102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f7c71dec9846e3bed307b4afef7bce0 with result from task 4102 parsl.dataflow.dflow DEBUG: Task 4102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e492f7e7-0bbe-4223-961f-039a7d25d717_isrForCrosstalkSources_855394_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e492f7e7-0bbe-4223-961f-039a7d25d717_isrForCrosstalkSources_855394_61.stderr parsl.dataflow.dflow INFO: Task 4318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ea11be2ca1798fa762e4ed4d3e8a3f with result from task 4318 parsl.dataflow.dflow DEBUG: Task 4318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c64e15d0-63e1-4daf-987f-e44baedf8025_isrForCrosstalkSources_855421_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c64e15d0-63e1-4daf-987f-e44baedf8025_isrForCrosstalkSources_855421_49.stderr parsl.dataflow.dflow INFO: Task 4220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6bed900eedd2c761798a9f104a1d813 with result from task 4220 parsl.dataflow.dflow DEBUG: Task 4220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3677a386-d1ce-4ff9-9805-e3bdd473eb87_isrForCrosstalkSources_855385_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3677a386-d1ce-4ff9-9805-e3bdd473eb87_isrForCrosstalkSources_855385_6.stderr parsl.dataflow.dflow INFO: Task 4100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43d318bf481a652302e73ef7abc3e871 with result from task 4100 parsl.dataflow.dflow DEBUG: Task 4100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad694f31-0982-4200-88ab-fda15a45583c_isrForCrosstalkSources_855429_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/ad694f31-0982-4200-88ab-fda15a45583c_isrForCrosstalkSources_855429_56.stderr 2024-03-09 14:55:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964307 proc_lsst.multi INFO: [multi] job_id 16964307 2024-03-09 14:55:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 215 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 215 2024-03-09 14:55:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964307 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964307 parsl.executors.status_handling DEBUG: Launched block 270 on executor multi with job ID 16964307 parsl.executors.status_handling INFO: Allocated block ID 271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-271 2024-03-09 14:55:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 215 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 215 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 4103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f91df7a5533f29b726f6d30430f1c96 with result from task 4103 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 4103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/74e4ccba-d10d-4377-a242-1ca898904c9d_isrForCrosstalkSources_855396_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/74e4ccba-d10d-4377-a242-1ca898904c9d_isrForCrosstalkSources_855396_5.stderr parsl.dataflow.dflow INFO: Task 4109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dadfc1d6b5aff808412fbd117055081e with result from task 4109 parsl.dataflow.dflow DEBUG: Task 4109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a72b6744-cbd3-4bd8-8dd1-a5ca31b409e5_isrForCrosstalkSources_855403_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/a72b6744-cbd3-4bd8-8dd1-a5ca31b409e5_isrForCrosstalkSources_855403_40.stderr parsl.dataflow.dflow INFO: Task 4101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfc2c605464e3e4f273c9d3c7fd3669d with result from task 4101 parsl.dataflow.dflow DEBUG: Task 4101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed2412aa-8382-46f9-8ded-6fb37558b064_isrForCrosstalkSources_855419_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/ed2412aa-8382-46f9-8ded-6fb37558b064_isrForCrosstalkSources_855419_27.stderr parsl.dataflow.dflow INFO: Task 4316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77fe694e2e41a06f5057244921731df8 with result from task 4316 parsl.dataflow.dflow DEBUG: Task 4316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/5779e492-7865-4e3a-809c-3e3856837bd5_isrForCrosstalkSources_855415_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/5779e492-7865-4e3a-809c-3e3856837bd5_isrForCrosstalkSources_855415_27.stderr parsl.dataflow.dflow INFO: Task 4098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d36b79e2fe4a92e93f78f3b94a35c45 with result from task 4098 parsl.dataflow.dflow DEBUG: Task 4098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f2f75584-d56e-4b37-bade-13d869e08e6e_isrForCrosstalkSources_855424_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/f2f75584-d56e-4b37-bade-13d869e08e6e_isrForCrosstalkSources_855424_51.stderr parsl.dataflow.dflow INFO: Task 4276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa34766cedbea4da2b1d99e704c2bbc4 with result from task 4276 parsl.dataflow.dflow DEBUG: Task 4276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/00e91dfa-368c-4152-b80f-da31dccae1e5_isrForCrosstalkSources_855393_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/00e91dfa-368c-4152-b80f-da31dccae1e5_isrForCrosstalkSources_855393_14.stderr parsl.dataflow.dflow INFO: Task 4261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 277ee15874c80b7f179f3945bfc842ac with result from task 4261 parsl.dataflow.dflow DEBUG: Task 4261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5fa1d87a-6169-49f9-b8ef-55c8cbd19c41_isrForCrosstalkSources_855416_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/5fa1d87a-6169-49f9-b8ef-55c8cbd19c41_isrForCrosstalkSources_855416_31.stderr parsl.dataflow.dflow INFO: Task 4322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0d363e3675c61ea3730ebc10b3a88ab with result from task 4322 parsl.dataflow.dflow DEBUG: Task 4322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4c888835-e1ed-4def-9500-8c3a81f0fe64_isrForCrosstalkSources_855421_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/4c888835-e1ed-4def-9500-8c3a81f0fe64_isrForCrosstalkSources_855421_48.stderr parsl.dataflow.dflow INFO: Task 4262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 879a06e7720b2115facfe4e01f4b6b5a with result from task 4262 parsl.dataflow.dflow DEBUG: Task 4262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fd4bfa10-71ab-4519-b962-b68a4641431a_isrForCrosstalkSources_855409_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/fd4bfa10-71ab-4519-b962-b68a4641431a_isrForCrosstalkSources_855409_14.stderr parsl.dataflow.dflow INFO: Task 4224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4da5cea837629d95e52c0dabf9bcfa with result from task 4224 parsl.dataflow.dflow DEBUG: Task 4224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d7fe212-0e97-42bf-a4b0-3d80349a5569_isrForCrosstalkSources_855439_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d7fe212-0e97-42bf-a4b0-3d80349a5569_isrForCrosstalkSources_855439_7.stderr parsl.dataflow.dflow INFO: Task 3981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2170dd57aa0d6c334bc6530daf1cd3d8 with result from task 3981 parsl.dataflow.dflow DEBUG: Task 3981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/791c04f5-8d98-4666-bb47-a74adc1ab1d8_isrForCrosstalkSources_855439_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/791c04f5-8d98-4666-bb47-a74adc1ab1d8_isrForCrosstalkSources_855439_3.stderr parsl.dataflow.dflow INFO: Task 4301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 253b6f7cb1a3d3b4ce9144e56bca3687 with result from task 4301 parsl.dataflow.dflow DEBUG: Task 4301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a20412f8-ecd1-47a5-9575-47a06441b008_isrForCrosstalkSources_855418_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/a20412f8-ecd1-47a5-9575-47a06441b008_isrForCrosstalkSources_855418_41.stderr parsl.dataflow.dflow INFO: Task 4298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 111638c31e11a03fbce276f94b8ce829 with result from task 4298 parsl.dataflow.dflow DEBUG: Task 4298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dda2ad34-605b-463f-a99c-06f701fc3b1c_isrForCrosstalkSources_855387_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/dda2ad34-605b-463f-a99c-06f701fc3b1c_isrForCrosstalkSources_855387_25.stderr parsl.dataflow.dflow INFO: Task 4228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa438be7bc2de9db084d07cc95f602ac with result from task 4228 parsl.dataflow.dflow DEBUG: Task 4228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b6333fa2-2efc-44d9-9058-c139c82ec492_isrForCrosstalkSources_855433_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/b6333fa2-2efc-44d9-9058-c139c82ec492_isrForCrosstalkSources_855433_28.stderr parsl.dataflow.dflow INFO: Task 4162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7848470ae4e9860b26feca3fc8c559b3 with result from task 4162 parsl.dataflow.dflow DEBUG: Task 4162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a914f258-0a3d-470e-beb2-5876ad95b178_isrForCrosstalkSources_855396_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/a914f258-0a3d-470e-beb2-5876ad95b178_isrForCrosstalkSources_855396_1.stderr parsl.dataflow.dflow INFO: Task 4165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ed09d885399c271251e06c8b4826615 with result from task 4165 parsl.dataflow.dflow DEBUG: Task 4165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/658de0ff-e747-4978-a443-d63110f81e42_isrForCrosstalkSources_855409_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/658de0ff-e747-4978-a443-d63110f81e42_isrForCrosstalkSources_855409_13.stderr parsl.dataflow.dflow INFO: Task 4232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc118af96300aa0a69660ae69445448e with result from task 4232 parsl.dataflow.dflow DEBUG: Task 4232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ba2d5c5a-b0d9-4d82-ae4e-7bb7d4fa7f53_isrForCrosstalkSources_855400_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/ba2d5c5a-b0d9-4d82-ae4e-7bb7d4fa7f53_isrForCrosstalkSources_855400_42.stderr parsl.dataflow.dflow INFO: Task 4271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 844d5528d3b475336f2350beb5387297 with result from task 4271 parsl.dataflow.dflow DEBUG: Task 4271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/33e7c969-99f7-4349-988e-6178dc2d520e_isrForCrosstalkSources_855399_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/33e7c969-99f7-4349-988e-6178dc2d520e_isrForCrosstalkSources_855399_61.stderr parsl.dataflow.dflow INFO: Task 4256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db1a739d4a5cf246c2bbeb8879643457 with result from task 4256 parsl.dataflow.dflow DEBUG: Task 4256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/93f6f914-d439-4102-b52a-7be9d549e3db_isrForCrosstalkSources_855412_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/93f6f914-d439-4102-b52a-7be9d549e3db_isrForCrosstalkSources_855412_39.stderr parsl.dataflow.dflow INFO: Task 4161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7ee370e7f09816934ea41e1cb01cf1d with result from task 4161 parsl.dataflow.dflow DEBUG: Task 4161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/4a1c8034-db71-4eee-b090-3aca2c26728d_isrForCrosstalkSources_855417_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/4a1c8034-db71-4eee-b090-3aca2c26728d_isrForCrosstalkSources_855417_43.stderr parsl.dataflow.dflow INFO: Task 4115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29ffc8936e5063bc5881486f2bbf100c with result from task 4115 parsl.dataflow.dflow DEBUG: Task 4115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5db95259-5d6a-4ca9-b1bd-0f7be1018392_isrForCrosstalkSources_855430_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/5db95259-5d6a-4ca9-b1bd-0f7be1018392_isrForCrosstalkSources_855430_14.stderr parsl.dataflow.dflow INFO: Task 4178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d825addf9eaf9113e41920179e7cfa5d with result from task 4178 parsl.dataflow.dflow DEBUG: Task 4178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f987570c-0fc7-42cb-9f87-1a4789ad53bd_isrForCrosstalkSources_855412_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/f987570c-0fc7-42cb-9f87-1a4789ad53bd_isrForCrosstalkSources_855412_29.stderr parsl.dataflow.dflow INFO: Task 3982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e41bde3fe733b2598b5ce9466ca84f5 with result from task 3982 parsl.dataflow.dflow DEBUG: Task 3982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6efab700-da07-4750-b504-8d8d5987aa8b_isrForCrosstalkSources_855395_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6efab700-da07-4750-b504-8d8d5987aa8b_isrForCrosstalkSources_855395_18.stderr parsl.dataflow.dflow INFO: Task 4099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db4ca1379e3b3f3cb72549ba442b3906 with result from task 4099 parsl.dataflow.dflow DEBUG: Task 4099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/415c60a6-f85b-41e6-8fc3-4bbba030d925_isrForCrosstalkSources_855425_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/415c60a6-f85b-41e6-8fc3-4bbba030d925_isrForCrosstalkSources_855425_39.stderr parsl.dataflow.dflow INFO: Task 4244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef01d190b0d6dca1f9b529552c8bfef with result from task 4244 parsl.dataflow.dflow DEBUG: Task 4244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b5c8932e-35b8-4e4b-9e1a-345584dc7935_isrForCrosstalkSources_855435_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/b5c8932e-35b8-4e4b-9e1a-345584dc7935_isrForCrosstalkSources_855435_30.stderr parsl.dataflow.dflow INFO: Task 4233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fef50db8180ed5a3e314bd9b0b9cb4b1 with result from task 4233 parsl.dataflow.dflow DEBUG: Task 4233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/34ec0bb8-3b32-457f-8422-b70dd031bf92_isrForCrosstalkSources_855416_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/34ec0bb8-3b32-457f-8422-b70dd031bf92_isrForCrosstalkSources_855416_41.stderr parsl.dataflow.dflow INFO: Task 4004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 455973d8a35e8d0022041c3ba38cc5d7 with result from task 4004 parsl.dataflow.dflow DEBUG: Task 4004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/4afa071f-5295-4690-ac67-ea005706b10c_isrForCrosstalkSources_855447_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/4afa071f-5295-4690-ac67-ea005706b10c_isrForCrosstalkSources_855447_20.stderr parsl.dataflow.dflow INFO: Task 4105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a7f819d2969742116334d554a80441c with result from task 4105 parsl.dataflow.dflow DEBUG: Task 4105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/ec895573-e668-4b48-9aed-ce4879ca8e23_isrForCrosstalkSources_855425_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/ec895573-e668-4b48-9aed-ce4879ca8e23_isrForCrosstalkSources_855425_46.stderr parsl.dataflow.dflow INFO: Task 4160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9380fcd9f4d247ced864aa8e4f2935b4 with result from task 4160 parsl.dataflow.dflow DEBUG: Task 4160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7d8c7c2e-8f09-454d-8847-76993cec3f84_isrForCrosstalkSources_855397_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7d8c7c2e-8f09-454d-8847-76993cec3f84_isrForCrosstalkSources_855397_21.stderr parsl.dataflow.dflow INFO: Task 4211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a1a117c0025a98f7012b0b0d4937e38 with result from task 4211 parsl.dataflow.dflow DEBUG: Task 4211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f4bdb279-4a5a-4e7a-bd54-368fc063f722_isrForCrosstalkSources_855459_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f4bdb279-4a5a-4e7a-bd54-368fc063f722_isrForCrosstalkSources_855459_61.stderr parsl.dataflow.dflow INFO: Task 4044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffbc2be992361213ce67d13dea53d13d with result from task 4044 parsl.dataflow.dflow DEBUG: Task 4044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eed02406-e04d-4051-9e5c-79d785b5e41d_isrForCrosstalkSources_855423_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/eed02406-e04d-4051-9e5c-79d785b5e41d_isrForCrosstalkSources_855423_60.stderr parsl.dataflow.dflow INFO: Task 4164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 565c116a4d0a530410aee1c2d2f0749c with result from task 4164 parsl.dataflow.dflow DEBUG: Task 4164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/02d641e1-cf3d-40b4-8fe7-edb00436e476_isrForCrosstalkSources_855410_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/02d641e1-cf3d-40b4-8fe7-edb00436e476_isrForCrosstalkSources_855410_6.stderr parsl.dataflow.dflow INFO: Task 4422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca9f3174de1d5cdaa8028b3a59b96aa with result from task 4422 parsl.dataflow.dflow DEBUG: Task 4422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1b42e89f-c7bb-4c4b-a678-6dc024310e4d_isrForCrosstalkSources_855399_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/1b42e89f-c7bb-4c4b-a678-6dc024310e4d_isrForCrosstalkSources_855399_37.stderr parsl.dataflow.dflow INFO: Task 4300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 356638f89de0894f2c150e09551d8a09 with result from task 4300 parsl.dataflow.dflow DEBUG: Task 4300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/de5c6215-a127-4db6-ba02-b1a8da047f6f_isrForCrosstalkSources_855387_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/de5c6215-a127-4db6-ba02-b1a8da047f6f_isrForCrosstalkSources_855387_48.stderr parsl.dataflow.dflow INFO: Task 4461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f291b0f3ad41cb5366c990d46ebde57f with result from task 4461 parsl.dataflow.dflow DEBUG: Task 4461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/448fc953-6bd2-457b-a495-3c502d7345b9_isrForCrosstalkSources_855426_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/448fc953-6bd2-457b-a495-3c502d7345b9_isrForCrosstalkSources_855426_52.stderr parsl.dataflow.dflow INFO: Task 3984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fbe2ddb9121fbb5a838bac8f78028ca with result from task 3984 parsl.dataflow.dflow DEBUG: Task 3984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/56c6eeb3-d9e4-4777-a8df-f03f79652968_isrForCrosstalkSources_855430_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/56c6eeb3-d9e4-4777-a8df-f03f79652968_isrForCrosstalkSources_855430_10.stderr parsl.dataflow.dflow INFO: Task 4504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f170fa50c2bfa063d80737f569bffc70 with result from task 4504 parsl.dataflow.dflow DEBUG: Task 4504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/53aaec3b-978e-4ad9-8161-92003f461bf6_isrForCrosstalkSources_855421_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/53aaec3b-978e-4ad9-8161-92003f461bf6_isrForCrosstalkSources_855421_33.stderr parsl.dataflow.dflow INFO: Task 4037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a51ce63942042b45574c0c10b04f1da with result from task 4037 parsl.dataflow.dflow DEBUG: Task 4037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f3ef4a97-dcc7-4079-bc62-92e8090ce2ad_isrForCrosstalkSources_855395_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f3ef4a97-dcc7-4079-bc62-92e8090ce2ad_isrForCrosstalkSources_855395_42.stderr parsl.dataflow.dflow INFO: Task 4272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f870b456622f92bc862fee710f369a0f with result from task 4272 parsl.dataflow.dflow DEBUG: Task 4272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a08a5c21-10ab-4332-a2ee-73a9cc5152ae_isrForCrosstalkSources_855453_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/a08a5c21-10ab-4332-a2ee-73a9cc5152ae_isrForCrosstalkSources_855453_2.stderr parsl.dataflow.dflow INFO: Task 4184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e4dc5cd2001be2d2dd116a884bae561 with result from task 4184 parsl.dataflow.dflow DEBUG: Task 4184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9b7eba53-f39c-40ad-9b05-afc9c1fb64e9_isrForCrosstalkSources_855422_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/9b7eba53-f39c-40ad-9b05-afc9c1fb64e9_isrForCrosstalkSources_855422_47.stderr parsl.dataflow.dflow INFO: Task 4456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e29cf5c7fbe1bd6c5016b6c406f909a with result from task 4456 parsl.dataflow.dflow DEBUG: Task 4456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc5c6add-5038-4f63-a7cf-b6946a30764e_isrForCrosstalkSources_855450_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc5c6add-5038-4f63-a7cf-b6946a30764e_isrForCrosstalkSources_855450_37.stderr parsl.dataflow.dflow INFO: Task 4250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4635d3ceeb5788216ae3f7ccccc3311 with result from task 4250 parsl.dataflow.dflow DEBUG: Task 4250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8200eb69-bb52-4633-9672-033ee72c3a46_isrForCrosstalkSources_855399_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/8200eb69-bb52-4633-9672-033ee72c3a46_isrForCrosstalkSources_855399_18.stderr parsl.dataflow.dflow INFO: Task 4163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef0832b62e69f90a723907a4bd06477e with result from task 4163 parsl.dataflow.dflow DEBUG: Task 4163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a220c9d-e97e-425e-aee5-2d1e72df5708_isrForCrosstalkSources_855434_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/7a220c9d-e97e-425e-aee5-2d1e72df5708_isrForCrosstalkSources_855434_12.stderr parsl.dataflow.dflow INFO: Task 4086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e84fecfc724338c9864c6fc83a1c900 with result from task 4086 parsl.dataflow.dflow DEBUG: Task 4086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e4b48e2d-22d6-4800-9c4e-27ae72bfa7c1_isrForCrosstalkSources_855392_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e4b48e2d-22d6-4800-9c4e-27ae72bfa7c1_isrForCrosstalkSources_855392_32.stderr parsl.dataflow.dflow INFO: Task 3989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7297608d04f9516f0ffdd499eef7ae6f with result from task 3989 parsl.dataflow.dflow DEBUG: Task 3989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/81f20587-de80-4a88-b5a0-a3b14d7d9101_isrForCrosstalkSources_855410_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/81f20587-de80-4a88-b5a0-a3b14d7d9101_isrForCrosstalkSources_855410_45.stderr parsl.dataflow.dflow INFO: Task 4010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3849ee5974f96ecdb186431651ec0eba with result from task 4010 parsl.dataflow.dflow DEBUG: Task 4010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6607d03f-7c79-4d3e-8a7a-441a6b8dfbb9_isrForCrosstalkSources_855392_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/6607d03f-7c79-4d3e-8a7a-441a6b8dfbb9_isrForCrosstalkSources_855392_13.stderr parsl.dataflow.dflow INFO: Task 3996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59f3326a13413b0ba620f91b0823065b with result from task 3996 parsl.dataflow.dflow DEBUG: Task 3996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/57970e2b-5960-47c2-a88b-c11bb769a74c_isrForCrosstalkSources_855408_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/57970e2b-5960-47c2-a88b-c11bb769a74c_isrForCrosstalkSources_855408_53.stderr parsl.dataflow.dflow INFO: Task 4166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed2164d7fa16c67c8d4c9dbb92ce49cc with result from task 4166 parsl.dataflow.dflow DEBUG: Task 4166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a431501a-a3e9-4383-905a-0e2c7f634de1_isrForCrosstalkSources_855443_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/a431501a-a3e9-4383-905a-0e2c7f634de1_isrForCrosstalkSources_855443_40.stderr parsl.dataflow.dflow INFO: Task 3953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c899688bef4722f2ff8678a4be91ca7 with result from task 3953 parsl.dataflow.dflow DEBUG: Task 3953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f8d9ac6e-bebc-492a-935e-ec604a0e1ada_isrForCrosstalkSources_855459_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/f8d9ac6e-bebc-492a-935e-ec604a0e1ada_isrForCrosstalkSources_855459_6.stderr parsl.dataflow.dflow INFO: Task 4282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 335b4456111771bb50b1ca5aee0c3a03 with result from task 4282 parsl.dataflow.dflow DEBUG: Task 4282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0ba84ccb-1602-483b-a7d7-3b2c31092ac3_isrForCrosstalkSources_855408_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0ba84ccb-1602-483b-a7d7-3b2c31092ac3_isrForCrosstalkSources_855408_54.stderr parsl.dataflow.dflow INFO: Task 4252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f321f6a5943ac338a8454ab3d075f56 with result from task 4252 parsl.dataflow.dflow DEBUG: Task 4252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72c75e58-9a3a-4aa7-8188-708ce7c516b4_isrForCrosstalkSources_855417_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/72c75e58-9a3a-4aa7-8188-708ce7c516b4_isrForCrosstalkSources_855417_56.stderr parsl.dataflow.dflow INFO: Task 4190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0ac68cac9c4ed76b52ad63593b1991 with result from task 4190 parsl.dataflow.dflow DEBUG: Task 4190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0792edd5-eed0-4ee9-8856-73abebd07371_isrForCrosstalkSources_855390_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/0792edd5-eed0-4ee9-8856-73abebd07371_isrForCrosstalkSources_855390_24.stderr parsl.dataflow.dflow INFO: Task 4028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c6d1d1c0db33aa769fb9eccd7551993 with result from task 4028 parsl.dataflow.dflow DEBUG: Task 4028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/cb79bdfe-3abe-40f3-9b26-0cee6d6fde9c_isrForCrosstalkSources_855383_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/cb79bdfe-3abe-40f3-9b26-0cee6d6fde9c_isrForCrosstalkSources_855383_33.stderr parsl.dataflow.dflow INFO: Task 4080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 710c9611a19528ed11dc7dd1b71968f7 with result from task 4080 parsl.dataflow.dflow DEBUG: Task 4080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b6183798-e658-4507-b86b-1b692c91bf69_isrForCrosstalkSources_855411_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/b6183798-e658-4507-b86b-1b692c91bf69_isrForCrosstalkSources_855411_25.stderr parsl.dataflow.dflow INFO: Task 4173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d7e64a13ec97ecc972ddb20101156c4 with result from task 4173 parsl.dataflow.dflow DEBUG: Task 4173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8be216d1-b579-442e-bef4-a65be791c8aa_isrForCrosstalkSources_855384_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/8be216d1-b579-442e-bef4-a65be791c8aa_isrForCrosstalkSources_855384_31.stderr parsl.dataflow.dflow INFO: Task 3954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d03dccba9db98242e99338e607b73773 with result from task 3954 parsl.dataflow.dflow DEBUG: Task 3954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2e0b2885-43bc-4e5f-b848-27eb3fc6f171_isrForCrosstalkSources_855432_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/2e0b2885-43bc-4e5f-b848-27eb3fc6f171_isrForCrosstalkSources_855432_50.stderr parsl.dataflow.dflow INFO: Task 3965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac934e66eef5385cfa5951e79514a237 with result from task 3965 parsl.dataflow.dflow DEBUG: Task 3965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70e6bc59-2b56-469f-96d5-7b8079b62fb2_isrForCrosstalkSources_855443_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70e6bc59-2b56-469f-96d5-7b8079b62fb2_isrForCrosstalkSources_855443_57.stderr parsl.dataflow.dflow INFO: Task 4413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d4a64e632f63e48cfb9a8e4383f779a with result from task 4413 parsl.dataflow.dflow DEBUG: Task 4413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a1406514-23a4-4a32-bf35-8211cd1e1eb0_isrForCrosstalkSources_855387_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/a1406514-23a4-4a32-bf35-8211cd1e1eb0_isrForCrosstalkSources_855387_50.stderr parsl.dataflow.dflow INFO: Task 4176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 254c28ce7ab6df2dbd7a6ace1117256e with result from task 4176 parsl.dataflow.dflow DEBUG: Task 4176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/0d02385a-ad6d-4709-82a9-1b95b653b203_isrForCrosstalkSources_855422_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/0d02385a-ad6d-4709-82a9-1b95b653b203_isrForCrosstalkSources_855422_45.stderr parsl.dataflow.dflow INFO: Task 4379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8aab19a1e72cc053314212eb4c8e7c2 with result from task 4379 parsl.dataflow.dflow DEBUG: Task 4379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/26bfe208-086e-4142-b69f-85b83af5e597_isrForCrosstalkSources_855392_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/26bfe208-086e-4142-b69f-85b83af5e597_isrForCrosstalkSources_855392_37.stderr parsl.dataflow.dflow INFO: Task 4210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed0abba67a08c551b919d1cea1540179 with result from task 4210 2024-03-09 14:55:57 proc_lsst.multi:170 [INFO] [multi] job_id 16964308 proc_lsst.multi INFO: [multi] job_id 16964308 2024-03-09 14:55:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 216 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 216 2024-03-09 14:55:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964308 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964308 parsl.executors.status_handling DEBUG: Launched block 271 on executor multi with job ID 16964308 parsl.executors.status_handling INFO: Allocated block ID 272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-272 2024-03-09 14:55:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 216 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 216 2024-03-09 14:55:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 4210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/460df7b8-7b08-4911-bd0b-3e4e0e472295_isrForCrosstalkSources_855438_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/460df7b8-7b08-4911-bd0b-3e4e0e472295_isrForCrosstalkSources_855438_1.stderr parsl.dataflow.dflow INFO: Task 4006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 845f15a40a5262f37370b2d6e96288e5 with result from task 4006 parsl.dataflow.dflow DEBUG: Task 4006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c3bc1176-1481-4e57-9c74-8722ead872cd_isrForCrosstalkSources_855385_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c3bc1176-1481-4e57-9c74-8722ead872cd_isrForCrosstalkSources_855385_10.stderr parsl.dataflow.dflow INFO: Task 4183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881f69890bc3bc3313c9ea8f3b33f90a with result from task 4183 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 4183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/dbe6283f-5e5b-4bbf-86e7-153733950c48_isrForCrosstalkSources_855398_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/dbe6283f-5e5b-4bbf-86e7-153733950c48_isrForCrosstalkSources_855398_50.stderr parsl.dataflow.dflow INFO: Task 4042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b665d4a00175af2488a6699e54d41073 with result from task 4042 parsl.dataflow.dflow DEBUG: Task 4042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c00b1945-3fc5-485c-90b0-4efd9ac523c8_isrForCrosstalkSources_855429_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c00b1945-3fc5-485c-90b0-4efd9ac523c8_isrForCrosstalkSources_855429_37.stderr parsl.dataflow.dflow INFO: Task 4120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aae12af0132d22b8382f817df3d25af with result from task 4120 parsl.dataflow.dflow DEBUG: Task 4120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/20799d7f-c212-46da-8845-1e680d01b231_isrForCrosstalkSources_855429_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/20799d7f-c212-46da-8845-1e680d01b231_isrForCrosstalkSources_855429_49.stderr parsl.dataflow.dflow INFO: Task 4169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 922e174a5f7616f270e7ae03c431c14b with result from task 4169 parsl.dataflow.dflow DEBUG: Task 4169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c8e14736-f2b0-4a07-9768-c3f2cb8c99c7_isrForCrosstalkSources_855457_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/c8e14736-f2b0-4a07-9768-c3f2cb8c99c7_isrForCrosstalkSources_855457_11.stderr parsl.dataflow.dflow INFO: Task 4121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deaf9664a18e65ebfbdd2ca6eb9c353f with result from task 4121 parsl.dataflow.dflow DEBUG: Task 4121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2ad2f5c2-2f6b-4339-bfde-9f4387644f43_isrForCrosstalkSources_855397_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/2ad2f5c2-2f6b-4339-bfde-9f4387644f43_isrForCrosstalkSources_855397_1.stderr parsl.dataflow.dflow INFO: Task 4094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74a4cdb53dc77f2cdfe5c08a3ac0a038 with result from task 4094 parsl.dataflow.dflow DEBUG: Task 4094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/419287fe-bc22-4e99-a5a4-5e14edcff36f_isrForCrosstalkSources_855416_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/419287fe-bc22-4e99-a5a4-5e14edcff36f_isrForCrosstalkSources_855416_51.stderr parsl.dataflow.dflow INFO: Task 4185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dca36f77790c80ca60be2d851db1a56 with result from task 4185 parsl.dataflow.dflow DEBUG: Task 4185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/be43ed3c-dfc3-456c-80c3-25234e0ebef0_isrForCrosstalkSources_855448_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/be43ed3c-dfc3-456c-80c3-25234e0ebef0_isrForCrosstalkSources_855448_15.stderr parsl.dataflow.dflow INFO: Task 4179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a3641ea302e4411a55a3e64db6a3a5c with result from task 4179 parsl.dataflow.dflow DEBUG: Task 4179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/90b658c9-6884-47a6-8237-9d09847bad06_isrForCrosstalkSources_855412_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/90b658c9-6884-47a6-8237-9d09847bad06_isrForCrosstalkSources_855412_11.stderr parsl.dataflow.dflow INFO: Task 4108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24a71d327c7def9e6b2556b485ee506c with result from task 4108 parsl.dataflow.dflow DEBUG: Task 4108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/60f6d9f8-b2e7-42c1-9692-a004364752a2_isrForCrosstalkSources_855437_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/60f6d9f8-b2e7-42c1-9692-a004364752a2_isrForCrosstalkSources_855437_14.stderr parsl.dataflow.dflow INFO: Task 4175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de4f5337fb9212fff289acf9e11a561 with result from task 4175 parsl.dataflow.dflow DEBUG: Task 4175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/12a23c94-291a-4d73-85bb-a623d1cc7eac_isrForCrosstalkSources_855445_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/12a23c94-291a-4d73-85bb-a623d1cc7eac_isrForCrosstalkSources_855445_57.stderr parsl.dataflow.dflow INFO: Task 4251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d39de8e1d3ebb333813ee69bafe97cba with result from task 4251 parsl.dataflow.dflow DEBUG: Task 4251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/58a56ceb-6424-4908-873d-197ce04f23af_isrForCrosstalkSources_855443_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/58a56ceb-6424-4908-873d-197ce04f23af_isrForCrosstalkSources_855443_43.stderr parsl.dataflow.dflow INFO: Task 4097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5eaae01d95cd1971be0c7afab7351f79 with result from task 4097 parsl.dataflow.dflow DEBUG: Task 4097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/ced8bc50-5c26-4152-bbac-71bfcfa95d8d_isrForCrosstalkSources_855402_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/ced8bc50-5c26-4152-bbac-71bfcfa95d8d_isrForCrosstalkSources_855402_50.stderr parsl.dataflow.dflow INFO: Task 4253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e02a70cf44968887f744e76bd3b7d7bd with result from task 4253 parsl.dataflow.dflow DEBUG: Task 4253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/83cb7257-9b4d-49a0-9306-a364ef4e1ee7_isrForCrosstalkSources_855437_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/83cb7257-9b4d-49a0-9306-a364ef4e1ee7_isrForCrosstalkSources_855437_34.stderr parsl.dataflow.dflow INFO: Task 4111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62e387979963de31f04e68216b98e7f3 with result from task 4111 parsl.dataflow.dflow DEBUG: Task 4111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/598ce0c6-70a7-4206-beb3-3f66a1b3d88f_isrForCrosstalkSources_855402_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/598ce0c6-70a7-4206-beb3-3f66a1b3d88f_isrForCrosstalkSources_855402_44.stderr parsl.dataflow.dflow INFO: Task 4384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15cdd48327cb1bf50a98e48a54a36da4 with result from task 4384 parsl.dataflow.dflow DEBUG: Task 4384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/de92076b-d802-4278-858c-d8a45f125bd4_isrForCrosstalkSources_855389_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/de92076b-d802-4278-858c-d8a45f125bd4_isrForCrosstalkSources_855389_58.stderr parsl.dataflow.dflow INFO: Task 4357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe56f862b9db7e599647722d3168d9cc with result from task 4357 parsl.dataflow.dflow DEBUG: Task 4357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91306bef-3d40-47e2-9646-a1c315afaf9f_isrForCrosstalkSources_855390_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91306bef-3d40-47e2-9646-a1c315afaf9f_isrForCrosstalkSources_855390_31.stderr parsl.dataflow.dflow INFO: Task 3925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a97e3def911d3ff3579f0e2f9c4676fb with result from task 3925 parsl.dataflow.dflow DEBUG: Task 3925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e3ed6000-e3d5-45e1-b46b-34fd3cffb48f_isrForCrosstalkSources_855432_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e3ed6000-e3d5-45e1-b46b-34fd3cffb48f_isrForCrosstalkSources_855432_46.stderr parsl.dataflow.dflow INFO: Task 4208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db1f2224d00a71336d02436c954d7020 with result from task 4208 parsl.dataflow.dflow DEBUG: Task 4208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/02f6dd88-d138-4d63-bcef-62e411218001_isrForCrosstalkSources_855393_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/02f6dd88-d138-4d63-bcef-62e411218001_isrForCrosstalkSources_855393_31.stderr parsl.dataflow.dflow INFO: Task 4125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1bfbba2bb9a4070a4971a2229520f7b with result from task 4125 parsl.dataflow.dflow DEBUG: Task 4125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/25eb4267-0a05-4043-8d69-cac24210672b_isrForCrosstalkSources_855388_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/25eb4267-0a05-4043-8d69-cac24210672b_isrForCrosstalkSources_855388_7.stderr parsl.dataflow.dflow INFO: Task 3957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3730c4ead6fc9119837c5fabf594ebf7 with result from task 3957 parsl.dataflow.dflow DEBUG: Task 3957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ce2df744-c55c-45bf-9a49-5735fe0cb65e_isrForCrosstalkSources_855454_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/ce2df744-c55c-45bf-9a49-5735fe0cb65e_isrForCrosstalkSources_855454_61.stderr parsl.dataflow.dflow INFO: Task 3651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 955096d4d7b6583bb7e0256f86cbc6d8 with result from task 3651 parsl.dataflow.dflow DEBUG: Task 3651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de14dc26-0d83-4764-9a4a-9c6b7bbaa183_isrForCrosstalkSources_855449_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/de14dc26-0d83-4764-9a4a-9c6b7bbaa183_isrForCrosstalkSources_855449_31.stderr parsl.dataflow.dflow INFO: Task 4246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 422babeeccd11f5cc8e3fc177c90157f with result from task 4246 parsl.dataflow.dflow DEBUG: Task 4246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/56e0d3a6-7bad-43a8-be5d-965479094e94_isrForCrosstalkSources_855379_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/56e0d3a6-7bad-43a8-be5d-965479094e94_isrForCrosstalkSources_855379_7.stderr parsl.dataflow.dflow INFO: Task 3969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49769807634be64795c5dbeb855a5a23 with result from task 3969 parsl.dataflow.dflow DEBUG: Task 3969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d9a86474-5dba-49da-beb2-6aab423a15d6_isrForCrosstalkSources_855407_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/d9a86474-5dba-49da-beb2-6aab423a15d6_isrForCrosstalkSources_855407_23.stderr parsl.dataflow.dflow INFO: Task 4015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52752e7ee0158cd810257df749e02064 with result from task 4015 parsl.dataflow.dflow DEBUG: Task 4015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/05cfa153-36dd-4904-9570-3acb4d701dd3_isrForCrosstalkSources_855413_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/05cfa153-36dd-4904-9570-3acb4d701dd3_isrForCrosstalkSources_855413_35.stderr parsl.dataflow.dflow INFO: Task 3999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08ca8736a733a04fecd1268636b29ed with result from task 3999 parsl.dataflow.dflow DEBUG: Task 3999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/759fa897-4544-40b3-9a9d-85814b265fcc_isrForCrosstalkSources_855455_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/759fa897-4544-40b3-9a9d-85814b265fcc_isrForCrosstalkSources_855455_5.stderr parsl.dataflow.dflow INFO: Task 4307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f0075e94b6e34ab70ea13e472f3c07d with result from task 4307 parsl.dataflow.dflow DEBUG: Task 4307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c70b02c9-8c15-4091-ab61-78e50b4f68c8_isrForCrosstalkSources_855440_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/c70b02c9-8c15-4091-ab61-78e50b4f68c8_isrForCrosstalkSources_855440_6.stderr parsl.dataflow.dflow INFO: Task 4126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f83d0a0000ebcaae5f8d933ebea3bcfb with result from task 4126 parsl.dataflow.dflow DEBUG: Task 4126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/64448712-e368-4676-85dc-992029533586_isrForCrosstalkSources_855458_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/64448712-e368-4676-85dc-992029533586_isrForCrosstalkSources_855458_16.stderr parsl.dataflow.dflow INFO: Task 4021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6e8af2a78d13fe7ce162cf879cc5591 with result from task 4021 parsl.dataflow.dflow DEBUG: Task 4021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/4bda4d30-3e98-4628-acfc-4d52cf4a34d2_isrForCrosstalkSources_855397_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/4bda4d30-3e98-4628-acfc-4d52cf4a34d2_isrForCrosstalkSources_855397_59.stderr parsl.dataflow.dflow INFO: Task 4225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6657ddb7f3a20afaa7928639e08ac925 with result from task 4225 parsl.dataflow.dflow DEBUG: Task 4225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1eba9c1e-2367-4ffb-80c2-c4f0f2be6b9c_isrForCrosstalkSources_855431_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/1eba9c1e-2367-4ffb-80c2-c4f0f2be6b9c_isrForCrosstalkSources_855431_3.stderr parsl.dataflow.dflow INFO: Task 4291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fdf16f925f9357d52c5f2b09aaa98f1 with result from task 4291 parsl.dataflow.dflow DEBUG: Task 4291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ef82b9e6-7212-49e5-af7d-07ae9c4b9da3_isrForCrosstalkSources_855449_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/ef82b9e6-7212-49e5-af7d-07ae9c4b9da3_isrForCrosstalkSources_855449_51.stderr parsl.dataflow.dflow INFO: Task 3709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5643839584be6733ef050754bbbb5d with result from task 3709 parsl.dataflow.dflow DEBUG: Task 3709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/94b18464-8413-4f87-8696-2e62dd69e1d7_isrForCrosstalkSources_855418_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/94b18464-8413-4f87-8696-2e62dd69e1d7_isrForCrosstalkSources_855418_57.stderr parsl.dataflow.dflow INFO: Task 4205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b20429af0649b3cce84fd4cad88898a7 with result from task 4205 parsl.dataflow.dflow DEBUG: Task 4205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ba147929-c484-45b2-b4bf-063779ec3d4f_isrForCrosstalkSources_855405_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ba147929-c484-45b2-b4bf-063779ec3d4f_isrForCrosstalkSources_855405_14.stderr parsl.dataflow.dflow INFO: Task 4106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d532a0a511a5450a393f8d93f892413 with result from task 4106 parsl.dataflow.dflow DEBUG: Task 4106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2b04521c-b51a-448e-b74e-8ec48fc8c437_isrForCrosstalkSources_855392_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/2b04521c-b51a-448e-b74e-8ec48fc8c437_isrForCrosstalkSources_855392_18.stderr parsl.dataflow.dflow INFO: Task 4081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c31474b149d8a445dd3de5b8768c6c5e with result from task 4081 parsl.dataflow.dflow DEBUG: Task 4081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/20e03570-c069-4da3-9937-0d247399fac1_isrForCrosstalkSources_855415_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/20e03570-c069-4da3-9937-0d247399fac1_isrForCrosstalkSources_855415_20.stderr parsl.dataflow.dflow INFO: Task 4089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b51f7977a775099fbd13d43daec2ed14 with result from task 4089 parsl.dataflow.dflow DEBUG: Task 4089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/706b59fd-c491-4ebd-b6e3-900fdf7ff20a_isrForCrosstalkSources_855455_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/706b59fd-c491-4ebd-b6e3-900fdf7ff20a_isrForCrosstalkSources_855455_40.stderr parsl.dataflow.dflow INFO: Task 4143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b305b6859e46febf2f7b4528937737d3 with result from task 4143 parsl.dataflow.dflow DEBUG: Task 4143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3e3183e4-70ee-4e01-9bcc-be6410adf9a1_isrForCrosstalkSources_855411_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3e3183e4-70ee-4e01-9bcc-be6410adf9a1_isrForCrosstalkSources_855411_60.stderr parsl.dataflow.dflow INFO: Task 4397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b63a33c7415c9791ac57e0e3ba5a3a with result from task 4397 parsl.dataflow.dflow DEBUG: Task 4397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/56ee23b6-8516-4a5b-8789-bc9695c09d2c_isrForCrosstalkSources_855403_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/56ee23b6-8516-4a5b-8789-bc9695c09d2c_isrForCrosstalkSources_855403_62.stderr parsl.dataflow.dflow INFO: Task 4082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecd853b88ddbb48a3f011b59c3187c74 with result from task 4082 parsl.dataflow.dflow DEBUG: Task 4082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5993ac30-8b18-415c-8e1e-1633454edcec_isrForCrosstalkSources_855455_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5993ac30-8b18-415c-8e1e-1633454edcec_isrForCrosstalkSources_855455_3.stderr parsl.dataflow.dflow INFO: Task 4151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 562a2e645ab923ee64fcad1ac6b0bbf3 with result from task 4151 parsl.dataflow.dflow DEBUG: Task 4151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3080f4d8-1cce-4f31-9fc7-a59ddd877062_isrForCrosstalkSources_855454_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/3080f4d8-1cce-4f31-9fc7-a59ddd877062_isrForCrosstalkSources_855454_4.stderr parsl.dataflow.dflow INFO: Task 3943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd02b4f0354d915c04e6cbedfd1b1e76 with result from task 3943 parsl.dataflow.dflow DEBUG: Task 3943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/68644483-ade7-4182-b667-eb0ccfcf5804_isrForCrosstalkSources_855436_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/68644483-ade7-4182-b667-eb0ccfcf5804_isrForCrosstalkSources_855436_54.stderr parsl.dataflow.dflow INFO: Task 4090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed6eeca1c07f14cbebda5ef84095760 with result from task 4090 parsl.dataflow.dflow DEBUG: Task 4090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/dc7e84e0-1546-45f6-858b-eef5a4c83835_isrForCrosstalkSources_855428_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/dc7e84e0-1546-45f6-858b-eef5a4c83835_isrForCrosstalkSources_855428_10.stderr parsl.dataflow.dflow INFO: Task 4128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ec59672626f30ab9662fb2b43d804c with result from task 4128 parsl.dataflow.dflow DEBUG: Task 4128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c18772c5-7925-4d52-bbb0-21e2282e7e90_isrForCrosstalkSources_855411_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/c18772c5-7925-4d52-bbb0-21e2282e7e90_isrForCrosstalkSources_855411_47.stderr parsl.dataflow.dflow INFO: Task 4180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfcb2104c26e26a57abcfd31190b094b with result from task 4180 parsl.dataflow.dflow DEBUG: Task 4180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/b7ad6cfd-5d12-479b-8c00-221924ecad5a_isrForCrosstalkSources_855434_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/b7ad6cfd-5d12-479b-8c00-221924ecad5a_isrForCrosstalkSources_855434_5.stderr parsl.dataflow.dflow INFO: Task 4020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50e02ff0554d61c71706613c4bd20ed9 with result from task 4020 parsl.dataflow.dflow DEBUG: Task 4020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0780edad-64a7-402b-b6e7-bed3e064e4c7_isrForCrosstalkSources_855420_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/0780edad-64a7-402b-b6e7-bed3e064e4c7_isrForCrosstalkSources_855420_39.stderr parsl.dataflow.dflow INFO: Task 3963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8561e7147c4c9c6553517d92e786bd27 with result from task 3963 parsl.dataflow.dflow DEBUG: Task 3963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/9945782e-a4e6-4185-ae57-d1cc09a7586b_isrForCrosstalkSources_855421_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/9945782e-a4e6-4185-ae57-d1cc09a7586b_isrForCrosstalkSources_855421_47.stderr parsl.dataflow.dflow INFO: Task 4158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a346aff27f695ab4c97774180170413d with result from task 4158 parsl.dataflow.dflow DEBUG: Task 4158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/2e1445f8-2549-42a5-bdea-7aae5e8f54ce_isrForCrosstalkSources_855438_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/2e1445f8-2549-42a5-bdea-7aae5e8f54ce_isrForCrosstalkSources_855438_31.stderr parsl.dataflow.dflow INFO: Task 4201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 097d8942503257d484a7b34be36371ba with result from task 4201 parsl.dataflow.dflow DEBUG: Task 4201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9348bb26-3850-421c-96fb-364e5234d9f7_isrForCrosstalkSources_855399_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9348bb26-3850-421c-96fb-364e5234d9f7_isrForCrosstalkSources_855399_15.stderr parsl.dataflow.dflow INFO: Task 4216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3001a92778606409a0c8d6c72311fc9 with result from task 4216 parsl.dataflow.dflow DEBUG: Task 4216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/169226f0-abe0-44b1-8bfb-a785c8de45bd_isrForCrosstalkSources_855446_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/169226f0-abe0-44b1-8bfb-a785c8de45bd_isrForCrosstalkSources_855446_35.stderr parsl.dataflow.dflow INFO: Task 3992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6094d77770b3697707046100d64cd5 with result from task 3992 parsl.dataflow.dflow DEBUG: Task 3992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/47dcad13-5bfd-4a0d-b8d2-204b2d4c94ee_isrForCrosstalkSources_855400_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/47dcad13-5bfd-4a0d-b8d2-204b2d4c94ee_isrForCrosstalkSources_855400_29.stderr parsl.dataflow.dflow INFO: Task 4207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0258e8910276ed3447a05ffda13a58ff with result from task 4207 parsl.dataflow.dflow DEBUG: Task 4207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/054940b6-48ff-4850-a47d-c45fec882897_isrForCrosstalkSources_855459_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/054940b6-48ff-4850-a47d-c45fec882897_isrForCrosstalkSources_855459_55.stderr parsl.dataflow.dflow INFO: Task 4139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89b23fd776f3b9a07729d6654122b666 with result from task 4139 parsl.dataflow.dflow DEBUG: Task 4139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/96c4d253-85b3-415d-bc04-4e14c262de9b_isrForCrosstalkSources_855457_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/96c4d253-85b3-415d-bc04-4e14c262de9b_isrForCrosstalkSources_855457_12.stderr parsl.dataflow.dflow INFO: Task 4057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc0aecc5801c51d9b0160d434c67afa2 with result from task 4057 parsl.dataflow.dflow DEBUG: Task 4057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f43dfe17-38d6-4e55-a0a3-9fececf0f44f_isrForCrosstalkSources_855407_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f43dfe17-38d6-4e55-a0a3-9fececf0f44f_isrForCrosstalkSources_855407_4.stderr parsl.dataflow.dflow INFO: Task 4297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba27e8582495a14cf06bdf2945651797 with result from task 4297 parsl.dataflow.dflow DEBUG: Task 4297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e97547da-140d-4e04-8a67-9f0da856fcee_isrForCrosstalkSources_855439_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e97547da-140d-4e04-8a67-9f0da856fcee_isrForCrosstalkSources_855439_10.stderr parsl.dataflow.dflow INFO: Task 4204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef51fb283dfe1dfafd846c884b297fc5 with result from task 4204 parsl.dataflow.dflow DEBUG: Task 4204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d383262-ba6b-452e-a5db-2d1e561f4193_isrForCrosstalkSources_855400_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/1d383262-ba6b-452e-a5db-2d1e561f4193_isrForCrosstalkSources_855400_35.stderr parsl.dataflow.dflow INFO: Task 4146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cadec2a6d357de2060c629103aff49f5 with result from task 4146 parsl.dataflow.dflow DEBUG: Task 4146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1d5ed7f7-ec75-4e70-ab09-a83d491c763f_isrForCrosstalkSources_855439_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/1d5ed7f7-ec75-4e70-ab09-a83d491c763f_isrForCrosstalkSources_855439_50.stderr parsl.dataflow.dflow INFO: Task 4354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34fcffefe719bd4ff1b48e754977a9b2 with result from task 4354 parsl.dataflow.dflow DEBUG: Task 4354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c35cbf34-ebe3-4364-a94e-3c17139cd353_isrForCrosstalkSources_855442_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c35cbf34-ebe3-4364-a94e-3c17139cd353_isrForCrosstalkSources_855442_12.stderr parsl.dataflow.dflow INFO: Task 4308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6be9ce9302c3b77783f335ed8a427d4 with result from task 4308 parsl.dataflow.dflow DEBUG: Task 4308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/392b882f-3600-484f-9748-13cb40acaafc_isrForCrosstalkSources_855382_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/392b882f-3600-484f-9748-13cb40acaafc_isrForCrosstalkSources_855382_39.stderr parsl.dataflow.dflow INFO: Task 4132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c7bb71b194818d229510a2c2332ba6b with result from task 4132 parsl.dataflow.dflow DEBUG: Task 4132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/eee5526f-bf37-43d5-819a-834359de4aa3_isrForCrosstalkSources_855432_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/eee5526f-bf37-43d5-819a-834359de4aa3_isrForCrosstalkSources_855432_10.stderr parsl.dataflow.dflow INFO: Task 4131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 889ccd3c80ab379be015a7d698a99105 with result from task 4131 parsl.dataflow.dflow DEBUG: Task 4131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d20a267f-09d8-42a1-b383-ff841b6bd42a_isrForCrosstalkSources_855457_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/d20a267f-09d8-42a1-b383-ff841b6bd42a_isrForCrosstalkSources_855457_38.stderr parsl.dataflow.dflow INFO: Task 4127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec701c04306ef97dc8d1fe899d5be7bc with result from task 4127 parsl.dataflow.dflow DEBUG: Task 4127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d629a469-b3b8-4250-93e7-39d6d91fd69b_isrForCrosstalkSources_855435_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/d629a469-b3b8-4250-93e7-39d6d91fd69b_isrForCrosstalkSources_855435_9.stderr parsl.dataflow.dflow INFO: Task 4005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 814232a1a11c84f460958a26ca054629 with result from task 4005 parsl.dataflow.dflow DEBUG: Task 4005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/25f38ffe-2c82-4fcd-81b9-d97765cf6745_isrForCrosstalkSources_855455_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/25f38ffe-2c82-4fcd-81b9-d97765cf6745_isrForCrosstalkSources_855455_58.stderr parsl.dataflow.dflow INFO: Task 4412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dccacc65852845092c97f80632dbf8d2 with result from task 4412 parsl.dataflow.dflow DEBUG: Task 4412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9a95bac9-ff8d-491c-bee2-c2cfc51d275f_isrForCrosstalkSources_855444_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/9a95bac9-ff8d-491c-bee2-c2cfc51d275f_isrForCrosstalkSources_855444_29.stderr parsl.dataflow.dflow INFO: Task 4147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01732ec1fc47b36b09d46193049192dc with result from task 4147 parsl.dataflow.dflow DEBUG: Task 4147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/118d71b1-6a56-425b-a403-bdc80f4056f4_isrForCrosstalkSources_855414_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/118d71b1-6a56-425b-a403-bdc80f4056f4_isrForCrosstalkSources_855414_22.stderr parsl.dataflow.dflow INFO: Task 4124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48bc0ed720e3e102e834397ac2aa922e with result from task 4124 parsl.dataflow.dflow DEBUG: Task 4124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f698743d-16c5-46a6-8964-82992d2db3ba_isrForCrosstalkSources_855382_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/f698743d-16c5-46a6-8964-82992d2db3ba_isrForCrosstalkSources_855382_43.stderr parsl.dataflow.dflow INFO: Task 4192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d519b4d114d4ece8e02b9bf1074c33cb with result from task 4192 parsl.dataflow.dflow DEBUG: Task 4192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/262640a6-1283-47c2-8696-98df33b3acaa_isrForCrosstalkSources_855426_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/262640a6-1283-47c2-8696-98df33b3acaa_isrForCrosstalkSources_855426_59.stderr parsl.dataflow.dflow INFO: Task 4114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39ca224cfbf42aac166b9e252cbeff3f with result from task 4114 parsl.dataflow.dflow DEBUG: Task 4114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/53ddf3cb-5cae-48cf-a923-9005c3f02260_isrForCrosstalkSources_855427_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/53ddf3cb-5cae-48cf-a923-9005c3f02260_isrForCrosstalkSources_855427_42.stderr parsl.dataflow.dflow INFO: Task 4093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 628cef5f9ebd1dceef0262367bfe32ad with result from task 4093 parsl.dataflow.dflow DEBUG: Task 4093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5a9ea3ce-a178-4212-b564-4c0a51a3d79c_isrForCrosstalkSources_855405_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5a9ea3ce-a178-4212-b564-4c0a51a3d79c_isrForCrosstalkSources_855405_28.stderr parsl.dataflow.dflow INFO: Task 4153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 683c2757186c1c5461d4eb8b4882024e with result from task 4153 parsl.dataflow.dflow DEBUG: Task 4153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/45e34636-ec02-40de-b586-d632af0a6300_isrForCrosstalkSources_855383_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/45e34636-ec02-40de-b586-d632af0a6300_isrForCrosstalkSources_855383_3.stderr parsl.dataflow.dflow INFO: Task 4157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 501f44d41fb78b404aa93c3ce7c1e64b with result from task 4157 parsl.dataflow.dflow DEBUG: Task 4157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/ed47a31d-69f0-4cec-b149-57e1b66f026a_isrForCrosstalkSources_855423_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/ed47a31d-69f0-4cec-b149-57e1b66f026a_isrForCrosstalkSources_855423_35.stderr parsl.dataflow.dflow INFO: Task 3921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f55f510e4eb66367746f8a8cc0a7b8c with result from task 3921 parsl.dataflow.dflow DEBUG: Task 3921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5fea71a8-98ed-4bef-8561-83bf2883842b_isrForCrosstalkSources_855402_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/5fea71a8-98ed-4bef-8561-83bf2883842b_isrForCrosstalkSources_855402_2.stderr parsl.dataflow.dflow INFO: Task 4335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b57944e2d17918da0f70739f5d90d9c with result from task 4335 parsl.dataflow.dflow DEBUG: Task 4335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0d471279-df8a-4989-a03f-a18ad4fb21e9_isrForCrosstalkSources_855441_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0d471279-df8a-4989-a03f-a18ad4fb21e9_isrForCrosstalkSources_855441_8.stderr parsl.dataflow.dflow INFO: Task 4118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8beb584da233a316425fd4909b54991d with result from task 4118 parsl.dataflow.dflow DEBUG: Task 4118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/75effb3a-1910-45f8-819c-f48cb8ec3925_isrForCrosstalkSources_855441_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/75effb3a-1910-45f8-819c-f48cb8ec3925_isrForCrosstalkSources_855441_58.stderr parsl.dataflow.dflow INFO: Task 4315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19a53b6502c92841cf183b4d766c36f7 with result from task 4315 parsl.dataflow.dflow DEBUG: Task 4315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a6b04538-33c6-4edf-9798-3fcfb7c8f696_isrForCrosstalkSources_855451_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/a6b04538-33c6-4edf-9798-3fcfb7c8f696_isrForCrosstalkSources_855451_10.stderr parsl.dataflow.dflow INFO: Task 4195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfb272677264fe7aff0b7a306b084727 with result from task 4195 parsl.dataflow.dflow DEBUG: Task 4195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/0e630fbb-87c3-4528-9292-8fd6977a4198_isrForCrosstalkSources_855449_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/0e630fbb-87c3-4528-9292-8fd6977a4198_isrForCrosstalkSources_855449_60.stderr parsl.dataflow.dflow INFO: Task 3926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fc2c52a20eeebd6d2e27f893ea540c2 with result from task 3926 parsl.dataflow.dflow DEBUG: Task 3926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/288d04a6-f452-48b3-843e-67ec515a9e30_isrForCrosstalkSources_855389_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/288d04a6-f452-48b3-843e-67ec515a9e30_isrForCrosstalkSources_855389_38.stderr parsl.dataflow.dflow INFO: Task 3998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d969036b297bfbfadbc68bcd9029183 with result from task 3998 parsl.dataflow.dflow DEBUG: Task 3998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7fa07982-0e76-4e19-ac8d-9a0376f87498_isrForCrosstalkSources_855390_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/7fa07982-0e76-4e19-ac8d-9a0376f87498_isrForCrosstalkSources_855390_48.stderr parsl.dataflow.dflow INFO: Task 3922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07a15b8898a8a0056cc35c12d35fca20 with result from task 3922 2024-03-09 14:55:58 proc_lsst.multi:170 [INFO] [multi] job_id 16964309 proc_lsst.multi INFO: [multi] job_id 16964309 2024-03-09 14:55:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 217 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 217 2024-03-09 14:55:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964309 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964309 parsl.executors.status_handling DEBUG: Launched block 272 on executor multi with job ID 16964309 parsl.executors.status_handling INFO: Allocated block ID 273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-273 2024-03-09 14:55:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 217 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 217 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 3922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/84684709-caa1-4b20-bca1-736c79a58a9f_isrForCrosstalkSources_855384_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/84684709-caa1-4b20-bca1-736c79a58a9f_isrForCrosstalkSources_855384_54.stderr parsl.dataflow.dflow INFO: Task 4145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82a67901b9d70ca186605d7a235b93e3 with result from task 4145 parsl.dataflow.dflow DEBUG: Task 4145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1de72c2a-faee-4395-a754-2c1223d3cd15_isrForCrosstalkSources_855381_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/1de72c2a-faee-4395-a754-2c1223d3cd15_isrForCrosstalkSources_855381_37.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e65b163fa59be9f41af589288ec505e4 with result from task 3788 parsl.dataflow.dflow DEBUG: Task 3788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e76eca89-3e0d-400e-b3af-e136275b5f4d_isrForCrosstalkSources_855456_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e76eca89-3e0d-400e-b3af-e136275b5f4d_isrForCrosstalkSources_855456_60.stderr parsl.dataflow.dflow INFO: Task 4116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77e5a2d66003ddccdfa8c5e2fd19d2a4 with result from task 4116 parsl.dataflow.dflow DEBUG: Task 4116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14d92a46-d586-45b0-b325-35d018e33b70_isrForCrosstalkSources_855433_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/14d92a46-d586-45b0-b325-35d018e33b70_isrForCrosstalkSources_855433_51.stderr parsl.dataflow.dflow INFO: Task 4263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2c439adf0fc3ea27aa60b401002214 with result from task 4263 parsl.dataflow.dflow DEBUG: Task 4263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c51382cb-0adc-47d7-995a-2cbe18822d3d_isrForCrosstalkSources_855395_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/c51382cb-0adc-47d7-995a-2cbe18822d3d_isrForCrosstalkSources_855395_60.stderr parsl.dataflow.dflow INFO: Task 4095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5daa816b67903216d9787d285df8acf8 with result from task 4095 parsl.dataflow.dflow DEBUG: Task 4095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b843cc33-9fac-4c08-aafb-f292d59a07db_isrForCrosstalkSources_855452_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b843cc33-9fac-4c08-aafb-f292d59a07db_isrForCrosstalkSources_855452_22.stderr parsl.dataflow.dflow INFO: Task 4187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a29d97be5572504a652175e11b1c713e with result from task 4187 parsl.dataflow.dflow DEBUG: Task 4187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/6068ea66-3b53-4491-bdfd-0958e520c5ed_isrForCrosstalkSources_855422_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/6068ea66-3b53-4491-bdfd-0958e520c5ed_isrForCrosstalkSources_855422_55.stderr parsl.dataflow.dflow INFO: Task 4196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7d6bc8be8f71797b94d3570011d8e49 with result from task 4196 parsl.dataflow.dflow DEBUG: Task 4196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f1a4ab5-c3cf-4e15-8121-0791ed0cfa2e_isrForCrosstalkSources_855441_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1f1a4ab5-c3cf-4e15-8121-0791ed0cfa2e_isrForCrosstalkSources_855441_46.stderr parsl.dataflow.dflow INFO: Task 4137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1280a3ec8ac33876057cfa4a057f826 with result from task 4137 parsl.dataflow.dflow DEBUG: Task 4137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cc192868-2694-4336-84ea-0fbcf59fcfbd_isrForCrosstalkSources_855415_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cc192868-2694-4336-84ea-0fbcf59fcfbd_isrForCrosstalkSources_855415_4.stderr parsl.dataflow.dflow INFO: Task 4393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41397e1c46997cc2af6fa0606fb58ea4 with result from task 4393 parsl.dataflow.dflow DEBUG: Task 4393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ad25e0d0-81dd-412c-a105-1d219dc9e66c_isrForCrosstalkSources_855409_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ad25e0d0-81dd-412c-a105-1d219dc9e66c_isrForCrosstalkSources_855409_4.stderr parsl.dataflow.dflow INFO: Task 4096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b1cd599b5333c97071e074c317b0a42 with result from task 4096 parsl.dataflow.dflow DEBUG: Task 4096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/209b97e8-bb03-440a-8fa3-6f2208b0c7f4_isrForCrosstalkSources_855426_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/209b97e8-bb03-440a-8fa3-6f2208b0c7f4_isrForCrosstalkSources_855426_7.stderr parsl.dataflow.dflow INFO: Task 4494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121b959ba17dca127d37e2e17495a2a1 with result from task 4494 parsl.dataflow.dflow DEBUG: Task 4494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/751de754-0798-4dc8-a72d-d053ed6d0426_isrForCrosstalkSources_855451_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/751de754-0798-4dc8-a72d-d053ed6d0426_isrForCrosstalkSources_855451_24.stderr parsl.dataflow.dflow INFO: Task 3862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aac385dcf72101f59c0b8b2d24fdc219 with result from task 3862 parsl.dataflow.dflow DEBUG: Task 3862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e3da1462-1419-49d7-9deb-f436f47c131d_isrForCrosstalkSources_855456_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/e3da1462-1419-49d7-9deb-f436f47c131d_isrForCrosstalkSources_855456_59.stderr parsl.dataflow.dflow INFO: Task 4117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcfe7bc37104518d2be6b0375ba00ee1 with result from task 4117 parsl.dataflow.dflow DEBUG: Task 4117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b9317006-da27-49c5-a8a8-82b2867c9aa3_isrForCrosstalkSources_855398_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b9317006-da27-49c5-a8a8-82b2867c9aa3_isrForCrosstalkSources_855398_18.stderr parsl.dataflow.dflow INFO: Task 4122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 611804a3950217edaa7eea70cfb17aae with result from task 4122 parsl.dataflow.dflow DEBUG: Task 4122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3f99e8ff-7d44-4635-affe-5ad3cc01375e_isrForCrosstalkSources_855402_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/3f99e8ff-7d44-4635-affe-5ad3cc01375e_isrForCrosstalkSources_855402_36.stderr parsl.dataflow.dflow INFO: Task 4032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8077113e031aaf1e3530981143308cb7 with result from task 4032 parsl.dataflow.dflow DEBUG: Task 4032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9ce74088-1fef-438d-8708-0eee8edac9ea_isrForCrosstalkSources_855399_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9ce74088-1fef-438d-8708-0eee8edac9ea_isrForCrosstalkSources_855399_11.stderr parsl.dataflow.dflow INFO: Task 3917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a20a006cca42d4d20a03bf58a1d5388 with result from task 3917 parsl.dataflow.dflow DEBUG: Task 3917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6d4e6402-a013-4885-832e-20d240909f61_isrForCrosstalkSources_855445_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/6d4e6402-a013-4885-832e-20d240909f61_isrForCrosstalkSources_855445_20.stderr parsl.dataflow.dflow INFO: Task 4001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d371d6291560ca4fd5c1510d079e80b7 with result from task 4001 parsl.dataflow.dflow DEBUG: Task 4001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85e75c3c-ed7d-4128-8970-cb9a4651dc36_isrForCrosstalkSources_855413_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/85e75c3c-ed7d-4128-8970-cb9a4651dc36_isrForCrosstalkSources_855413_24.stderr parsl.dataflow.dflow INFO: Task 3863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49efa70eb3f128a002de67c99e737b15 with result from task 3863 parsl.dataflow.dflow DEBUG: Task 3863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/230f2c7d-05be-4885-ab1f-3085930f9ddf_isrForCrosstalkSources_855451_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/230f2c7d-05be-4885-ab1f-3085930f9ddf_isrForCrosstalkSources_855451_28.stderr parsl.dataflow.dflow INFO: Task 4119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 202050e2c177afd388b0b9f6be3baf58 with result from task 4119 parsl.dataflow.dflow DEBUG: Task 4119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/353576fd-e977-4151-8f6a-590ba893de6d_isrForCrosstalkSources_855429_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/353576fd-e977-4151-8f6a-590ba893de6d_isrForCrosstalkSources_855429_24.stderr parsl.dataflow.dflow INFO: Task 4017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4d1af51e8e2efe4ee2525d05636d828 with result from task 4017 parsl.dataflow.dflow DEBUG: Task 4017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/45200f85-6923-4ab7-aaea-95f8f3519044_isrForCrosstalkSources_855429_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/45200f85-6923-4ab7-aaea-95f8f3519044_isrForCrosstalkSources_855429_34.stderr parsl.dataflow.dflow INFO: Task 4343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94639724a823488051d76489d4424a19 with result from task 4343 parsl.dataflow.dflow DEBUG: Task 4343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/45da5ece-174d-4d1d-8bff-e1019e5c055e_isrForCrosstalkSources_855386_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/45da5ece-174d-4d1d-8bff-e1019e5c055e_isrForCrosstalkSources_855386_45.stderr parsl.dataflow.dflow INFO: Task 4339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9755881d4f11256096ff71bc9de4879e with result from task 4339 parsl.dataflow.dflow DEBUG: Task 4339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/84bfc195-7682-4c27-89d2-be325a2d8cea_isrForCrosstalkSources_855381_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/84bfc195-7682-4c27-89d2-be325a2d8cea_isrForCrosstalkSources_855381_33.stderr parsl.dataflow.dflow INFO: Task 4024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97f306ecbc0da7aa49ca034aa52c24cf with result from task 4024 parsl.dataflow.dflow DEBUG: Task 4024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70b0af10-085b-48e7-9888-19d77e079edc_isrForCrosstalkSources_855443_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/70b0af10-085b-48e7-9888-19d77e079edc_isrForCrosstalkSources_855443_37.stderr parsl.dataflow.dflow INFO: Task 4059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f586cb3087dcfa1fe3d629dbc3feff4 with result from task 4059 parsl.dataflow.dflow DEBUG: Task 4059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/251396bd-9553-4ed8-88fb-995f5c59f071_isrForCrosstalkSources_855442_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/251396bd-9553-4ed8-88fb-995f5c59f071_isrForCrosstalkSources_855442_50.stderr parsl.dataflow.dflow INFO: Task 4395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276c8f66ac6c0285f5fc33397baa807a with result from task 4395 parsl.dataflow.dflow DEBUG: Task 4395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c27df980-0470-4dca-93e6-fac8ca858183_isrForCrosstalkSources_855387_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c27df980-0470-4dca-93e6-fac8ca858183_isrForCrosstalkSources_855387_20.stderr parsl.dataflow.dflow INFO: Task 3918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46cdcc4a11cfa83129d416baac974896 with result from task 3918 parsl.dataflow.dflow DEBUG: Task 3918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/463be0c1-97e4-4059-8a20-75bc6bcbe0dc_isrForCrosstalkSources_855417_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/463be0c1-97e4-4059-8a20-75bc6bcbe0dc_isrForCrosstalkSources_855417_11.stderr parsl.dataflow.dflow INFO: Task 4148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbb6e2b8fc31e19d1b6fb1261a908faf with result from task 4148 parsl.dataflow.dflow DEBUG: Task 4148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/41297312-710a-48d0-8dbf-a6042b172345_isrForCrosstalkSources_855440_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/41297312-710a-48d0-8dbf-a6042b172345_isrForCrosstalkSources_855440_24.stderr parsl.dataflow.dflow INFO: Task 4199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ebcb2c66bbd39703fa4e825c555070b with result from task 4199 parsl.dataflow.dflow DEBUG: Task 4199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8680142e-17d6-4ec5-853e-5fe9272f74af_isrForCrosstalkSources_855454_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8680142e-17d6-4ec5-853e-5fe9272f74af_isrForCrosstalkSources_855454_54.stderr parsl.dataflow.dflow INFO: Task 4012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c364ce1ca3e12eb6392adb9cf97e1b2 with result from task 4012 parsl.dataflow.dflow DEBUG: Task 4012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9cc0025c-405a-4aeb-8d70-2da3c3e40e5a_isrForCrosstalkSources_855434_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9cc0025c-405a-4aeb-8d70-2da3c3e40e5a_isrForCrosstalkSources_855434_61.stderr parsl.dataflow.dflow INFO: Task 4214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d72415f973f1f8565ca4a984d739bd1 with result from task 4214 parsl.dataflow.dflow DEBUG: Task 4214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/714e3474-f843-41ec-8faa-0ed5cc70dbc4_isrForCrosstalkSources_855432_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/714e3474-f843-41ec-8faa-0ed5cc70dbc4_isrForCrosstalkSources_855432_61.stderr parsl.dataflow.dflow INFO: Task 4203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33e5189863f6985a821958ee45136cb3 with result from task 4203 parsl.dataflow.dflow DEBUG: Task 4203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ecee7586-d847-4160-a244-8af8fe195fc8_isrForCrosstalkSources_855442_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/ecee7586-d847-4160-a244-8af8fe195fc8_isrForCrosstalkSources_855442_45.stderr parsl.dataflow.dflow INFO: Task 4152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 605ddd9b358d4aa647a52f9bad81131f with result from task 4152 parsl.dataflow.dflow DEBUG: Task 4152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/e7110059-3296-4f5f-9bf8-002c6c76eec6_isrForCrosstalkSources_855448_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/e7110059-3296-4f5f-9bf8-002c6c76eec6_isrForCrosstalkSources_855448_5.stderr parsl.dataflow.dflow INFO: Task 4202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37d28a47cb2be03bcbddf272503ae9f3 with result from task 4202 parsl.dataflow.dflow DEBUG: Task 4202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/37c1d55a-1182-4296-936c-edcaa3275641_isrForCrosstalkSources_855406_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/37c1d55a-1182-4296-936c-edcaa3275641_isrForCrosstalkSources_855406_52.stderr parsl.dataflow.dflow INFO: Task 4227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24dbe3d69888e99b93994cdb02ff1dbd with result from task 4227 parsl.dataflow.dflow DEBUG: Task 4227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/495d90a6-889f-4904-8382-1bc1e49d2355_isrForCrosstalkSources_855379_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/495d90a6-889f-4904-8382-1bc1e49d2355_isrForCrosstalkSources_855379_45.stderr parsl.dataflow.dflow INFO: Task 4156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc7a8eb5a2c85bcf42a7f43d20daf24 with result from task 4156 parsl.dataflow.dflow DEBUG: Task 4156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c26332c2-7f3a-45f2-870e-b954f9c5ba46_isrForCrosstalkSources_855397_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c26332c2-7f3a-45f2-870e-b954f9c5ba46_isrForCrosstalkSources_855397_4.stderr parsl.dataflow.dflow INFO: Task 4191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afef275d12c1fa67270fefcb09aa956b with result from task 4191 parsl.dataflow.dflow DEBUG: Task 4191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c6b4eda9-dece-4f30-8f7b-06f21aabf110_isrForCrosstalkSources_855387_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c6b4eda9-dece-4f30-8f7b-06f21aabf110_isrForCrosstalkSources_855387_51.stderr parsl.dataflow.dflow INFO: Task 4217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d4f41fa0d8e6f39a779d89d92dac9d4 with result from task 4217 parsl.dataflow.dflow DEBUG: Task 4217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fce061c7-de25-44d4-8d6b-23e76b19c46c_isrForCrosstalkSources_855385_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fce061c7-de25-44d4-8d6b-23e76b19c46c_isrForCrosstalkSources_855385_47.stderr parsl.dataflow.dflow INFO: Task 4215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b31daf319f1848fc5e789d01208047e with result from task 4215 parsl.dataflow.dflow DEBUG: Task 4215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/724e32b7-9195-48f7-a7c6-4b4318242370_isrForCrosstalkSources_855388_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/724e32b7-9195-48f7-a7c6-4b4318242370_isrForCrosstalkSources_855388_55.stderr 2024-03-09 14:55:58 proc_lsst.multi:170 [INFO] [multi] job_id 16964310 proc_lsst.multi INFO: [multi] job_id 16964310 2024-03-09 14:55:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 218 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 218 2024-03-09 14:55:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964310 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964310 parsl.executors.status_handling DEBUG: Launched block 273 on executor multi with job ID 16964310 parsl.executors.status_handling INFO: Allocated block ID 274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-274 2024-03-09 14:55:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 218 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 218 2024-03-09 14:55:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:55:59 proc_lsst.multi:170 [INFO] [multi] job_id 16964311 proc_lsst.multi INFO: [multi] job_id 16964311 2024-03-09 14:55:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 219 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 219 2024-03-09 14:55:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964311 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964311 parsl.executors.status_handling DEBUG: Launched block 274 on executor multi with job ID 16964311 parsl.executors.status_handling INFO: Allocated block ID 275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-275 2024-03-09 14:55:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 219 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 219 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:55:59 proc_lsst.multi:170 [INFO] [multi] job_id 16964312 proc_lsst.multi INFO: [multi] job_id 16964312 2024-03-09 14:55:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 220 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 220 2024-03-09 14:55:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964312 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964312 parsl.executors.status_handling DEBUG: Launched block 275 on executor multi with job ID 16964312 parsl.executors.status_handling INFO: Allocated block ID 276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-276 2024-03-09 14:55:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:55:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:55:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 220 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 220 2024-03-09 14:55:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:55:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea88e04bd949e94d411a3141dc79d50f with result from task 4462 parsl.dataflow.dflow DEBUG: Task 4462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6a98b60d-dee1-4a41-ad76-c92a59979f81_isrForCrosstalkSources_855429_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/6a98b60d-dee1-4a41-ad76-c92a59979f81_isrForCrosstalkSources_855429_3.stderr parsl.dataflow.dflow INFO: Task 4446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ea5544ffbf3c27f29abd5391fb70337 with result from task 4446 parsl.dataflow.dflow DEBUG: Task 4446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b444e26-211d-4c34-aff5-8cc3fee68fb2_isrForCrosstalkSources_855447_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b444e26-211d-4c34-aff5-8cc3fee68fb2_isrForCrosstalkSources_855447_32.stderr 2024-03-09 14:56:00 proc_lsst.multi:170 [INFO] [multi] job_id 16964313 proc_lsst.multi INFO: [multi] job_id 16964313 2024-03-09 14:56:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 221 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 221 2024-03-09 14:56:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964313 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964313 parsl.executors.status_handling DEBUG: Launched block 276 on executor multi with job ID 16964313 parsl.executors.status_handling INFO: Allocated block ID 277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-277 2024-03-09 14:56:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 221 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 221 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98a593904d2dac1d43d712722d98fd31 with result from task 4495 parsl.dataflow.dflow DEBUG: Task 4495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/79776475-b895-4fe7-982b-bce81eacf342_isrForCrosstalkSources_855444_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/79776475-b895-4fe7-982b-bce81eacf342_isrForCrosstalkSources_855444_14.stderr parsl.dataflow.dflow INFO: Task 4475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00bef76857175750574f5982d44353db with result from task 4475 parsl.dataflow.dflow DEBUG: Task 4475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/9f0fd6cc-15c2-4865-be11-1ced50bcaedd_isrForCrosstalkSources_855415_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/9f0fd6cc-15c2-4865-be11-1ced50bcaedd_isrForCrosstalkSources_855415_32.stderr 2024-03-09 14:56:00 proc_lsst.multi:170 [INFO] [multi] job_id 16964314 proc_lsst.multi INFO: [multi] job_id 16964314 2024-03-09 14:56:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 222 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 222 2024-03-09 14:56:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964314 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964314 parsl.executors.status_handling DEBUG: Launched block 277 on executor multi with job ID 16964314 parsl.executors.status_handling INFO: Allocated block ID 278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-278 2024-03-09 14:56:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 222 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 222 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:00 proc_lsst.multi:170 [INFO] [multi] job_id 16964315 proc_lsst.multi INFO: [multi] job_id 16964315 2024-03-09 14:56:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 223 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 223 2024-03-09 14:56:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964315 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964315 parsl.executors.status_handling DEBUG: Launched block 278 on executor multi with job ID 16964315 parsl.executors.status_handling INFO: Allocated block ID 279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-279 2024-03-09 14:56:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 223 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 223 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:00 proc_lsst.multi:170 [INFO] [multi] job_id 16964316 proc_lsst.multi INFO: [multi] job_id 16964316 2024-03-09 14:56:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 224 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 224 2024-03-09 14:56:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964316 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964316 parsl.executors.status_handling DEBUG: Launched block 279 on executor multi with job ID 16964316 parsl.executors.status_handling INFO: Allocated block ID 280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-280 2024-03-09 14:56:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 224 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 224 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:00 proc_lsst.multi:170 [INFO] [multi] job_id 16964317 proc_lsst.multi INFO: [multi] job_id 16964317 2024-03-09 14:56:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 225 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 225 2024-03-09 14:56:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964317 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964317 parsl.executors.status_handling DEBUG: Launched block 280 on executor multi with job ID 16964317 parsl.executors.status_handling INFO: Allocated block ID 281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-281 2024-03-09 14:56:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 225 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 225 2024-03-09 14:56:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964318 proc_lsst.multi INFO: [multi] job_id 16964318 2024-03-09 14:56:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 226 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 226 2024-03-09 14:56:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964318 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964318 parsl.executors.status_handling DEBUG: Launched block 281 on executor multi with job ID 16964318 parsl.executors.status_handling INFO: Allocated block ID 282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-282 2024-03-09 14:56:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 226 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 226 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:01 proc_lsst.multi:170 [INFO] [multi] job_id 16964319 proc_lsst.multi INFO: [multi] job_id 16964319 2024-03-09 14:56:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 227 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 227 2024-03-09 14:56:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964319 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964319 parsl.executors.status_handling DEBUG: Launched block 282 on executor multi with job ID 16964319 parsl.executors.status_handling INFO: Allocated block ID 283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-283 2024-03-09 14:56:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 227 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 227 2024-03-09 14:56:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964320 proc_lsst.multi INFO: [multi] job_id 16964320 2024-03-09 14:56:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 228 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 228 2024-03-09 14:56:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964320 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964320 parsl.executors.status_handling DEBUG: Launched block 283 on executor multi with job ID 16964320 parsl.executors.status_handling INFO: Allocated block ID 284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-284 2024-03-09 14:56:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 228 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 228 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964321 proc_lsst.multi INFO: [multi] job_id 16964321 2024-03-09 14:56:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 229 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 229 2024-03-09 14:56:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964321 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964321 parsl.executors.status_handling DEBUG: Launched block 284 on executor multi with job ID 16964321 parsl.executors.status_handling INFO: Allocated block ID 285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-285 2024-03-09 14:56:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 229 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 229 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964322 proc_lsst.multi INFO: [multi] job_id 16964322 2024-03-09 14:56:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 230 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 230 2024-03-09 14:56:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964322 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964322 parsl.executors.status_handling DEBUG: Launched block 285 on executor multi with job ID 16964322 parsl.executors.status_handling INFO: Allocated block ID 286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-286 2024-03-09 14:56:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 230 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 230 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964323 proc_lsst.multi INFO: [multi] job_id 16964323 2024-03-09 14:56:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 231 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 231 2024-03-09 14:56:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964323 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964323 parsl.executors.status_handling DEBUG: Launched block 286 on executor multi with job ID 16964323 parsl.executors.status_handling INFO: Allocated block ID 287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-287 2024-03-09 14:56:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 231 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 231 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:02 proc_lsst.multi:170 [INFO] [multi] job_id 16964324 proc_lsst.multi INFO: [multi] job_id 16964324 2024-03-09 14:56:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 232 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 232 2024-03-09 14:56:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964324 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964324 parsl.executors.status_handling DEBUG: Launched block 287 on executor multi with job ID 16964324 parsl.executors.status_handling INFO: Allocated block ID 288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-288 2024-03-09 14:56:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 232 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 232 2024-03-09 14:56:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964325 proc_lsst.multi INFO: [multi] job_id 16964325 2024-03-09 14:56:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 233 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 233 2024-03-09 14:56:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964325 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964325 parsl.executors.status_handling DEBUG: Launched block 288 on executor multi with job ID 16964325 parsl.executors.status_handling INFO: Allocated block ID 289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-289 2024-03-09 14:56:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 233 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 233 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964326 proc_lsst.multi INFO: [multi] job_id 16964326 2024-03-09 14:56:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 234 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 234 2024-03-09 14:56:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964326 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964326 parsl.executors.status_handling DEBUG: Launched block 289 on executor multi with job ID 16964326 parsl.executors.status_handling INFO: Allocated block ID 290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-290 2024-03-09 14:56:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 234 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 234 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964327 proc_lsst.multi INFO: [multi] job_id 16964327 2024-03-09 14:56:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 235 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 235 2024-03-09 14:56:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964327 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964327 parsl.executors.status_handling DEBUG: Launched block 290 on executor multi with job ID 16964327 parsl.executors.status_handling INFO: Allocated block ID 291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-291 2024-03-09 14:56:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 235 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 235 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964328 proc_lsst.multi INFO: [multi] job_id 16964328 2024-03-09 14:56:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 236 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 236 2024-03-09 14:56:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964328 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964328 parsl.executors.status_handling DEBUG: Launched block 291 on executor multi with job ID 16964328 parsl.executors.status_handling INFO: Allocated block ID 292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-292 2024-03-09 14:56:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 236 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 236 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:03 proc_lsst.multi:170 [INFO] [multi] job_id 16964329 proc_lsst.multi INFO: [multi] job_id 16964329 2024-03-09 14:56:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 237 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 237 2024-03-09 14:56:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964329 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964329 parsl.executors.status_handling DEBUG: Launched block 292 on executor multi with job ID 16964329 parsl.executors.status_handling INFO: Allocated block ID 293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-293 2024-03-09 14:56:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 237 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 237 2024-03-09 14:56:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964330 proc_lsst.multi INFO: [multi] job_id 16964330 2024-03-09 14:56:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 238 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 238 2024-03-09 14:56:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964330 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964330 parsl.executors.status_handling DEBUG: Launched block 293 on executor multi with job ID 16964330 parsl.executors.status_handling INFO: Allocated block ID 294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-294 2024-03-09 14:56:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 238 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 238 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964331 proc_lsst.multi INFO: [multi] job_id 16964331 2024-03-09 14:56:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 239 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 239 2024-03-09 14:56:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964331 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964331 parsl.executors.status_handling DEBUG: Launched block 294 on executor multi with job ID 16964331 parsl.executors.status_handling INFO: Allocated block ID 295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-295 2024-03-09 14:56:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 239 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 239 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964332 proc_lsst.multi INFO: [multi] job_id 16964332 2024-03-09 14:56:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 240 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 240 2024-03-09 14:56:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964332 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964332 parsl.executors.status_handling DEBUG: Launched block 295 on executor multi with job ID 16964332 parsl.executors.status_handling INFO: Allocated block ID 296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-296 2024-03-09 14:56:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 240 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 240 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964333 proc_lsst.multi INFO: [multi] job_id 16964333 2024-03-09 14:56:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 241 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 241 2024-03-09 14:56:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964333 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964333 parsl.executors.status_handling DEBUG: Launched block 296 on executor multi with job ID 16964333 parsl.executors.status_handling INFO: Allocated block ID 297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-297 2024-03-09 14:56:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 241 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 241 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:04 proc_lsst.multi:170 [INFO] [multi] job_id 16964334 proc_lsst.multi INFO: [multi] job_id 16964334 2024-03-09 14:56:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 242 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 242 2024-03-09 14:56:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964334 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964334 parsl.executors.status_handling DEBUG: Launched block 297 on executor multi with job ID 16964334 parsl.executors.status_handling INFO: Allocated block ID 298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-298 2024-03-09 14:56:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 242 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 242 2024-03-09 14:56:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af2bfabfab43be27b2f8f0df15f6934f with result from task 1160 parsl.dataflow.dflow DEBUG: Task 1160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/41529b21-29a6-44e8-b8b1-4e5d96c875eb_isrForCrosstalkSources_855428_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/41529b21-29a6-44e8-b8b1-4e5d96c875eb_isrForCrosstalkSources_855428_34.stderr parsl.dataflow.dflow INFO: Task 1564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90931785f8558e8ce1482cf863337121 with result from task 1564 parsl.dataflow.dflow DEBUG: Task 1564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/12602ff5-7e25-4473-8321-889ccb2c04b8_isrForCrosstalkSources_855426_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/12602ff5-7e25-4473-8321-889ccb2c04b8_isrForCrosstalkSources_855426_13.stderr parsl.dataflow.dflow INFO: Task 1157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478e87550c39c9f783de89fc43240fa5 with result from task 1157 parsl.dataflow.dflow DEBUG: Task 1157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bfdefd54-6547-49a8-baec-41fd306b4fb0_isrForCrosstalkSources_855424_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bfdefd54-6547-49a8-baec-41fd306b4fb0_isrForCrosstalkSources_855424_34.stderr parsl.dataflow.dflow INFO: Task 1166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17d59c59f02ce183fb469da23a09c2bb with result from task 1166 parsl.dataflow.dflow DEBUG: Task 1166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ad663e9-9367-4f30-87f2-fe6f725223bf_isrForCrosstalkSources_855445_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/4ad663e9-9367-4f30-87f2-fe6f725223bf_isrForCrosstalkSources_855445_26.stderr parsl.dataflow.dflow INFO: Task 1155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d05f63667e36afd3cc1f0ddf0e200be with result from task 1155 parsl.dataflow.dflow DEBUG: Task 1155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/36d64beb-f580-421d-a1a1-2495aae50c22_isrForCrosstalkSources_855432_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/36d64beb-f580-421d-a1a1-2495aae50c22_isrForCrosstalkSources_855432_28.stderr parsl.dataflow.dflow INFO: Task 1150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84d8470bdb224b9abe958270c859aab3 with result from task 1150 parsl.dataflow.dflow DEBUG: Task 1150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0936f4c1-e53e-4d93-9166-286cb9ebb051_isrForCrosstalkSources_855408_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0936f4c1-e53e-4d93-9166-286cb9ebb051_isrForCrosstalkSources_855408_5.stderr parsl.dataflow.dflow INFO: Task 1577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba036bc3be26651023989afd049a4cdc with result from task 1577 parsl.dataflow.dflow DEBUG: Task 1577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8c4d424d-ad91-4c1a-8b59-c5fe2e8fc7a0_isrForCrosstalkSources_855395_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/8c4d424d-ad91-4c1a-8b59-c5fe2e8fc7a0_isrForCrosstalkSources_855395_41.stderr parsl.dataflow.dflow INFO: Task 1151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5af56ae5cef2d2f2375865bd0d66b7ef with result from task 1151 parsl.dataflow.dflow DEBUG: Task 1151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e29cffcc-a015-466e-948c-92d0c19d6953_isrForCrosstalkSources_855385_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/e29cffcc-a015-466e-948c-92d0c19d6953_isrForCrosstalkSources_855385_7.stderr parsl.dataflow.dflow INFO: Task 1152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d311e186ebb547f2cacaa3b790cd91f3 with result from task 1152 parsl.dataflow.dflow DEBUG: Task 1152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/41a88532-7ec6-464d-a274-aced7224ea91_isrForCrosstalkSources_855429_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/41a88532-7ec6-464d-a274-aced7224ea91_isrForCrosstalkSources_855429_38.stderr parsl.dataflow.dflow INFO: Task 1574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d0a015e960051a6fa936f19ac172772 with result from task 1574 parsl.dataflow.dflow DEBUG: Task 1574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c5945b7b-f560-4a90-9056-dfe7d39dd3d3_isrForCrosstalkSources_855389_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/c5945b7b-f560-4a90-9056-dfe7d39dd3d3_isrForCrosstalkSources_855389_44.stderr parsl.dataflow.dflow INFO: Task 1572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a11cb6638f9a958aa7d51597740b2031 with result from task 1572 parsl.dataflow.dflow DEBUG: Task 1572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/ebace9a5-72ad-4969-88f0-94d28d56dcf9_isrForCrosstalkSources_855456_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/ebace9a5-72ad-4969-88f0-94d28d56dcf9_isrForCrosstalkSources_855456_7.stderr parsl.dataflow.dflow INFO: Task 1162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d2856bd243274ecebb4128e17b21a0 with result from task 1162 parsl.dataflow.dflow DEBUG: Task 1162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c8169b75-cbb6-4e37-b0d8-6b3c90acab5e_isrForCrosstalkSources_855392_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/c8169b75-cbb6-4e37-b0d8-6b3c90acab5e_isrForCrosstalkSources_855392_46.stderr parsl.dataflow.dflow INFO: Task 1163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ea4814dd084cf22573b5f9c777520a with result from task 1163 parsl.dataflow.dflow DEBUG: Task 1163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1fb59a67-8969-4266-952a-dd44381c8a12_isrForCrosstalkSources_855422_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1fb59a67-8969-4266-952a-dd44381c8a12_isrForCrosstalkSources_855422_18.stderr parsl.dataflow.dflow INFO: Task 1562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6bb82f7d5ca4a06bf087468b55b8360 with result from task 1562 2024-03-09 14:56:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964335 proc_lsst.multi INFO: [multi] job_id 16964335 2024-03-09 14:56:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 243 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 243 2024-03-09 14:56:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964335 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964335 parsl.executors.status_handling DEBUG: Launched block 298 on executor multi with job ID 16964335 parsl.executors.status_handling INFO: Allocated block ID 299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-299 2024-03-09 14:56:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 243 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 243 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/00527f69-5fef-4857-81b4-04aa03ea6579_isrForCrosstalkSources_855456_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/00527f69-5fef-4857-81b4-04aa03ea6579_isrForCrosstalkSources_855456_29.stderr parsl.dataflow.dflow INFO: Task 1569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3156529980239f144d5acd3b2210509 with result from task 1569 parsl.dataflow.dflow DEBUG: Task 1569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/819fb5a4-9421-4f17-8c69-e54f9341af8d_isrForCrosstalkSources_855400_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/819fb5a4-9421-4f17-8c69-e54f9341af8d_isrForCrosstalkSources_855400_51.stderr parsl.dataflow.dflow INFO: Task 1172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065c9cba9b419c700ba04a9cdcd4908d with result from task 1172 parsl.dataflow.dflow DEBUG: Task 1172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e7377bf9-01e3-435d-875e-7496a1f70252_isrForCrosstalkSources_855419_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/e7377bf9-01e3-435d-875e-7496a1f70252_isrForCrosstalkSources_855419_49.stderr parsl.dataflow.dflow INFO: Task 1169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbe67a72913ccedc2c356b4fbca8a81b with result from task 1169 parsl.dataflow.dflow DEBUG: Task 1169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fd2f93c7-3e99-4ea1-9d8b-485323632824_isrForCrosstalkSources_855401_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/fd2f93c7-3e99-4ea1-9d8b-485323632824_isrForCrosstalkSources_855401_28.stderr parsl.dataflow.dflow INFO: Task 1164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 442036f90bfb0750069dbbb1a1052d2d with result from task 1164 parsl.dataflow.dflow DEBUG: Task 1164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/db6a928e-ba82-4e45-930d-9bff60392aee_isrForCrosstalkSources_855432_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/db6a928e-ba82-4e45-930d-9bff60392aee_isrForCrosstalkSources_855432_26.stderr parsl.dataflow.dflow INFO: Task 1548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cae6a422e78a43f1108211cc5c041cce with result from task 1548 parsl.dataflow.dflow DEBUG: Task 1548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32b6a40f-2108-4c4e-820a-ac3979e2457c_isrForCrosstalkSources_855448_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/32b6a40f-2108-4c4e-820a-ac3979e2457c_isrForCrosstalkSources_855448_40.stderr parsl.dataflow.dflow INFO: Task 1551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b22087f2f3e22aad7f1e905fc384c808 with result from task 1551 parsl.dataflow.dflow DEBUG: Task 1551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7add53af-d084-4cba-a82b-eaf2fa53c21f_isrForCrosstalkSources_855397_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/7add53af-d084-4cba-a82b-eaf2fa53c21f_isrForCrosstalkSources_855397_57.stderr parsl.dataflow.dflow INFO: Task 1546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0bfb19761171e6d3ee080abea64f4fa with result from task 1546 parsl.dataflow.dflow DEBUG: Task 1546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b63a10bf-3dc2-4c4f-a1d7-8a61f6fcc546_isrForCrosstalkSources_855410_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/b63a10bf-3dc2-4c4f-a1d7-8a61f6fcc546_isrForCrosstalkSources_855410_37.stderr parsl.dataflow.dflow INFO: Task 1557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9f2e455f540bb1ac384d1f2640aa27e with result from task 1557 parsl.dataflow.dflow DEBUG: Task 1557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/27adbb8c-108e-4a46-85d6-d258d01a55ef_isrForCrosstalkSources_855458_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/27adbb8c-108e-4a46-85d6-d258d01a55ef_isrForCrosstalkSources_855458_35.stderr parsl.dataflow.dflow INFO: Task 1554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a014f5ba2d24ed194c5847837be6f67a with result from task 1554 parsl.dataflow.dflow DEBUG: Task 1554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e58ab48-262d-4186-b7fb-5cca35ba1be0_isrForCrosstalkSources_855423_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3e58ab48-262d-4186-b7fb-5cca35ba1be0_isrForCrosstalkSources_855423_21.stderr parsl.dataflow.dflow INFO: Task 1563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8fcd8fd788df7e3b0a836179ee15697 with result from task 1563 parsl.dataflow.dflow DEBUG: Task 1563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5146463b-5c32-4b98-ab7e-de511743cb7e_isrForCrosstalkSources_855403_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/5146463b-5c32-4b98-ab7e-de511743cb7e_isrForCrosstalkSources_855403_38.stderr parsl.dataflow.dflow INFO: Task 1159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06471e2e904e7b6027037de9028cdfa9 with result from task 1159 parsl.dataflow.dflow DEBUG: Task 1159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1a5afa20-2f38-4804-95b6-574ec21ddd0d_isrForCrosstalkSources_855391_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/1a5afa20-2f38-4804-95b6-574ec21ddd0d_isrForCrosstalkSources_855391_25.stderr parsl.dataflow.dflow INFO: Task 1556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9dad5e5df268b123cc73365b0720a1f with result from task 1556 parsl.dataflow.dflow DEBUG: Task 1556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8c9a67bf-2ad3-4405-b287-6a1cb19b87da_isrForCrosstalkSources_855446_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/8c9a67bf-2ad3-4405-b287-6a1cb19b87da_isrForCrosstalkSources_855446_50.stderr parsl.dataflow.dflow INFO: Task 1571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6083aaaf41a9180f2f7c922d52e5532d with result from task 1571 parsl.dataflow.dflow DEBUG: Task 1571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/581eff3a-2f47-4131-875f-c7ff454d1ec8_isrForCrosstalkSources_855422_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/581eff3a-2f47-4131-875f-c7ff454d1ec8_isrForCrosstalkSources_855422_35.stderr parsl.dataflow.dflow INFO: Task 1573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dd907a6fc9a993e8ad91093dbf93cb6 with result from task 1573 parsl.dataflow.dflow DEBUG: Task 1573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f7f1fa85-1c2f-4729-93c0-6e743eeeeefb_isrForCrosstalkSources_855416_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/f7f1fa85-1c2f-4729-93c0-6e743eeeeefb_isrForCrosstalkSources_855416_3.stderr parsl.dataflow.dflow INFO: Task 1570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12edb39be866f07c31ae9f8d533b65cd with result from task 1570 parsl.dataflow.dflow DEBUG: Task 1570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64c49390-54da-4c7f-af02-a62a37584451_isrForCrosstalkSources_855424_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/64c49390-54da-4c7f-af02-a62a37584451_isrForCrosstalkSources_855424_22.stderr parsl.dataflow.dflow INFO: Task 1576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f397f0cf9e5254e3e45bd39968c91b with result from task 1576 parsl.dataflow.dflow DEBUG: Task 1576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2c2cb593-6e83-4da7-9af8-f9afb2683b4b_isrForCrosstalkSources_855457_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2c2cb593-6e83-4da7-9af8-f9afb2683b4b_isrForCrosstalkSources_855457_46.stderr parsl.dataflow.dflow INFO: Task 1555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca88d58b01f952fcc0fc7c217799372e with result from task 1555 parsl.dataflow.dflow DEBUG: Task 1555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac135b95-7c72-44cc-ad32-498e9c7e819a_isrForCrosstalkSources_855401_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ac135b95-7c72-44cc-ad32-498e9c7e819a_isrForCrosstalkSources_855401_31.stderr parsl.dataflow.dflow INFO: Task 1560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd0a62c21b33bbade146683da4456bfc with result from task 1560 parsl.dataflow.dflow DEBUG: Task 1560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c032f2a7-0bfd-4cb0-80bd-e12cf21d427f_isrForCrosstalkSources_855444_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/c032f2a7-0bfd-4cb0-80bd-e12cf21d427f_isrForCrosstalkSources_855444_42.stderr parsl.dataflow.dflow INFO: Task 1559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0154a2b65b2ddddef3553731e7a47b0f with result from task 1559 parsl.dataflow.dflow DEBUG: Task 1559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a26034b4-efab-4bf5-b3da-a3b43fe3c85c_isrForCrosstalkSources_855452_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a26034b4-efab-4bf5-b3da-a3b43fe3c85c_isrForCrosstalkSources_855452_14.stderr parsl.dataflow.dflow INFO: Task 1148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b863bcd95c897530b7f98ec54b84ea0 with result from task 1148 parsl.dataflow.dflow DEBUG: Task 1148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/cb7cec95-694f-47f9-9ac4-75655d98dce7_isrForCrosstalkSources_855413_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/cb7cec95-694f-47f9-9ac4-75655d98dce7_isrForCrosstalkSources_855413_33.stderr parsl.dataflow.dflow INFO: Task 1153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49ce91d94bf371472ed91e5fd547950f with result from task 1153 parsl.dataflow.dflow DEBUG: Task 1153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/02ec537e-9b30-41c9-8525-a621c87d8ec9_isrForCrosstalkSources_855386_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/02ec537e-9b30-41c9-8525-a621c87d8ec9_isrForCrosstalkSources_855386_20.stderr parsl.dataflow.dflow INFO: Task 1566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97167f787edb915366eff35225a9a997 with result from task 1566 parsl.dataflow.dflow DEBUG: Task 1566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b7e404-85ae-4ef5-9e9b-1650f1cf3096_isrForCrosstalkSources_855416_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b7e404-85ae-4ef5-9e9b-1650f1cf3096_isrForCrosstalkSources_855416_15.stderr parsl.dataflow.dflow INFO: Task 1158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53b779d0e189bc1b1802095977b32906 with result from task 1158 parsl.dataflow.dflow DEBUG: Task 1158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/11c86f94-1d59-43c2-a085-a73ed4aa7f15_isrForCrosstalkSources_855453_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/11c86f94-1d59-43c2-a085-a73ed4aa7f15_isrForCrosstalkSources_855453_18.stderr parsl.dataflow.dflow INFO: Task 1552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcb4efa6daf7a7c986b55e7e7db21bd5 with result from task 1552 parsl.dataflow.dflow DEBUG: Task 1552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/7d27de45-3c97-4ac4-a9d1-68cd6b30d1da_isrForCrosstalkSources_855409_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/7d27de45-3c97-4ac4-a9d1-68cd6b30d1da_isrForCrosstalkSources_855409_50.stderr parsl.dataflow.dflow INFO: Task 1165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 141409f1be9b2d2ea4ab427b52c10af4 with result from task 1165 parsl.dataflow.dflow DEBUG: Task 1165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b04168c0-c285-41c3-b294-610644a98f09_isrForCrosstalkSources_855451_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b04168c0-c285-41c3-b294-610644a98f09_isrForCrosstalkSources_855451_48.stderr parsl.dataflow.dflow INFO: Task 1629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59d38d3b9d1852c6d172427d45d99eb5 with result from task 1629 parsl.dataflow.dflow DEBUG: Task 1629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ff3e6171-1a96-4362-b360-0f52cb130e48_isrForCrosstalkSources_855394_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ff3e6171-1a96-4362-b360-0f52cb130e48_isrForCrosstalkSources_855394_22.stderr parsl.dataflow.dflow INFO: Task 1604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7728dd5ef25f35df829373efca0a38ce with result from task 1604 parsl.dataflow.dflow DEBUG: Task 1604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00c4acbe-f676-4409-809b-b398d7c108c7_isrForCrosstalkSources_855390_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/00c4acbe-f676-4409-809b-b398d7c108c7_isrForCrosstalkSources_855390_18.stderr parsl.dataflow.dflow INFO: Task 1611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cb1ab5a506e8d7fd244d634b19d22e8 with result from task 1611 parsl.dataflow.dflow DEBUG: Task 1611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/52ae6f11-cbf7-4ac3-afea-6f473e8f530a_isrForCrosstalkSources_855392_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/52ae6f11-cbf7-4ac3-afea-6f473e8f530a_isrForCrosstalkSources_855392_61.stderr parsl.dataflow.dflow INFO: Task 1170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba564039454c3a280009cda4ac41920b with result from task 1170 parsl.dataflow.dflow DEBUG: Task 1170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c9aa0789-a218-4cd1-900c-563e4bf436b4_isrForCrosstalkSources_855388_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/c9aa0789-a218-4cd1-900c-563e4bf436b4_isrForCrosstalkSources_855388_32.stderr parsl.dataflow.dflow INFO: Task 1553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89233a0dd995a835ff2599042fba0774 with result from task 1553 parsl.dataflow.dflow DEBUG: Task 1553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a974c91c-e2de-430f-9e74-c6ac9d138f7d_isrForCrosstalkSources_855404_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/a974c91c-e2de-430f-9e74-c6ac9d138f7d_isrForCrosstalkSources_855404_36.stderr parsl.dataflow.dflow INFO: Task 1547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f315c495064cd9fd176822fadc22741 with result from task 1547 parsl.dataflow.dflow DEBUG: Task 1547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1346c674-573a-4db2-a222-dc51e3a8ba61_isrForCrosstalkSources_855397_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/1346c674-573a-4db2-a222-dc51e3a8ba61_isrForCrosstalkSources_855397_32.stderr parsl.dataflow.dflow INFO: Task 1550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e415a9f9458942521510570d0476faf9 with result from task 1550 parsl.dataflow.dflow DEBUG: Task 1550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/11a6c670-b2c1-44e4-97f4-e6d458dd80d9_isrForCrosstalkSources_855429_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/11a6c670-b2c1-44e4-97f4-e6d458dd80d9_isrForCrosstalkSources_855429_60.stderr parsl.dataflow.dflow INFO: Task 1567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d2cabfcf53e5d51a9c2e63a42fed09f with result from task 1567 parsl.dataflow.dflow DEBUG: Task 1567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/228d49ff-d8d0-4a5b-a371-d785090bd582_isrForCrosstalkSources_855390_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/228d49ff-d8d0-4a5b-a371-d785090bd582_isrForCrosstalkSources_855390_34.stderr parsl.dataflow.dflow INFO: Task 1636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba35be1cd9611458a45a4f641381181f with result from task 1636 parsl.dataflow.dflow DEBUG: Task 1636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/94eb1664-85a8-4c71-8296-6c8c8810462b_isrForCrosstalkSources_855447_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/94eb1664-85a8-4c71-8296-6c8c8810462b_isrForCrosstalkSources_855447_10.stderr parsl.dataflow.dflow INFO: Task 1634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e86f54f8ec532764d80568e7cada1ac with result from task 1634 parsl.dataflow.dflow DEBUG: Task 1634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0653f3bd-4348-480a-9f9c-af0aae846971_isrForCrosstalkSources_855409_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/0653f3bd-4348-480a-9f9c-af0aae846971_isrForCrosstalkSources_855409_22.stderr parsl.dataflow.dflow INFO: Task 1632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d1fd025e740ecd0e92bb127c6dd49f with result from task 1632 parsl.dataflow.dflow DEBUG: Task 1632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dfcf2489-8a1e-413a-a15e-333a4c179fa9_isrForCrosstalkSources_855441_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/dfcf2489-8a1e-413a-a15e-333a4c179fa9_isrForCrosstalkSources_855441_4.stderr parsl.dataflow.dflow INFO: Task 1635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6871912212fb4684d4bc1905acb1aab1 with result from task 1635 parsl.dataflow.dflow DEBUG: Task 1635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/fb402c24-68c3-4efd-8f69-68a75c58b1ad_isrForCrosstalkSources_855408_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/fb402c24-68c3-4efd-8f69-68a75c58b1ad_isrForCrosstalkSources_855408_35.stderr parsl.dataflow.dflow INFO: Task 1633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d785cde519b995c0dc8ada92d4930eff with result from task 1633 parsl.dataflow.dflow DEBUG: Task 1633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aca0594-ec99-499d-86bb-d837e36cdcb2_isrForCrosstalkSources_855407_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8aca0594-ec99-499d-86bb-d837e36cdcb2_isrForCrosstalkSources_855407_39.stderr parsl.dataflow.dflow INFO: Task 1605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cff6bf1b03bfbd051221ade339de6f1 with result from task 1605 parsl.dataflow.dflow DEBUG: Task 1605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bf18d034-020c-44c5-9b60-a6f738d602b9_isrForCrosstalkSources_855412_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/bf18d034-020c-44c5-9b60-a6f738d602b9_isrForCrosstalkSources_855412_6.stderr parsl.dataflow.dflow INFO: Task 1608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a7623ac08a53dba0c064ec8fe2076c with result from task 1608 parsl.dataflow.dflow DEBUG: Task 1608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6d491b2-0c09-4f29-8d81-bb0f81ee2f91_isrForCrosstalkSources_855380_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/f6d491b2-0c09-4f29-8d81-bb0f81ee2f91_isrForCrosstalkSources_855380_3.stderr parsl.dataflow.dflow INFO: Task 1609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e71e94bbf39d9905a9a5305d9d07b2e1 with result from task 1609 parsl.dataflow.dflow DEBUG: Task 1609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/51964eed-b3f6-4fc3-9f5c-7b2563ceaaeb_isrForCrosstalkSources_855459_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/51964eed-b3f6-4fc3-9f5c-7b2563ceaaeb_isrForCrosstalkSources_855459_26.stderr parsl.dataflow.dflow INFO: Task 1610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24686cf8b25c1e92f073e81539b79303 with result from task 1610 parsl.dataflow.dflow DEBUG: Task 1610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/22e6e697-ad3d-4767-b88d-eab0b3a8ede7_isrForCrosstalkSources_855423_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/22e6e697-ad3d-4767-b88d-eab0b3a8ede7_isrForCrosstalkSources_855423_45.stderr parsl.dataflow.dflow INFO: Task 1607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 125276193578c31e609d0cfa18b5ddaa with result from task 1607 parsl.dataflow.dflow DEBUG: Task 1607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/d6741511-7c34-4619-876a-57d572f9f818_isrForCrosstalkSources_855393_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/d6741511-7c34-4619-876a-57d572f9f818_isrForCrosstalkSources_855393_7.stderr parsl.dataflow.dflow INFO: Task 1606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc917e8664c0ef200227efe6d89e42b with result from task 1606 parsl.dataflow.dflow DEBUG: Task 1606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/ef3fea41-a052-438b-b957-43ffe6d30c7f_isrForCrosstalkSources_855387_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/ef3fea41-a052-438b-b957-43ffe6d30c7f_isrForCrosstalkSources_855387_43.stderr parsl.dataflow.dflow INFO: Task 1630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c71e3fb0e7c51da84152db2db1aa7c96 with result from task 1630 parsl.dataflow.dflow DEBUG: Task 1630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ddd60077-ad04-449d-90ac-bd3128698487_isrForCrosstalkSources_855383_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/ddd60077-ad04-449d-90ac-bd3128698487_isrForCrosstalkSources_855383_46.stderr parsl.dataflow.dflow INFO: Task 1631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a4b5aabe8002cdf5176dfcdeb6ef11e with result from task 1631 parsl.dataflow.dflow DEBUG: Task 1631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/80a769f8-b156-4a4f-b180-0addc208d711_isrForCrosstalkSources_855430_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/80a769f8-b156-4a4f-b180-0addc208d711_isrForCrosstalkSources_855430_53.stderr parsl.dataflow.dflow INFO: Task 1568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76501f48591cb1de82d4ee150f6c8f38 with result from task 1568 parsl.dataflow.dflow DEBUG: Task 1568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/22bd8083-e906-45bc-aa23-c4396809fa7a_isrForCrosstalkSources_855406_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/22bd8083-e906-45bc-aa23-c4396809fa7a_isrForCrosstalkSources_855406_61.stderr parsl.dataflow.dflow INFO: Task 1149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a25eaa038a64e9a0148d1afa01fa136 with result from task 1149 parsl.dataflow.dflow DEBUG: Task 1149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f738f891-4c7b-4b99-acbe-10c13e3f3fe1_isrForCrosstalkSources_855395_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/f738f891-4c7b-4b99-acbe-10c13e3f3fe1_isrForCrosstalkSources_855395_31.stderr parsl.dataflow.dflow INFO: Task 1561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46e2c998dec45562caceca1c76e6681 with result from task 1561 parsl.dataflow.dflow DEBUG: Task 1561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/dc9c1d2c-0c8c-420a-8474-0d005a8317a8_isrForCrosstalkSources_855454_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/dc9c1d2c-0c8c-420a-8474-0d005a8317a8_isrForCrosstalkSources_855454_56.stderr parsl.dataflow.dflow INFO: Task 1647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1a7bf91839cb9dfe6d24e718955be65 with result from task 1647 parsl.dataflow.dflow DEBUG: Task 1647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/daaf818e-e95b-4817-9781-861726646926_isrForCrosstalkSources_855383_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/daaf818e-e95b-4817-9781-861726646926_isrForCrosstalkSources_855383_55.stderr parsl.dataflow.dflow INFO: Task 1644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24b4f1a464a4ac241212b3486b9b6994 with result from task 1644 parsl.dataflow.dflow DEBUG: Task 1644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/54eb0520-4874-43b9-b817-04b9c0100c21_isrForCrosstalkSources_855398_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/54eb0520-4874-43b9-b817-04b9c0100c21_isrForCrosstalkSources_855398_23.stderr parsl.dataflow.dflow INFO: Task 1639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790c28780344d871d94782e0fdb26cb4 with result from task 1639 parsl.dataflow.dflow DEBUG: Task 1639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3344b202-6125-47fb-a384-2ded742fe9ff_isrForCrosstalkSources_855425_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3344b202-6125-47fb-a384-2ded742fe9ff_isrForCrosstalkSources_855425_21.stderr parsl.dataflow.dflow INFO: Task 1549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 638362570183beb962716abf0b279803 with result from task 1549 parsl.dataflow.dflow DEBUG: Task 1549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cfc4a8c8-7e36-4880-b364-65222d6ae3c8_isrForCrosstalkSources_855387_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cfc4a8c8-7e36-4880-b364-65222d6ae3c8_isrForCrosstalkSources_855387_49.stderr parsl.dataflow.dflow INFO: Task 1168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b23aa6c9107655d09415d430afb4fafd with result from task 1168 parsl.dataflow.dflow DEBUG: Task 1168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/2e2a5419-434e-4447-976a-7f33379f7b1d_isrForCrosstalkSources_855439_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/2e2a5419-434e-4447-976a-7f33379f7b1d_isrForCrosstalkSources_855439_61.stderr parsl.dataflow.dflow INFO: Task 1575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62f9a580e0b49d739630be3b633a2cb9 with result from task 1575 parsl.dataflow.dflow DEBUG: Task 1575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e11eadc8-aeee-44b7-a838-4f3ec5ef0296_isrForCrosstalkSources_855391_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/e11eadc8-aeee-44b7-a838-4f3ec5ef0296_isrForCrosstalkSources_855391_24.stderr parsl.dataflow.dflow INFO: Task 1649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84e3c521e28be6b37e3de35f4cb56178 with result from task 1649 parsl.dataflow.dflow DEBUG: Task 1649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/0b74d105-a385-4138-87c1-3bbf8b5f205f_isrForCrosstalkSources_855416_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/0b74d105-a385-4138-87c1-3bbf8b5f205f_isrForCrosstalkSources_855416_12.stderr parsl.dataflow.dflow INFO: Task 1650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a75a15f94146d462379e7d9f6e84de92 with result from task 1650 parsl.dataflow.dflow DEBUG: Task 1650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a0d27a91-b4c8-436c-bc21-d1b1e8120e9a_isrForCrosstalkSources_855405_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/a0d27a91-b4c8-436c-bc21-d1b1e8120e9a_isrForCrosstalkSources_855405_26.stderr parsl.dataflow.dflow INFO: Task 1558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bceb115929953207d160004731779e89 with result from task 1558 parsl.dataflow.dflow DEBUG: Task 1558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d77c9c1f-5ba3-4709-82a6-9f739e7a6fb0_isrForCrosstalkSources_855438_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d77c9c1f-5ba3-4709-82a6-9f739e7a6fb0_isrForCrosstalkSources_855438_59.stderr parsl.dataflow.dflow INFO: Task 1171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d39a1294de0a2995ad7242f8d092256f with result from task 1171 parsl.dataflow.dflow DEBUG: Task 1171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4257af37-396b-4205-9858-57777ecb8d37_isrForCrosstalkSources_855409_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/4257af37-396b-4205-9858-57777ecb8d37_isrForCrosstalkSources_855409_33.stderr parsl.dataflow.dflow INFO: Task 1640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 336532fe9a5a6d970e13011a2fce73a1 with result from task 1640 parsl.dataflow.dflow DEBUG: Task 1640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/08eec2df-d9be-412a-afee-62d21ae5209d_isrForCrosstalkSources_855390_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/08eec2df-d9be-412a-afee-62d21ae5209d_isrForCrosstalkSources_855390_37.stderr parsl.dataflow.dflow INFO: Task 1615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 045206164c7f1614396296181af95800 with result from task 1615 parsl.dataflow.dflow DEBUG: Task 1615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cce2dcd1-cd22-40f4-83e1-36b1fb2ca186_isrForCrosstalkSources_855415_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/cce2dcd1-cd22-40f4-83e1-36b1fb2ca186_isrForCrosstalkSources_855415_47.stderr parsl.dataflow.dflow INFO: Task 1617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 682f40a48dfb2801f5fec575b068af40 with result from task 1617 parsl.dataflow.dflow DEBUG: Task 1617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/74d4a24e-3be4-4afc-a19b-abc079f1d168_isrForCrosstalkSources_855392_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/74d4a24e-3be4-4afc-a19b-abc079f1d168_isrForCrosstalkSources_855392_9.stderr parsl.dataflow.dflow INFO: Task 1619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d28b9f6a00d63292cee0eb8c1d8155c6 with result from task 1619 parsl.dataflow.dflow DEBUG: Task 1619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/8a8e9522-0e74-48fe-ab99-eff873ce6152_isrForCrosstalkSources_855380_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/8a8e9522-0e74-48fe-ab99-eff873ce6152_isrForCrosstalkSources_855380_8.stderr parsl.dataflow.dflow INFO: Task 1638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47c3adb4278557cfdc5af83e3f48e45d with result from task 1638 parsl.dataflow.dflow DEBUG: Task 1638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a6abdd33-c75f-48ca-8250-f49e9bfde36b_isrForCrosstalkSources_855394_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a6abdd33-c75f-48ca-8250-f49e9bfde36b_isrForCrosstalkSources_855394_32.stderr parsl.dataflow.dflow INFO: Task 1626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68f04d36abd990b7fb534702c51d429 with result from task 1626 parsl.dataflow.dflow DEBUG: Task 1626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3f55fb9b-4a1a-4748-869c-4151b6366f2c_isrForCrosstalkSources_855405_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/3f55fb9b-4a1a-4748-869c-4151b6366f2c_isrForCrosstalkSources_855405_7.stderr parsl.dataflow.dflow INFO: Task 1637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8a69494bc69360d6c404c668792fe95 with result from task 1637 parsl.dataflow.dflow DEBUG: Task 1637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/a406a6e0-1b79-47fa-b4f6-62da6c5269e9_isrForCrosstalkSources_855458_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/a406a6e0-1b79-47fa-b4f6-62da6c5269e9_isrForCrosstalkSources_855458_51.stderr parsl.dataflow.dflow INFO: Task 1613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 746448d1a60ca7f5d0a2a6dd82d58f1a with result from task 1613 parsl.dataflow.dflow DEBUG: Task 1613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/3d424ffd-669c-4ac3-b4a5-d1c01f63d32a_isrForCrosstalkSources_855421_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/3d424ffd-669c-4ac3-b4a5-d1c01f63d32a_isrForCrosstalkSources_855421_25.stderr parsl.dataflow.dflow INFO: Task 1645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d0ea6ad4adc395a158bab568a1f302d with result from task 1645 parsl.dataflow.dflow DEBUG: Task 1645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2ca336ff-b99b-4b1e-b838-477abc0dd105_isrForCrosstalkSources_855395_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/2ca336ff-b99b-4b1e-b838-477abc0dd105_isrForCrosstalkSources_855395_34.stderr parsl.dataflow.dflow INFO: Task 1621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a611c1444c2e931a555809ec28a73d99 with result from task 1621 parsl.dataflow.dflow DEBUG: Task 1621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/14e9cb5d-7819-4cb9-b64c-5e92b38ec39a_isrForCrosstalkSources_855400_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/14e9cb5d-7819-4cb9-b64c-5e92b38ec39a_isrForCrosstalkSources_855400_59.stderr parsl.dataflow.dflow INFO: Task 1625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b72693a00d237cad5d4bdd266acbff9 with result from task 1625 parsl.dataflow.dflow DEBUG: Task 1625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9a81517f-4db0-4b04-9402-43a3f13cd132_isrForCrosstalkSources_855448_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/9a81517f-4db0-4b04-9402-43a3f13cd132_isrForCrosstalkSources_855448_47.stderr parsl.dataflow.dflow INFO: Task 1643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb85e22c3eb01ac253f6c6fb093b9b19 with result from task 1643 parsl.dataflow.dflow DEBUG: Task 1643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/36542a30-5cd0-4f95-b8c6-9ff5f165ad42_isrForCrosstalkSources_855448_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/36542a30-5cd0-4f95-b8c6-9ff5f165ad42_isrForCrosstalkSources_855448_23.stderr parsl.dataflow.dflow INFO: Task 1628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cd37757ecb23338868f7068673ddb54 with result from task 1628 parsl.dataflow.dflow DEBUG: Task 1628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5e330391-98f5-4b5d-b611-b8a746104e88_isrForCrosstalkSources_855444_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5e330391-98f5-4b5d-b611-b8a746104e88_isrForCrosstalkSources_855444_13.stderr parsl.dataflow.dflow INFO: Task 1622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 699e1c207beb117adf330fda58ea8a8d with result from task 1622 parsl.dataflow.dflow DEBUG: Task 1622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8e9866a2-0623-47ac-9baa-6a8f3bacebe7_isrForCrosstalkSources_855428_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/8e9866a2-0623-47ac-9baa-6a8f3bacebe7_isrForCrosstalkSources_855428_37.stderr parsl.dataflow.dflow INFO: Task 1652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 010ed17802c01a42dbb26e5ae3f14b9e with result from task 1652 parsl.dataflow.dflow DEBUG: Task 1652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/a462c518-7ef4-4df0-8a66-cb19175cbaa3_isrForCrosstalkSources_855413_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/a462c518-7ef4-4df0-8a66-cb19175cbaa3_isrForCrosstalkSources_855413_49.stderr parsl.dataflow.dflow INFO: Task 1648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 868ada754ff1d77f9979550454e28836 with result from task 1648 parsl.dataflow.dflow DEBUG: Task 1648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/69e7fffa-7222-442f-9318-a11d09380887_isrForCrosstalkSources_855434_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/69e7fffa-7222-442f-9318-a11d09380887_isrForCrosstalkSources_855434_9.stderr parsl.dataflow.dflow INFO: Task 1616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2e85df481ada8712a37447f99e5a080 with result from task 1616 parsl.dataflow.dflow DEBUG: Task 1616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/980111e6-f4d2-4283-aa20-5ad4687b951c_isrForCrosstalkSources_855427_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/980111e6-f4d2-4283-aa20-5ad4687b951c_isrForCrosstalkSources_855427_24.stderr parsl.dataflow.dflow INFO: Task 1627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41b5956780873dc98fcc7aab3d68e6d9 with result from task 1627 parsl.dataflow.dflow DEBUG: Task 1627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8f5c2d7a-436e-4f50-b874-a289c1b4aee1_isrForCrosstalkSources_855402_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8f5c2d7a-436e-4f50-b874-a289c1b4aee1_isrForCrosstalkSources_855402_21.stderr parsl.dataflow.dflow INFO: Task 1161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b951cb9e640aecf6caa91693d05d72 with result from task 1161 parsl.dataflow.dflow DEBUG: Task 1161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91074912-ef4d-451b-b1bf-b329d637f442_isrForCrosstalkSources_855390_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/91074912-ef4d-451b-b1bf-b329d637f442_isrForCrosstalkSources_855390_28.stderr parsl.dataflow.dflow INFO: Task 1614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6076c6d845335e86f4e9f8fe8f715eca with result from task 1614 parsl.dataflow.dflow DEBUG: Task 1614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1706d8d8-6601-4009-af19-7900ec9aa905_isrForCrosstalkSources_855424_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/1706d8d8-6601-4009-af19-7900ec9aa905_isrForCrosstalkSources_855424_38.stderr parsl.dataflow.dflow INFO: Task 1623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1017c720456a9bbaff0639b16a9315d with result from task 1623 parsl.dataflow.dflow DEBUG: Task 1623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/08e7fcaa-fc38-4abb-a540-012768f2d398_isrForCrosstalkSources_855434_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/08e7fcaa-fc38-4abb-a540-012768f2d398_isrForCrosstalkSources_855434_10.stderr 2024-03-09 14:56:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964336 proc_lsst.multi INFO: [multi] job_id 16964336 2024-03-09 14:56:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 244 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 244 2024-03-09 14:56:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964336 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964336 parsl.executors.status_handling DEBUG: Launched block 299 on executor multi with job ID 16964336 parsl.executors.status_handling INFO: Allocated block ID 300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-300 2024-03-09 14:56:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 244 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 244 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 1620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1715cafae690400178f13021a900c2 with result from task 1620 parsl.dataflow.dflow DEBUG: Task 1620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3900f274-fe0d-4d26-ac62-cdaabc48c02a_isrForCrosstalkSources_855416_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/3900f274-fe0d-4d26-ac62-cdaabc48c02a_isrForCrosstalkSources_855416_17.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e18faa93135feef52957a5bc02dcba with result from task 1641 parsl.dataflow.dflow DEBUG: Task 1641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/55930dcc-fb5c-433f-abda-472e2a530fe3_isrForCrosstalkSources_855427_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/55930dcc-fb5c-433f-abda-472e2a530fe3_isrForCrosstalkSources_855427_52.stderr parsl.dataflow.dflow INFO: Task 1167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e23ca56c9423dac4a6fed6ad318a17 with result from task 1167 parsl.dataflow.dflow DEBUG: Task 1167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c11e2b42-ec9e-46f7-be3c-f94bba256568_isrForCrosstalkSources_855429_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c11e2b42-ec9e-46f7-be3c-f94bba256568_isrForCrosstalkSources_855429_26.stderr parsl.dataflow.dflow INFO: Task 1651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8444ab750d992e43f0f8f611c858d58f with result from task 1651 parsl.dataflow.dflow DEBUG: Task 1651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/1c8ccb28-ea3d-4e51-8507-725c3521b795_isrForCrosstalkSources_855430_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/1c8ccb28-ea3d-4e51-8507-725c3521b795_isrForCrosstalkSources_855430_23.stderr parsl.dataflow.dflow INFO: Task 1624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 133e6b3279cc5b6b43029432fb8cade6 with result from task 1624 parsl.dataflow.dflow DEBUG: Task 1624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ca223f9d-0a26-4031-8081-7a1b8ce227b9_isrForCrosstalkSources_855433_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/ca223f9d-0a26-4031-8081-7a1b8ce227b9_isrForCrosstalkSources_855433_5.stderr parsl.dataflow.dflow INFO: Task 1154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59cf88589955df486bd27b454f5b94b5 with result from task 1154 parsl.dataflow.dflow DEBUG: Task 1154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5cf7617f-65ca-42ce-8d4e-8fae13830690_isrForCrosstalkSources_855441_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/5cf7617f-65ca-42ce-8d4e-8fae13830690_isrForCrosstalkSources_855441_14.stderr parsl.dataflow.dflow INFO: Task 1658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ddbcb1e3410b9c7271fe88fff13f7ae with result from task 1658 parsl.dataflow.dflow DEBUG: Task 1658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e81895a8-47a9-41ce-8299-6a44036c1bc6_isrForCrosstalkSources_855408_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/e81895a8-47a9-41ce-8299-6a44036c1bc6_isrForCrosstalkSources_855408_42.stderr parsl.dataflow.dflow INFO: Task 1657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56e174125b46b5b47fa6a1ff4ad474ae with result from task 1657 parsl.dataflow.dflow DEBUG: Task 1657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/00a2d0b3-f3de-4734-9d8b-4662b422f0cd_isrForCrosstalkSources_855415_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/00a2d0b3-f3de-4734-9d8b-4662b422f0cd_isrForCrosstalkSources_855415_44.stderr parsl.dataflow.dflow INFO: Task 1659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2780389fe8526003c273e9fdbd6ba6 with result from task 1659 parsl.dataflow.dflow DEBUG: Task 1659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/838d8c99-052b-4cc9-b70c-dd3328f2c41a_isrForCrosstalkSources_855426_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/838d8c99-052b-4cc9-b70c-dd3328f2c41a_isrForCrosstalkSources_855426_47.stderr parsl.dataflow.dflow INFO: Task 1653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 683d38f7a4ccfe205313b8deb1c9fd6d with result from task 1653 parsl.dataflow.dflow DEBUG: Task 1653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b406c6c-c3e9-437b-9350-1d3bb9be8e79_isrForCrosstalkSources_855447_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/0b406c6c-c3e9-437b-9350-1d3bb9be8e79_isrForCrosstalkSources_855447_12.stderr parsl.dataflow.dflow INFO: Task 1674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138e96d8a314fef852576eed091ef104 with result from task 1674 parsl.dataflow.dflow DEBUG: Task 1674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b29e330-ce35-4366-aa3f-211e453e8fc0_isrForCrosstalkSources_855398_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/8b29e330-ce35-4366-aa3f-211e453e8fc0_isrForCrosstalkSources_855398_29.stderr parsl.dataflow.dflow INFO: Task 1675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be72964ec072b6b23d33cd5a2ecd0416 with result from task 1675 parsl.dataflow.dflow DEBUG: Task 1675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d579c26-4040-4dad-8d61-59825f8d3d9a_isrForCrosstalkSources_855444_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/5d579c26-4040-4dad-8d61-59825f8d3d9a_isrForCrosstalkSources_855444_4.stderr parsl.dataflow.dflow INFO: Task 1668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29c2184fb807a6770cb47f138d0d866b with result from task 1668 parsl.dataflow.dflow DEBUG: Task 1668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1b99ef52-8e5a-492d-b7da-35dbb5156158_isrForCrosstalkSources_855437_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1b99ef52-8e5a-492d-b7da-35dbb5156158_isrForCrosstalkSources_855437_28.stderr parsl.dataflow.dflow INFO: Task 1660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0072a164b48d6088a0c0237b07b4c9ff with result from task 1660 parsl.dataflow.dflow DEBUG: Task 1660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a68a9cf5-c0ab-4588-a2f4-d72b0b72fe20_isrForCrosstalkSources_855381_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/a68a9cf5-c0ab-4588-a2f4-d72b0b72fe20_isrForCrosstalkSources_855381_45.stderr parsl.dataflow.dflow INFO: Task 1672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08acfb4f762812106b78426d2bb647f2 with result from task 1672 parsl.dataflow.dflow DEBUG: Task 1672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/affa07b4-ea68-4281-aea1-cf3f33ad16c5_isrForCrosstalkSources_855402_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/affa07b4-ea68-4281-aea1-cf3f33ad16c5_isrForCrosstalkSources_855402_39.stderr parsl.dataflow.dflow INFO: Task 1663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcef23ef19c86e81db4129f86bd6fb4e with result from task 1663 parsl.dataflow.dflow DEBUG: Task 1663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/cc477949-2725-42ca-91a3-37f9427a9410_isrForCrosstalkSources_855434_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/cc477949-2725-42ca-91a3-37f9427a9410_isrForCrosstalkSources_855434_11.stderr parsl.dataflow.dflow INFO: Task 1662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f4f7dedcd727c39a92778844ea724d8 with result from task 1662 parsl.dataflow.dflow DEBUG: Task 1662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/21df260b-1aa7-4c72-a022-bcc1a87146f6_isrForCrosstalkSources_855417_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/21df260b-1aa7-4c72-a022-bcc1a87146f6_isrForCrosstalkSources_855417_27.stderr parsl.dataflow.dflow INFO: Task 1673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0051b1e470c45ce5e41227d9c6d0b7d8 with result from task 1673 parsl.dataflow.dflow DEBUG: Task 1673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/a1579e62-59cc-472b-9a9a-e6abd0c57181_isrForCrosstalkSources_855447_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/a1579e62-59cc-472b-9a9a-e6abd0c57181_isrForCrosstalkSources_855447_28.stderr parsl.dataflow.dflow INFO: Task 1669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36763112862df3822d9eeed2dccf21f4 with result from task 1669 parsl.dataflow.dflow DEBUG: Task 1669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/6b8eac9c-f6d6-4406-a1ae-71151b1a8a82_isrForCrosstalkSources_855384_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/6b8eac9c-f6d6-4406-a1ae-71151b1a8a82_isrForCrosstalkSources_855384_3.stderr parsl.dataflow.dflow INFO: Task 1661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ded388f54e713e9e61367226086ffe37 with result from task 1661 parsl.dataflow.dflow DEBUG: Task 1661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/34fbdaf4-e501-4b0f-92e7-9318e3d9d208_isrForCrosstalkSources_855384_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/34fbdaf4-e501-4b0f-92e7-9318e3d9d208_isrForCrosstalkSources_855384_20.stderr parsl.dataflow.dflow INFO: Task 1671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf8046593f43931ba5dc3b008ccb868f with result from task 1671 parsl.dataflow.dflow DEBUG: Task 1671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/298add35-747c-4eb8-94c5-55a825ca3528_isrForCrosstalkSources_855407_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/298add35-747c-4eb8-94c5-55a825ca3528_isrForCrosstalkSources_855407_44.stderr parsl.dataflow.dflow INFO: Task 1646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca86b6d51756d1200f931ff32deb77d9 with result from task 1646 parsl.dataflow.dflow DEBUG: Task 1646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/2f9eba95-a0d0-4c93-9c88-1fde030b456d_isrForCrosstalkSources_855433_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/2f9eba95-a0d0-4c93-9c88-1fde030b456d_isrForCrosstalkSources_855433_15.stderr parsl.dataflow.dflow INFO: Task 1667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78391d9319647ee251b4b8cb5679c3d4 with result from task 1667 parsl.dataflow.dflow DEBUG: Task 1667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2ee79f94-2f7c-4ed4-8f27-624429e83418_isrForCrosstalkSources_855422_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/2ee79f94-2f7c-4ed4-8f27-624429e83418_isrForCrosstalkSources_855422_20.stderr parsl.dataflow.dflow INFO: Task 1664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f58ba86b384008f1534c89b038a1abc7 with result from task 1664 parsl.dataflow.dflow DEBUG: Task 1664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/048e3640-8036-4488-a7e3-ddeec8500658_isrForCrosstalkSources_855428_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/048e3640-8036-4488-a7e3-ddeec8500658_isrForCrosstalkSources_855428_45.stderr parsl.dataflow.dflow INFO: Task 1665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5ef8718085f42bdc8c7ae9a387ab697 with result from task 1665 parsl.dataflow.dflow DEBUG: Task 1665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/92e1c4c2-552e-49dc-a192-2d5d8bce5b99_isrForCrosstalkSources_855405_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/92e1c4c2-552e-49dc-a192-2d5d8bce5b99_isrForCrosstalkSources_855405_29.stderr parsl.dataflow.dflow INFO: Task 1670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e901df9079aceea9f4452477271b61a8 with result from task 1670 parsl.dataflow.dflow DEBUG: Task 1670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/afc89b97-a142-419a-a6cb-9f1416f21deb_isrForCrosstalkSources_855440_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/afc89b97-a142-419a-a6cb-9f1416f21deb_isrForCrosstalkSources_855440_43.stderr parsl.dataflow.dflow INFO: Task 1618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8167cce75ebaec99a9f1b712ec1a61 with result from task 1618 parsl.dataflow.dflow DEBUG: Task 1618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4013b9ad-8f28-40c8-9821-82d4176da316_isrForCrosstalkSources_855385_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4013b9ad-8f28-40c8-9821-82d4176da316_isrForCrosstalkSources_855385_14.stderr parsl.dataflow.dflow INFO: Task 1656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 243446f3b17073a854f7ca57af486181 with result from task 1656 parsl.dataflow.dflow DEBUG: Task 1656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6fb1cf1f-3939-4c53-a3d9-a4aae52ec7b1_isrForCrosstalkSources_855420_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/6fb1cf1f-3939-4c53-a3d9-a4aae52ec7b1_isrForCrosstalkSources_855420_41.stderr 2024-03-09 14:56:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964337 proc_lsst.multi INFO: [multi] job_id 16964337 2024-03-09 14:56:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 245 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 245 2024-03-09 14:56:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964337 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964337 parsl.executors.status_handling DEBUG: Launched block 300 on executor multi with job ID 16964337 parsl.executors.status_handling INFO: Allocated block ID 301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-301 2024-03-09 14:56:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 245 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 245 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b16f8de9d4d3dcad72751a65ce90b8f6 with result from task 1565 parsl.dataflow.dflow DEBUG: Task 1565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/69a60aa5-9a45-4a1d-a894-c588ec595756_isrForCrosstalkSources_855408_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/69a60aa5-9a45-4a1d-a894-c588ec595756_isrForCrosstalkSources_855408_14.stderr parsl.dataflow.dflow INFO: Task 1654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b3e8aa8dcb143f82ab3ae24dccf8027 with result from task 1654 parsl.dataflow.dflow DEBUG: Task 1654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b5ae10fb-ea71-4367-80bf-5994b1ca8efb_isrForCrosstalkSources_855438_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b5ae10fb-ea71-4367-80bf-5994b1ca8efb_isrForCrosstalkSources_855438_18.stderr parsl.dataflow.dflow INFO: Task 1642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691a2f69195c72a12841a5082459387b with result from task 1642 parsl.dataflow.dflow DEBUG: Task 1642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ba50978-7644-44ef-9e68-5ac6bc3d8268_isrForCrosstalkSources_855383_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/5ba50978-7644-44ef-9e68-5ac6bc3d8268_isrForCrosstalkSources_855383_43.stderr parsl.dataflow.dflow INFO: Task 1676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f0b8f217cac0cbd96dea832222b3d3 with result from task 1676 parsl.dataflow.dflow DEBUG: Task 1676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8cf1c856-b7c3-4a64-b83b-1b957224d7e4_isrForCrosstalkSources_855453_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8cf1c856-b7c3-4a64-b83b-1b957224d7e4_isrForCrosstalkSources_855453_25.stderr parsl.dataflow.dflow INFO: Task 1655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2c8e0b1a252ac1377c2dd1271cdf909 with result from task 1655 parsl.dataflow.dflow DEBUG: Task 1655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/fabfb599-02c2-48de-9349-0b11231ce336_isrForCrosstalkSources_855437_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/fabfb599-02c2-48de-9349-0b11231ce336_isrForCrosstalkSources_855437_57.stderr parsl.dataflow.dflow INFO: Task 1579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d1bb162337f0d856cc4701c1c0456a3 with result from task 1579 parsl.dataflow.dflow DEBUG: Task 1579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aacf0d71-6ef0-4724-ba20-51d06b38ef55_isrForCrosstalkSources_855426_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/aacf0d71-6ef0-4724-ba20-51d06b38ef55_isrForCrosstalkSources_855426_27.stderr parsl.dataflow.dflow INFO: Task 1537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5745a9f83e55de64cf0fe1dec672901 with result from task 1537 parsl.dataflow.dflow DEBUG: Task 1537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/fd993576-4c50-4354-b5a4-271d7f888cd2_isrForCrosstalkSources_855445_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/fd993576-4c50-4354-b5a4-271d7f888cd2_isrForCrosstalkSources_855445_5.stderr parsl.dataflow.dflow INFO: Task 1581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 499fe39ceda4f7900167073fc7554ca9 with result from task 1581 parsl.dataflow.dflow DEBUG: Task 1581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a0e23a70-9b12-4736-920c-34b62b31eed8_isrForCrosstalkSources_855395_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/a0e23a70-9b12-4736-920c-34b62b31eed8_isrForCrosstalkSources_855395_8.stderr parsl.dataflow.dflow INFO: Task 1600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6254911fee447ee959db5b6f16c7844 with result from task 1600 parsl.dataflow.dflow DEBUG: Task 1600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2363e20a-2c30-4b9b-ab13-c81fa46dcc1d_isrForCrosstalkSources_855418_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2363e20a-2c30-4b9b-ab13-c81fa46dcc1d_isrForCrosstalkSources_855418_47.stderr parsl.dataflow.dflow INFO: Task 1585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc78d9b637eaa9a874d5601914037f59 with result from task 1585 parsl.dataflow.dflow DEBUG: Task 1585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4b09ecc4-cd08-4644-b707-aad2c1617683_isrForCrosstalkSources_855389_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/4b09ecc4-cd08-4644-b707-aad2c1617683_isrForCrosstalkSources_855389_24.stderr parsl.dataflow.dflow INFO: Task 1580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06b3d240173982cb9801ef788f3ebc39 with result from task 1580 parsl.dataflow.dflow DEBUG: Task 1580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9879b347-14db-4226-aaaf-1f87cf329755_isrForCrosstalkSources_855389_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/9879b347-14db-4226-aaaf-1f87cf329755_isrForCrosstalkSources_855389_35.stderr parsl.dataflow.dflow INFO: Task 1590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78e5b8febc9e0b6e175bd719077afdab with result from task 1590 parsl.dataflow.dflow DEBUG: Task 1590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/04c31364-febb-4154-ae4c-a9858a969e23_isrForCrosstalkSources_855426_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/04c31364-febb-4154-ae4c-a9858a969e23_isrForCrosstalkSources_855426_1.stderr parsl.dataflow.dflow INFO: Task 1589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f92f14ad325f0ff397636e713b3c5549 with result from task 1589 parsl.dataflow.dflow DEBUG: Task 1589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/08f36281-6dcb-46f3-a61c-96df9703f129_isrForCrosstalkSources_855455_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/08f36281-6dcb-46f3-a61c-96df9703f129_isrForCrosstalkSources_855455_50.stderr parsl.dataflow.dflow INFO: Task 1541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e021efea54f450d0305941d42fdbd04d with result from task 1541 parsl.dataflow.dflow DEBUG: Task 1541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/7b91a2df-d5a7-4dbe-b04e-5f03e620b9c7_isrForCrosstalkSources_855383_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/7b91a2df-d5a7-4dbe-b04e-5f03e620b9c7_isrForCrosstalkSources_855383_39.stderr parsl.dataflow.dflow INFO: Task 1583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d19f752df8d6eb5296357a04e1a11a5 with result from task 1583 parsl.dataflow.dflow DEBUG: Task 1583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7446fc0b-af74-40b4-830d-0d1fb69122f6_isrForCrosstalkSources_855393_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7446fc0b-af74-40b4-830d-0d1fb69122f6_isrForCrosstalkSources_855393_6.stderr parsl.dataflow.dflow INFO: Task 1601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab79107f6e5b6573383c4b0ef38f0f1c with result from task 1601 parsl.dataflow.dflow DEBUG: Task 1601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/7a1250a2-8a27-4f85-a5fe-41e175f7acc2_isrForCrosstalkSources_855416_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/7a1250a2-8a27-4f85-a5fe-41e175f7acc2_isrForCrosstalkSources_855416_34.stderr parsl.dataflow.dflow INFO: Task 1539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a54c98af35937e42f4b7ed17ce9ab7df with result from task 1539 parsl.dataflow.dflow DEBUG: Task 1539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8652b756-8de9-497c-b29e-1ffee48f5bc4_isrForCrosstalkSources_855407_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/8652b756-8de9-497c-b29e-1ffee48f5bc4_isrForCrosstalkSources_855407_47.stderr parsl.dataflow.dflow INFO: Task 1584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcce67e71ad6617960fb1f72969a1370 with result from task 1584 parsl.dataflow.dflow DEBUG: Task 1584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fefe9b90-1c9c-4809-8f28-10dd7ff7682e_isrForCrosstalkSources_855385_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/fefe9b90-1c9c-4809-8f28-10dd7ff7682e_isrForCrosstalkSources_855385_22.stderr parsl.dataflow.dflow INFO: Task 1587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce3c3ca9b630b741a5f2303e01bb6aa9 with result from task 1587 parsl.dataflow.dflow DEBUG: Task 1587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ea32baf6-d850-4c13-af69-631a1cb647e8_isrForCrosstalkSources_855396_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ea32baf6-d850-4c13-af69-631a1cb647e8_isrForCrosstalkSources_855396_33.stderr parsl.dataflow.dflow INFO: Task 1594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f021d27ed49270e6ead965611bc0b60 with result from task 1594 parsl.dataflow.dflow DEBUG: Task 1594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/079a7deb-eed3-4598-90a7-68dd8c317c28_isrForCrosstalkSources_855453_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/079a7deb-eed3-4598-90a7-68dd8c317c28_isrForCrosstalkSources_855453_27.stderr parsl.dataflow.dflow INFO: Task 1598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40c8b85195d2417f0def0225c7da762d with result from task 1598 parsl.dataflow.dflow DEBUG: Task 1598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/5001c8b2-d62c-4d25-a0aa-83f34a6f3fbe_isrForCrosstalkSources_855447_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/5001c8b2-d62c-4d25-a0aa-83f34a6f3fbe_isrForCrosstalkSources_855447_31.stderr parsl.dataflow.dflow INFO: Task 1595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e3c21a5ef29efb789234eb38bd0f262 with result from task 1595 parsl.dataflow.dflow DEBUG: Task 1595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/753c04a6-5a41-4167-8267-508234a21d98_isrForCrosstalkSources_855406_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/753c04a6-5a41-4167-8267-508234a21d98_isrForCrosstalkSources_855406_21.stderr parsl.dataflow.dflow INFO: Task 1597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7efcb5ba0fc698ad94497dd21b35ee45 with result from task 1597 parsl.dataflow.dflow DEBUG: Task 1597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2d5d00c9-1286-4520-b7f5-7d5eb47cffeb_isrForCrosstalkSources_855396_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2d5d00c9-1286-4520-b7f5-7d5eb47cffeb_isrForCrosstalkSources_855396_61.stderr parsl.dataflow.dflow INFO: Task 1593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9175d05730cb29cbbeae005f4e79ad8 with result from task 1593 parsl.dataflow.dflow DEBUG: Task 1593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d75f4b4c-5ca7-4f5e-8eed-c6aff9532db2_isrForCrosstalkSources_855417_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/d75f4b4c-5ca7-4f5e-8eed-c6aff9532db2_isrForCrosstalkSources_855417_26.stderr parsl.dataflow.dflow INFO: Task 1586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b2b5dab7733f5cb18c4e37c4921ccf4 with result from task 1586 parsl.dataflow.dflow DEBUG: Task 1586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8d20eb3c-feb3-4e28-846b-f4d15a1a3756_isrForCrosstalkSources_855408_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/8d20eb3c-feb3-4e28-846b-f4d15a1a3756_isrForCrosstalkSources_855408_12.stderr parsl.dataflow.dflow INFO: Task 1542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18d278424da16c7315987cc3bacbac48 with result from task 1542 parsl.dataflow.dflow DEBUG: Task 1542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/523baaa9-1b63-4424-aa68-f5d58f1586a8_isrForCrosstalkSources_855390_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/523baaa9-1b63-4424-aa68-f5d58f1586a8_isrForCrosstalkSources_855390_29.stderr parsl.dataflow.dflow INFO: Task 1538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e54a7f39bf11303dd2b372a8d2ba62 with result from task 1538 parsl.dataflow.dflow DEBUG: Task 1538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b247b6e-2cd7-4193-bebe-af66c6fa9b02_isrForCrosstalkSources_855459_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/6b247b6e-2cd7-4193-bebe-af66c6fa9b02_isrForCrosstalkSources_855459_52.stderr parsl.dataflow.dflow INFO: Task 1582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3955c6cc7bf2411edbc6fc106c2f74e8 with result from task 1582 parsl.dataflow.dflow DEBUG: Task 1582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0c40509-c6af-4c50-a9f0-78f7aa174604_isrForCrosstalkSources_855408_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/b0c40509-c6af-4c50-a9f0-78f7aa174604_isrForCrosstalkSources_855408_56.stderr parsl.dataflow.dflow INFO: Task 1596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7dc0d03d8409f054c7b0db9eb9e589f with result from task 1596 parsl.dataflow.dflow DEBUG: Task 1596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/baf6598c-f3a2-4179-9284-53600cd31cfb_isrForCrosstalkSources_855427_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/baf6598c-f3a2-4179-9284-53600cd31cfb_isrForCrosstalkSources_855427_16.stderr parsl.dataflow.dflow INFO: Task 1599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbbd8c23a4d0818a0788348b0fc41294 with result from task 1599 parsl.dataflow.dflow DEBUG: Task 1599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b289ea68-2f27-4129-998d-152530921f86_isrForCrosstalkSources_855407_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b289ea68-2f27-4129-998d-152530921f86_isrForCrosstalkSources_855407_37.stderr parsl.dataflow.dflow INFO: Task 1588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cc7dbbaf6206c6110b045554c246b3f with result from task 1588 parsl.dataflow.dflow DEBUG: Task 1588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fa28ce68-162a-4cc9-b0db-0dc0b7cec237_isrForCrosstalkSources_855420_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/fa28ce68-162a-4cc9-b0db-0dc0b7cec237_isrForCrosstalkSources_855420_11.stderr parsl.dataflow.dflow INFO: Task 1592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 306a2286a0c6fa3fa5032e81eb18630a with result from task 1592 parsl.dataflow.dflow DEBUG: Task 1592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a72d6aec-28b3-4cfb-a3dc-e1442b9ac0d5_isrForCrosstalkSources_855431_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/a72d6aec-28b3-4cfb-a3dc-e1442b9ac0d5_isrForCrosstalkSources_855431_5.stderr parsl.dataflow.dflow INFO: Task 1543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 786a954c71848a95ddb361766e4306b5 with result from task 1543 parsl.dataflow.dflow DEBUG: Task 1543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bd500bbb-ad57-4819-bf82-c41a984166e9_isrForCrosstalkSources_855443_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/bd500bbb-ad57-4819-bf82-c41a984166e9_isrForCrosstalkSources_855443_9.stderr 2024-03-09 14:56:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964338 proc_lsst.multi INFO: [multi] job_id 16964338 2024-03-09 14:56:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 246 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 246 2024-03-09 14:56:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964338 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964338 parsl.executors.status_handling DEBUG: Launched block 301 on executor multi with job ID 16964338 parsl.executors.status_handling INFO: Allocated block ID 302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-302 2024-03-09 14:56:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 246 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 246 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9172a15df8b7b5aa0c2ddd063417144e with result from task 1591 parsl.dataflow.dflow DEBUG: Task 1591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3ec14f21-a786-4e20-b31b-0f996df837b8_isrForCrosstalkSources_855386_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/3ec14f21-a786-4e20-b31b-0f996df837b8_isrForCrosstalkSources_855386_57.stderr parsl.dataflow.dflow INFO: Task 1540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73257de939d687c1d331108e3cf5e336 with result from task 1540 parsl.dataflow.dflow DEBUG: Task 1540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/295f32f4-49fa-43d2-bfeb-65cedf898932_isrForCrosstalkSources_855449_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/295f32f4-49fa-43d2-bfeb-65cedf898932_isrForCrosstalkSources_855449_61.stderr parsl.dataflow.dflow INFO: Task 1602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39113511d016b77a78a68f4011ad7524 with result from task 1602 parsl.dataflow.dflow DEBUG: Task 1602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9e005463-3055-40e6-ad92-d0b3188fa1b2_isrForCrosstalkSources_855411_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9e005463-3055-40e6-ad92-d0b3188fa1b2_isrForCrosstalkSources_855411_61.stderr parsl.dataflow.dflow INFO: Task 1544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8d1a890f71c9a23871e4036dae8d7ca with result from task 1544 parsl.dataflow.dflow DEBUG: Task 1544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3273138-d17d-4a52-a926-cf5f6dcfd51a_isrForCrosstalkSources_855414_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/e3273138-d17d-4a52-a926-cf5f6dcfd51a_isrForCrosstalkSources_855414_12.stderr 2024-03-09 14:56:05 proc_lsst.multi:170 [INFO] [multi] job_id 16964339 proc_lsst.multi INFO: [multi] job_id 16964339 2024-03-09 14:56:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 247 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 247 2024-03-09 14:56:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964339 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964339 parsl.executors.status_handling DEBUG: Launched block 302 on executor multi with job ID 16964339 parsl.executors.status_handling INFO: Allocated block ID 303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-303 2024-03-09 14:56:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 247 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 247 2024-03-09 14:56:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964340 proc_lsst.multi INFO: [multi] job_id 16964340 2024-03-09 14:56:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 248 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 248 2024-03-09 14:56:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964340 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964340 parsl.executors.status_handling DEBUG: Launched block 303 on executor multi with job ID 16964340 parsl.executors.status_handling INFO: Allocated block ID 304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-304 2024-03-09 14:56:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 248 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 248 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964341 proc_lsst.multi INFO: [multi] job_id 16964341 2024-03-09 14:56:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 249 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 249 2024-03-09 14:56:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964341 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964341 parsl.executors.status_handling DEBUG: Launched block 304 on executor multi with job ID 16964341 parsl.executors.status_handling INFO: Allocated block ID 305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-305 2024-03-09 14:56:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 249 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 249 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964342 proc_lsst.multi INFO: [multi] job_id 16964342 2024-03-09 14:56:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 250 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 250 2024-03-09 14:56:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964342 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964342 parsl.executors.status_handling DEBUG: Launched block 305 on executor multi with job ID 16964342 parsl.executors.status_handling INFO: Allocated block ID 306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-306 2024-03-09 14:56:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 250 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 250 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:06 proc_lsst.multi:170 [INFO] [multi] job_id 16964343 proc_lsst.multi INFO: [multi] job_id 16964343 2024-03-09 14:56:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 251 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 251 2024-03-09 14:56:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964343 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964343 parsl.executors.status_handling DEBUG: Launched block 306 on executor multi with job ID 16964343 parsl.executors.status_handling INFO: Allocated block ID 307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-307 2024-03-09 14:56:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 251 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 251 2024-03-09 14:56:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0f1b6030b702a9f90b846ab3c33ba50 with result from task 1686 parsl.dataflow.dflow DEBUG: Task 1686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/86e3efa3-4153-4554-88d1-622cdbd400be_isrForCrosstalkSources_855423_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/86e3efa3-4153-4554-88d1-622cdbd400be_isrForCrosstalkSources_855423_28.stderr parsl.dataflow.dflow INFO: Task 1683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 772f9ec448263b3ade5fe92ed430e036 with result from task 1683 parsl.dataflow.dflow DEBUG: Task 1683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/cd28c47b-24c8-4b37-b204-11dbc786d18b_isrForCrosstalkSources_855457_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/cd28c47b-24c8-4b37-b204-11dbc786d18b_isrForCrosstalkSources_855457_56.stderr parsl.dataflow.dflow INFO: Task 1682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46060b239b62749d8868b0c521ef4279 with result from task 1682 parsl.dataflow.dflow DEBUG: Task 1682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0b4df0c5-2e06-4970-9812-b2b4f513dfe6_isrForCrosstalkSources_855424_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0b4df0c5-2e06-4970-9812-b2b4f513dfe6_isrForCrosstalkSources_855424_36.stderr parsl.dataflow.dflow INFO: Task 1699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26ab668fa29beda244be42443175ae24 with result from task 1699 parsl.dataflow.dflow DEBUG: Task 1699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c420ed87-26e8-4edb-b154-f3b4ec9134ba_isrForCrosstalkSources_855417_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c420ed87-26e8-4edb-b154-f3b4ec9134ba_isrForCrosstalkSources_855417_20.stderr parsl.dataflow.dflow INFO: Task 1704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 494315d8181ae9b1b858a9d188c14fcf with result from task 1704 parsl.dataflow.dflow DEBUG: Task 1704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/08a2fead-a820-4ede-86cc-e01b38ed34ef_isrForCrosstalkSources_855385_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/08a2fead-a820-4ede-86cc-e01b38ed34ef_isrForCrosstalkSources_855385_21.stderr parsl.dataflow.dflow INFO: Task 1697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d14db5cda72c09288538f6eafa11bd4d with result from task 1697 2024-03-09 14:56:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964344 proc_lsst.multi INFO: [multi] job_id 16964344 2024-03-09 14:56:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 252 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 252 2024-03-09 14:56:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964344 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964344 parsl.dataflow.dflow DEBUG: Task 1697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60ec4336-a9b6-4d17-84f9-6835540f42e6_isrForCrosstalkSources_855433_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/60ec4336-a9b6-4d17-84f9-6835540f42e6_isrForCrosstalkSources_855433_12.stderr parsl.dataflow.dflow INFO: Task 1696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515b6a1ede694e12e978da8599416098 with result from task 1696 parsl.executors.status_handling DEBUG: Launched block 307 on executor multi with job ID 16964344 parsl.executors.status_handling INFO: Allocated block ID 308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-308 2024-03-09 14:56:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 252 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 252 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 1696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8a7688b9-5f10-446c-b223-bb2873598832_isrForCrosstalkSources_855447_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8a7688b9-5f10-446c-b223-bb2873598832_isrForCrosstalkSources_855447_8.stderr parsl.dataflow.dflow INFO: Task 4067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd285e67568b958f8043fd637ca0934d with result from task 4067 parsl.dataflow.dflow DEBUG: Task 4067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/88ef6291-d910-430f-a865-f5c712ffceb7_isrForCrosstalkSources_855398_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/88ef6291-d910-430f-a865-f5c712ffceb7_isrForCrosstalkSources_855398_55.stderr parsl.dataflow.dflow INFO: Task 1684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3387595304fa98aac585a57d4e0a7ca with result from task 1684 parsl.dataflow.dflow DEBUG: Task 1684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/293f3222-bede-42f6-b000-fd6d3fac93c1_isrForCrosstalkSources_855446_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/293f3222-bede-42f6-b000-fd6d3fac93c1_isrForCrosstalkSources_855446_1.stderr parsl.dataflow.dflow INFO: Task 1701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8420b7730cfb58e9eb013717394a876 with result from task 1701 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 1701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/db8e3bab-d755-40a8-aa83-6d435e7ea652_isrForCrosstalkSources_855446_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/db8e3bab-d755-40a8-aa83-6d435e7ea652_isrForCrosstalkSources_855446_22.stderr parsl.dataflow.dflow INFO: Task 1700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ea737f3bf23aa7a671de75848852848 with result from task 1700 parsl.dataflow.dflow DEBUG: Task 1700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/dc66eca6-a5eb-4827-9d5d-6716935aea59_isrForCrosstalkSources_855390_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/dc66eca6-a5eb-4827-9d5d-6716935aea59_isrForCrosstalkSources_855390_26.stderr parsl.dataflow.dflow INFO: Task 1691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34de3851bd2d8633511dbda65e78455c with result from task 1691 parsl.dataflow.dflow DEBUG: Task 1691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8ef54967-6965-4c47-bde9-28779d95c952_isrForCrosstalkSources_855441_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/8ef54967-6965-4c47-bde9-28779d95c952_isrForCrosstalkSources_855441_13.stderr parsl.dataflow.dflow INFO: Task 1692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e09a710cd660f53ffa60bee717608b0 with result from task 1692 parsl.dataflow.dflow DEBUG: Task 1692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/82248b50-c601-44bc-bb2e-8d42105505ae_isrForCrosstalkSources_855387_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/82248b50-c601-44bc-bb2e-8d42105505ae_isrForCrosstalkSources_855387_2.stderr parsl.dataflow.dflow INFO: Task 4069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2013a38afe315b0e175e6e9de3c47565 with result from task 4069 parsl.dataflow.dflow DEBUG: Task 4069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/90ae8827-d561-46c4-9f90-40e176883118_isrForCrosstalkSources_855440_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/90ae8827-d561-46c4-9f90-40e176883118_isrForCrosstalkSources_855440_20.stderr parsl.dataflow.dflow INFO: Task 4068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ce7ec6cbc328f93a583eabafbb21ed0 with result from task 4068 parsl.dataflow.dflow DEBUG: Task 4068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67ecdf70-c74a-4acc-80ad-8711c3f490da_isrForCrosstalkSources_855382_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67ecdf70-c74a-4acc-80ad-8711c3f490da_isrForCrosstalkSources_855382_19.stderr parsl.dataflow.dflow INFO: Task 4066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad4a8b4125d7eec3c3207ddb06a81187 with result from task 4066 parsl.dataflow.dflow DEBUG: Task 4066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f7d574d0-43ab-4678-8a0b-66e0c21bb352_isrForCrosstalkSources_855391_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/f7d574d0-43ab-4678-8a0b-66e0c21bb352_isrForCrosstalkSources_855391_44.stderr parsl.dataflow.dflow INFO: Task 1687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e29d6232201082bcf5f55833fbbcb89 with result from task 1687 parsl.dataflow.dflow DEBUG: Task 1687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/b38ce4ee-a64e-4d86-a151-a6418ecebf25_isrForCrosstalkSources_855383_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/b38ce4ee-a64e-4d86-a151-a6418ecebf25_isrForCrosstalkSources_855383_58.stderr parsl.dataflow.dflow INFO: Task 1685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03d9e88925022e41cf4a399cd941fe4f with result from task 1685 parsl.dataflow.dflow DEBUG: Task 1685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb0df5db-f9d0-4dfc-8d3f-57cd72ed3b61_isrForCrosstalkSources_855386_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/eb0df5db-f9d0-4dfc-8d3f-57cd72ed3b61_isrForCrosstalkSources_855386_50.stderr parsl.dataflow.dflow INFO: Task 1694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0b99ed85c534e27b733c1b843351350 with result from task 1694 parsl.dataflow.dflow DEBUG: Task 1694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0269c3bf-099a-42cf-add3-5a1de4281538_isrForCrosstalkSources_855430_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/0269c3bf-099a-42cf-add3-5a1de4281538_isrForCrosstalkSources_855430_33.stderr parsl.dataflow.dflow INFO: Task 1688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb64e95a183574624ea84cd00a7605da with result from task 1688 parsl.dataflow.dflow DEBUG: Task 1688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/265739eb-af6e-4fb2-872b-9d67cda32dea_isrForCrosstalkSources_855408_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/265739eb-af6e-4fb2-872b-9d67cda32dea_isrForCrosstalkSources_855408_41.stderr parsl.dataflow.dflow INFO: Task 1702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 054a96ca816e53de5aa5b551b105ff0f with result from task 1702 parsl.dataflow.dflow DEBUG: Task 1702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1fc488b8-5907-4556-9ccd-4b1979ed811a_isrForCrosstalkSources_855407_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/1fc488b8-5907-4556-9ccd-4b1979ed811a_isrForCrosstalkSources_855407_5.stderr parsl.dataflow.dflow INFO: Task 4065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f5da4b8f99f87823bfa92a9f0d33016 with result from task 4065 parsl.dataflow.dflow DEBUG: Task 4065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a776f574-d089-42ca-a3cb-e0cb955450d3_isrForCrosstalkSources_855434_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a776f574-d089-42ca-a3cb-e0cb955450d3_isrForCrosstalkSources_855434_62.stderr parsl.dataflow.dflow INFO: Task 1698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca05f80f1fbacd050454588a1ae88ce with result from task 1698 parsl.dataflow.dflow DEBUG: Task 1698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b98d7050-aab7-48e5-8dde-8b24f9c56465_isrForCrosstalkSources_855407_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b98d7050-aab7-48e5-8dde-8b24f9c56465_isrForCrosstalkSources_855407_8.stderr parsl.dataflow.dflow INFO: Task 1705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 377257d29c35d71a3406eb05573b840d with result from task 1705 parsl.dataflow.dflow DEBUG: Task 1705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/35371575-5b04-4187-bd7a-416c0cd69f9f_isrForCrosstalkSources_855458_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/35371575-5b04-4187-bd7a-416c0cd69f9f_isrForCrosstalkSources_855458_42.stderr parsl.dataflow.dflow INFO: Task 1695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32d3e341da6ca2e29e07b132870a35d7 with result from task 1695 parsl.dataflow.dflow DEBUG: Task 1695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8c7a55d3-908b-4752-9c62-e8b2547970ff_isrForCrosstalkSources_855449_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/8c7a55d3-908b-4752-9c62-e8b2547970ff_isrForCrosstalkSources_855449_24.stderr parsl.dataflow.dflow INFO: Task 1689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e141845aea96816f475cb07a52b7252c with result from task 1689 parsl.dataflow.dflow DEBUG: Task 1689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/80a0c7d7-f58f-4733-9b56-cd3709f371e9_isrForCrosstalkSources_855442_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/80a0c7d7-f58f-4733-9b56-cd3709f371e9_isrForCrosstalkSources_855442_40.stderr parsl.dataflow.dflow INFO: Task 1703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b38fc8564263f4a89b7fc3554ec7002 with result from task 1703 parsl.dataflow.dflow DEBUG: Task 1703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/bcf45702-47a2-4831-b3e2-8b575eb36a23_isrForCrosstalkSources_855399_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/bcf45702-47a2-4831-b3e2-8b575eb36a23_isrForCrosstalkSources_855399_46.stderr parsl.dataflow.dflow INFO: Task 1693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d0bc890b784870e6f2a8eaf19de7fb with result from task 1693 parsl.dataflow.dflow DEBUG: Task 1693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/eacf4fcf-a147-49db-b5cb-2c0e26b3dc9d_isrForCrosstalkSources_855425_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/eacf4fcf-a147-49db-b5cb-2c0e26b3dc9d_isrForCrosstalkSources_855425_27.stderr parsl.dataflow.dflow INFO: Task 4063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a629dd8ff0aec0376bb4e311d894647c with result from task 4063 parsl.dataflow.dflow DEBUG: Task 4063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c873f042-4a45-4997-8d7b-c0831f5b9a5c_isrForCrosstalkSources_855442_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/c873f042-4a45-4997-8d7b-c0831f5b9a5c_isrForCrosstalkSources_855442_23.stderr parsl.dataflow.dflow INFO: Task 4062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3c08f942c5d162502d7c95d3a2de4ff with result from task 4062 parsl.dataflow.dflow DEBUG: Task 4062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c27b1780-ec4b-4e3a-8a55-7915af2af541_isrForCrosstalkSources_855429_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/c27b1780-ec4b-4e3a-8a55-7915af2af541_isrForCrosstalkSources_855429_48.stderr parsl.dataflow.dflow INFO: Task 4064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aaa715e513c12ff66fd4bf3d825cc1d with result from task 4064 parsl.dataflow.dflow DEBUG: Task 4064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e8ecc3a2-1384-43c9-a6cb-2172c30648ca_isrForCrosstalkSources_855421_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e8ecc3a2-1384-43c9-a6cb-2172c30648ca_isrForCrosstalkSources_855421_10.stderr parsl.dataflow.dflow INFO: Task 1706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a36f16f0316fa072410aacbbcf52f304 with result from task 1706 parsl.dataflow.dflow DEBUG: Task 1706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1825a882-058b-4b14-83ce-928cff89d347_isrForCrosstalkSources_855401_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/1825a882-058b-4b14-83ce-928cff89d347_isrForCrosstalkSources_855401_24.stderr 2024-03-09 14:56:07 proc_lsst.multi:170 [INFO] [multi] job_id 16964345 proc_lsst.multi INFO: [multi] job_id 16964345 2024-03-09 14:56:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 253 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 253 2024-03-09 14:56:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964345 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964345 parsl.executors.status_handling DEBUG: Launched block 308 on executor multi with job ID 16964345 parsl.executors.status_handling INFO: Allocated block ID 309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-309 2024-03-09 14:56:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 253 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 253 2024-03-09 14:56:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964346 proc_lsst.multi INFO: [multi] job_id 16964346 2024-03-09 14:56:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 254 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 254 2024-03-09 14:56:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964346 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964346 parsl.executors.status_handling DEBUG: Launched block 309 on executor multi with job ID 16964346 parsl.executors.status_handling INFO: Allocated block ID 310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-310 2024-03-09 14:56:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 254 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 254 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:08 proc_lsst.multi:170 [INFO] [multi] job_id 16964347 proc_lsst.multi INFO: [multi] job_id 16964347 2024-03-09 14:56:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 255 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 255 2024-03-09 14:56:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964347 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964347 parsl.executors.status_handling DEBUG: Launched block 310 on executor multi with job ID 16964347 parsl.executors.status_handling INFO: Allocated block ID 311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-311 2024-03-09 14:56:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 14:56:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 14:56:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 255 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 255 2024-03-09 14:56:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,127.0.0.1,10.64.65.10,10.64.129.10,198.48.92.26,n3010 -p 0 -c 1.0 -m None --poll 10 --task_port=54749 --result_port=54298 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 14:56:09 proc_lsst.multi:170 [INFO] [multi] job_id 16964348 proc_lsst.multi INFO: [multi] job_id 16964348 2024-03-09 14:56:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 256 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 256 2024-03-09 14:56:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 14:56:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16964348 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16964348 parsl.executors.status_handling DEBUG: Launched block 311 on executor multi with job ID 16964348 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22375328864144 2024-03-09 14:56:09 proc_lsst.multi:146 [INFO] found job 77954 in provider local proc_lsst.multi INFO: found job 77954 in provider local parsl.dataflow.dflow INFO: Task 4744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf889f6e6c241ce35e38213629d19bfb with result from task 4744 2024-03-09 14:56:09 proc_lsst.multi:146 [INFO] found job 16964026 in provider astro proc_lsst.multi INFO: found job 16964026 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow DEBUG: Task 4744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ae224175-2962-4759-8e52-201bd73c3dd6_isrForCrosstalkSources_855420_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/ae224175-2962-4759-8e52-201bd73c3dd6_isrForCrosstalkSources_855420_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:09 proc_lsst.multi:146 [INFO] found job 16964027 in provider astro proc_lsst.multi INFO: found job 16964027 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:10 proc_lsst.multi:146 [INFO] found job 16964028 in provider astro proc_lsst.multi INFO: found job 16964028 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:10 proc_lsst.multi:146 [INFO] found job 16964029 in provider astro proc_lsst.multi INFO: found job 16964029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c470fc817af0021a172e729c1994cf98 with result from task 4638 parsl.dataflow.dflow DEBUG: Task 4638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/0fc6b1a5-1c0a-4c29-8547-19709c2e1535_isrForCrosstalkSources_855379_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/0fc6b1a5-1c0a-4c29-8547-19709c2e1535_isrForCrosstalkSources_855379_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:11 proc_lsst.multi:146 [INFO] found job 16964030 in provider astro proc_lsst.multi INFO: found job 16964030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec62c3beabb39e17b8e4389cfd55bcb with result from task 4676 parsl.dataflow.dflow DEBUG: Task 4676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c3f6120c-5dde-4a57-83e4-75d208022470_isrForCrosstalkSources_855397_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/c3f6120c-5dde-4a57-83e4-75d208022470_isrForCrosstalkSources_855397_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:11 proc_lsst.multi:146 [INFO] found job 16964031 in provider astro proc_lsst.multi INFO: found job 16964031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:11 proc_lsst.multi:146 [INFO] found job 16964032 in provider astro proc_lsst.multi INFO: found job 16964032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:11 proc_lsst.multi:146 [INFO] found job 16964033 in provider astro proc_lsst.multi INFO: found job 16964033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:12 proc_lsst.multi:146 [INFO] found job 16964034 in provider astro proc_lsst.multi INFO: found job 16964034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:12 proc_lsst.multi:146 [INFO] found job 16964035 in provider astro proc_lsst.multi INFO: found job 16964035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:12 proc_lsst.multi:146 [INFO] found job 16964036 in provider astro proc_lsst.multi INFO: found job 16964036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:12 proc_lsst.multi:146 [INFO] found job 16964037 in provider astro proc_lsst.multi INFO: found job 16964037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:13 proc_lsst.multi:146 [INFO] found job 16964038 in provider astro proc_lsst.multi INFO: found job 16964038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:13 proc_lsst.multi:146 [INFO] found job 16964039 in provider astro proc_lsst.multi INFO: found job 16964039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 4693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd1269f9e7958dac2caa61394eb495e4 with result from task 4693 parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 4693 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 4693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/f012c92a-07d3-4a5d-9843-b37998372471_isrForCrosstalkSources_855447_24.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 4693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/f012c92a-07d3-4a5d-9843-b37998372471_isrForCrosstalkSources_855447_24.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:13 proc_lsst.multi:146 [INFO] found job 16964040 in provider astro proc_lsst.multi INFO: found job 16964040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:14 proc_lsst.multi:146 [INFO] found job 16964041 in provider astro proc_lsst.multi INFO: found job 16964041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:14 proc_lsst.multi:146 [INFO] found job 16964042 in provider astro proc_lsst.multi INFO: found job 16964042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:14 proc_lsst.multi:146 [INFO] found job 16964043 in provider astro proc_lsst.multi INFO: found job 16964043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3648c43c40e5b1457fa803a912cbf30 with result from task 4734 parsl.dataflow.dflow DEBUG: Task 4734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/98be16d3-675e-42c3-8dfa-42bbac476242_isrForCrosstalkSources_855423_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/98be16d3-675e-42c3-8dfa-42bbac476242_isrForCrosstalkSources_855423_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:14 proc_lsst.multi:146 [INFO] found job 16964044 in provider astro proc_lsst.multi INFO: found job 16964044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:14 proc_lsst.multi:146 [INFO] found job 16964045 in provider astro proc_lsst.multi INFO: found job 16964045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aedf4e2e68bbf0c9517b49db4b4159b0 with result from task 4742 parsl.dataflow.dflow DEBUG: Task 4742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bbc282f9-9322-4753-9f90-548c70e13dfe_isrForCrosstalkSources_855424_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/bbc282f9-9322-4753-9f90-548c70e13dfe_isrForCrosstalkSources_855424_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:15 proc_lsst.multi:146 [INFO] found job 16964046 in provider astro proc_lsst.multi INFO: found job 16964046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:15 proc_lsst.multi:146 [INFO] found job 16964047 in provider astro proc_lsst.multi INFO: found job 16964047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408c517bf3ad36efdae441b14f718685 with result from task 4710 parsl.dataflow.dflow DEBUG: Task 4710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5c5fab54-8b75-4a97-af9f-f38c0ddb2317_isrForCrosstalkSources_855418_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5c5fab54-8b75-4a97-af9f-f38c0ddb2317_isrForCrosstalkSources_855418_40.stderr parsl.dataflow.dflow INFO: Task 4088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8152a393c1ed89f7bf3ec3fc324994 with result from task 4088 parsl.dataflow.dflow DEBUG: Task 4088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3fe68613-b225-4547-be19-8d928a75bdc3_isrForCrosstalkSources_855443_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3fe68613-b225-4547-be19-8d928a75bdc3_isrForCrosstalkSources_855443_23.stderr parsl.dataflow.dflow INFO: Task 1666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14292994f2c75c4e3c804e606687b7fb with result from task 1666 parsl.dataflow.dflow DEBUG: Task 1666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/902df79e-96e4-4ca5-bd17-62ed5a71455f_isrForCrosstalkSources_855422_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/902df79e-96e4-4ca5-bd17-62ed5a71455f_isrForCrosstalkSources_855422_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:15 proc_lsst.multi:146 [INFO] found job 16964048 in provider astro proc_lsst.multi INFO: found job 16964048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db06785e93761b238c6be04a0c0d1eea with result from task 4683 parsl.dataflow.dflow DEBUG: Task 4683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ce579ecf-41dd-428d-827f-8003b201e2d3_isrForCrosstalkSources_855388_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/ce579ecf-41dd-428d-827f-8003b201e2d3_isrForCrosstalkSources_855388_27.stderr parsl.dataflow.dflow INFO: Task 4689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 711b75bea022e7bba7923898427210a1 with result from task 4689 parsl.dataflow.dflow DEBUG: Task 4689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9a60d519-a5ba-489c-b2fa-6a41e8b4b288_isrForCrosstalkSources_855399_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/9a60d519-a5ba-489c-b2fa-6a41e8b4b288_isrForCrosstalkSources_855399_4.stderr parsl.dataflow.dflow INFO: Task 4637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc3817bf7bd57df66197ece90d240e5a with result from task 4637 parsl.dataflow.dflow DEBUG: Task 4637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/1137f112-6e4b-4578-8758-cda3dde6cabb_isrForCrosstalkSources_855402_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/1137f112-6e4b-4578-8758-cda3dde6cabb_isrForCrosstalkSources_855402_23.stderr parsl.dataflow.dflow INFO: Task 4447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba93d5d4aee2402d859367f7f432ff33 with result from task 4447 parsl.dataflow.dflow DEBUG: Task 4447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5e867190-e099-4fe5-8e87-b3426e78f217_isrForCrosstalkSources_855391_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/5e867190-e099-4fe5-8e87-b3426e78f217_isrForCrosstalkSources_855391_38.stderr parsl.dataflow.dflow INFO: Task 4009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dd4d7aa6b1b0f7bc6a36a016256c465 with result from task 4009 parsl.dataflow.dflow DEBUG: Task 4009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/36b9ce7f-487a-44ae-9773-dd525c7ec042_isrForCrosstalkSources_855430_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/36b9ce7f-487a-44ae-9773-dd525c7ec042_isrForCrosstalkSources_855430_16.stderr parsl.dataflow.dflow INFO: Task 4686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 095434fb08c9bd69a8391f4cc3940d1f with result from task 4686 parsl.dataflow.dflow DEBUG: Task 4686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3ebb8cf8-2a25-47e4-bc5f-b7b0a187b845_isrForCrosstalkSources_855379_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/3ebb8cf8-2a25-47e4-bc5f-b7b0a187b845_isrForCrosstalkSources_855379_21.stderr parsl.dataflow.dflow INFO: Task 4323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77dda5a931dfde8ac39837141cd17cf3 with result from task 4323 parsl.dataflow.dflow DEBUG: Task 4323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/e93b57d3-e999-46c0-8a71-55d9b1c3b7b8_isrForCrosstalkSources_855395_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/e93b57d3-e999-46c0-8a71-55d9b1c3b7b8_isrForCrosstalkSources_855395_15.stderr parsl.dataflow.dflow INFO: Task 4025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2107bc0cb85acb0e2ac0a9af7e8e2d6a with result from task 4025 parsl.dataflow.dflow DEBUG: Task 4025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/231b3e97-7c13-4278-a41a-29c6e9fe526e_isrForCrosstalkSources_855448_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/231b3e97-7c13-4278-a41a-29c6e9fe526e_isrForCrosstalkSources_855448_20.stderr parsl.dataflow.dflow INFO: Task 4385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 349eb536da0be0cd838a1cf63db16cc4 with result from task 4385 parsl.dataflow.dflow DEBUG: Task 4385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/aa0d2277-8f6a-437c-993f-65549dbed3c4_isrForCrosstalkSources_855379_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/aa0d2277-8f6a-437c-993f-65549dbed3c4_isrForCrosstalkSources_855379_53.stderr parsl.dataflow.dflow INFO: Task 3988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9727494c08b6bd7d877c8f9cb8ac44e5 with result from task 3988 parsl.dataflow.dflow DEBUG: Task 3988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/aedbb355-0b54-4ecd-a2c6-848845d0eeba_isrForCrosstalkSources_855415_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/aedbb355-0b54-4ecd-a2c6-848845d0eeba_isrForCrosstalkSources_855415_17.stderr parsl.dataflow.dflow INFO: Task 4476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 174063b25bc4489da5fbfd2f48244d7b with result from task 4476 parsl.dataflow.dflow DEBUG: Task 4476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0bb67001-ff7b-42fe-86b2-562599b6da3d_isrForCrosstalkSources_855444_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/0bb67001-ff7b-42fe-86b2-562599b6da3d_isrForCrosstalkSources_855444_6.stderr parsl.dataflow.dflow INFO: Task 4402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68aacf9eaa3f01762459d60def162f2 with result from task 4402 parsl.dataflow.dflow DEBUG: Task 4402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/392ba925-34cc-4ea5-8b7e-6187bfe47afd_isrForCrosstalkSources_855394_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/392ba925-34cc-4ea5-8b7e-6187bfe47afd_isrForCrosstalkSources_855394_34.stderr parsl.dataflow.dflow INFO: Task 4022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46088bbcf597ba9cae3ded39b41c14c9 with result from task 4022 parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:15 proc_lsst.multi:146 [INFO] found job 16964049 in provider astro proc_lsst.multi INFO: found job 16964049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow DEBUG: Task 4022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8d208a-3c01-44fc-abff-b213ff883e44_isrForCrosstalkSources_855404_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/0a8d208a-3c01-44fc-abff-b213ff883e44_isrForCrosstalkSources_855404_60.stderr parsl.dataflow.dflow INFO: Task 4400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7be6b078192efa117b0322ec9807d72f with result from task 4400 parsl.dataflow.dflow DEBUG: Task 4400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/db3f70bc-87dc-4306-b8c1-906b450157c0_isrForCrosstalkSources_855399_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/db3f70bc-87dc-4306-b8c1-906b450157c0_isrForCrosstalkSources_855399_1.stderr parsl.dataflow.dflow INFO: Task 4136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 173fcd1d4550c2daad66e0a20967f8da with result from task 4136 parsl.dataflow.dflow DEBUG: Task 4136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/70061ece-7f89-484a-b6ec-79ddc9a5a72a_isrForCrosstalkSources_855381_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/70061ece-7f89-484a-b6ec-79ddc9a5a72a_isrForCrosstalkSources_855381_10.stderr parsl.dataflow.dflow INFO: Task 3977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3b0ec620f02383effcf1c158046ff8 with result from task 3977 parsl.dataflow.dflow DEBUG: Task 3977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5391bd99-0992-489c-8dca-c6c92e48a00f_isrForCrosstalkSources_855419_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5391bd99-0992-489c-8dca-c6c92e48a00f_isrForCrosstalkSources_855419_6.stderr parsl.dataflow.dflow INFO: Task 4193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb802b5b13dc9a2f7ff694d0ef44bcc6 with result from task 4193 parsl.dataflow.dflow DEBUG: Task 4193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/663bcc08-6a7e-4c08-b926-74c23a7f8a6d_isrForCrosstalkSources_855412_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/663bcc08-6a7e-4c08-b926-74c23a7f8a6d_isrForCrosstalkSources_855412_10.stderr parsl.dataflow.dflow INFO: Task 4493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32fc916e6bd4bcb0b5bc21b926d31333 with result from task 4493 parsl.dataflow.dflow DEBUG: Task 4493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5e58c47e-a384-4e92-9c16-642d7444fdd5_isrForCrosstalkSources_855404_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/5e58c47e-a384-4e92-9c16-642d7444fdd5_isrForCrosstalkSources_855404_30.stderr parsl.dataflow.dflow INFO: Task 3902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce66b35be87a10fb5829e96fe197702e with result from task 3902 parsl.dataflow.dflow DEBUG: Task 3902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c4bb778f-5ae4-4684-819b-0b212dab2ec8_isrForCrosstalkSources_855385_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c4bb778f-5ae4-4684-819b-0b212dab2ec8_isrForCrosstalkSources_855385_58.stderr parsl.dataflow.dflow INFO: Task 4188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbea3d9c19e8df475dde85292da71bff with result from task 4188 parsl.dataflow.dflow DEBUG: Task 4188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c5581362-f072-42e1-89a5-485914f05372_isrForCrosstalkSources_855415_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/c5581362-f072-42e1-89a5-485914f05372_isrForCrosstalkSources_855415_37.stderr parsl.dataflow.dflow INFO: Task 4198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83a1942de1ece6be2d91ecbf1dbdb8fa with result from task 4198 parsl.dataflow.dflow DEBUG: Task 4198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d4087853-e3fd-4caf-bba2-f60bcc1a8537_isrForCrosstalkSources_855414_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d4087853-e3fd-4caf-bba2-f60bcc1a8537_isrForCrosstalkSources_855414_23.stderr parsl.dataflow.dflow INFO: Task 4007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e78ab654725b2e2bee189476de5e5823 with result from task 4007 parsl.dataflow.dflow DEBUG: Task 4007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e35b501d-6e54-4881-963b-12c99d41a179_isrForCrosstalkSources_855439_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/e35b501d-6e54-4881-963b-12c99d41a179_isrForCrosstalkSources_855439_9.stderr parsl.dataflow.dflow INFO: Task 4023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78c3bf2238724ba5f62afa755bff46f3 with result from task 4023 parsl.dataflow.dflow DEBUG: Task 4023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/68a54f7e-0d75-4e83-839a-61c9b30ff1f9_isrForCrosstalkSources_855392_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/68a54f7e-0d75-4e83-839a-61c9b30ff1f9_isrForCrosstalkSources_855392_1.stderr parsl.dataflow.dflow INFO: Task 3947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab34b0411034636ae5aa7ae4a079ae57 with result from task 3947 parsl.dataflow.dflow DEBUG: Task 3947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/64459758-5ec8-4135-85d9-0c2d4b14da79_isrForCrosstalkSources_855419_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/64459758-5ec8-4135-85d9-0c2d4b14da79_isrForCrosstalkSources_855419_62.stderr parsl.dataflow.dflow INFO: Task 4544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c37a2a616daa733087d5abc9d71336bb with result from task 4544 parsl.dataflow.dflow DEBUG: Task 4544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8440e62b-4d88-46d2-b353-c08f9e2a3d2b_isrForCrosstalkSources_855404_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/8440e62b-4d88-46d2-b353-c08f9e2a3d2b_isrForCrosstalkSources_855404_26.stderr parsl.dataflow.dflow INFO: Task 4027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb8d64a9e8bfdc161c9e2dd894afdb6 with result from task 4027 parsl.dataflow.dflow DEBUG: Task 4027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eadc066-99d3-463f-802f-e217c2513453_isrForCrosstalkSources_855434_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/3eadc066-99d3-463f-802f-e217c2513453_isrForCrosstalkSources_855434_32.stderr parsl.dataflow.dflow INFO: Task 4526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b62adf1f99ff6e74217e6c0985ac94a with result from task 4526 parsl.dataflow.dflow DEBUG: Task 4526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/72836117-ccac-4bca-80e6-9f3b222894f0_isrForCrosstalkSources_855404_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/72836117-ccac-4bca-80e6-9f3b222894f0_isrForCrosstalkSources_855404_42.stderr parsl.dataflow.dflow INFO: Task 4123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90c497de1c8b2e1469a55a03ed23ee6d with result from task 4123 parsl.dataflow.dflow DEBUG: Task 4123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/20c68603-d8c2-4d79-ad0a-31b70c377d50_isrForCrosstalkSources_855405_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/20c68603-d8c2-4d79-ad0a-31b70c377d50_isrForCrosstalkSources_855405_47.stderr parsl.dataflow.dflow INFO: Task 4008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 489c12e23402f44ce6111bf805510cca with result from task 4008 parsl.dataflow.dflow DEBUG: Task 4008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1d0a8981-5492-40ab-99a0-84bec853605e_isrForCrosstalkSources_855441_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/1d0a8981-5492-40ab-99a0-84bec853605e_isrForCrosstalkSources_855441_28.stderr parsl.dataflow.dflow INFO: Task 4279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af45f5b882446ee8a0972c3991f09a15 with result from task 4279 parsl.dataflow.dflow DEBUG: Task 4279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/396af17f-6a54-4a30-ae02-d38e601a9b00_isrForCrosstalkSources_855386_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/396af17f-6a54-4a30-ae02-d38e601a9b00_isrForCrosstalkSources_855386_7.stderr parsl.dataflow.dflow INFO: Task 4087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5946d50809f48c1dc019d12bd1aa718e with result from task 4087 parsl.dataflow.dflow DEBUG: Task 4087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/45c2e4ca-0de0-4fd3-9149-f5b784f66962_isrForCrosstalkSources_855426_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/45c2e4ca-0de0-4fd3-9149-f5b784f66962_isrForCrosstalkSources_855426_8.stderr parsl.dataflow.dflow INFO: Task 4651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87105f69ed65f92017f25db392b26b3e with result from task 4651 parsl.dataflow.dflow DEBUG: Task 4651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ae37a3a5-583b-462d-b576-d04e345f4532_isrForCrosstalkSources_855440_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/ae37a3a5-583b-462d-b576-d04e345f4532_isrForCrosstalkSources_855440_25.stderr parsl.dataflow.dflow INFO: Task 4144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0120721f488e93f8d998d5cc8f527b8 with result from task 4144 parsl.dataflow.dflow DEBUG: Task 4144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d4a8232d-2c9b-4913-b895-a1e7d07e20ee_isrForCrosstalkSources_855384_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/d4a8232d-2c9b-4913-b895-a1e7d07e20ee_isrForCrosstalkSources_855384_5.stderr parsl.dataflow.dflow INFO: Task 3900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57908a1ceed663b6372a9f893e95ad7f with result from task 3900 parsl.dataflow.dflow DEBUG: Task 3900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a90d2da6-38f1-48c7-bb7d-62679f831cb2_isrForCrosstalkSources_855455_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a90d2da6-38f1-48c7-bb7d-62679f831cb2_isrForCrosstalkSources_855455_12.stderr parsl.dataflow.dflow INFO: Task 3945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a3b77b862350f35c00fe926cc9049e2 with result from task 3945 parsl.dataflow.dflow DEBUG: Task 3945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/5289e483-47bd-415f-a6e6-dbda71e45928_isrForCrosstalkSources_855411_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/5289e483-47bd-415f-a6e6-dbda71e45928_isrForCrosstalkSources_855411_57.stderr parsl.dataflow.dflow INFO: Task 4213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d88c8ed67bf00317780233ef988709be with result from task 4213 parsl.dataflow.dflow DEBUG: Task 4213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bf13f712-c331-4580-a0db-1cc18b91bf0f_isrForCrosstalkSources_855442_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/bf13f712-c331-4580-a0db-1cc18b91bf0f_isrForCrosstalkSources_855442_29.stderr parsl.dataflow.dflow INFO: Task 4034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210f4becc088b64e97ba8085d564aa67 with result from task 4034 parsl.dataflow.dflow DEBUG: Task 4034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/14f2cad6-bff2-4adb-a3af-43daec8d0fc0_isrForCrosstalkSources_855439_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/14f2cad6-bff2-4adb-a3af-43daec8d0fc0_isrForCrosstalkSources_855439_56.stderr parsl.dataflow.dflow INFO: Task 3898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40c7027ab451adefcb0e04accfe5c737 with result from task 3898 parsl.dataflow.dflow DEBUG: Task 3898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/79b20288-bac2-42fc-888f-4bed5fc9cb0b_isrForCrosstalkSources_855458_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/79b20288-bac2-42fc-888f-4bed5fc9cb0b_isrForCrosstalkSources_855458_11.stderr parsl.dataflow.dflow INFO: Task 4278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76141cf2ad62fd3077fd12120f5e572 with result from task 4278 parsl.dataflow.dflow DEBUG: Task 4278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0a35b54a-5793-4155-824d-ea948c2261d4_isrForCrosstalkSources_855456_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/0a35b54a-5793-4155-824d-ea948c2261d4_isrForCrosstalkSources_855456_30.stderr parsl.dataflow.dflow INFO: Task 4219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e003542f513477a92b90061c2e9fbfb4 with result from task 4219 parsl.dataflow.dflow DEBUG: Task 4219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/369c815d-5537-494f-9736-3ab320aaa0d1_isrForCrosstalkSources_855449_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/369c815d-5537-494f-9736-3ab320aaa0d1_isrForCrosstalkSources_855449_35.stderr parsl.dataflow.dflow INFO: Task 3923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9264bad1c1baffb99b406f586582cffa with result from task 3923 parsl.dataflow.dflow DEBUG: Task 3923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/3696c9b7-c0e9-4e02-ba55-7777fa2e73a9_isrForCrosstalkSources_855410_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/3696c9b7-c0e9-4e02-ba55-7777fa2e73a9_isrForCrosstalkSources_855410_60.stderr parsl.dataflow.dflow INFO: Task 4050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff31f1e834fe8b037086a233537ad4a with result from task 4050 parsl.dataflow.dflow DEBUG: Task 4050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/def584f9-868d-4895-b674-cfcd0c9d2222_isrForCrosstalkSources_855399_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/def584f9-868d-4895-b674-cfcd0c9d2222_isrForCrosstalkSources_855399_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:16 proc_lsst.multi:146 [INFO] found job 16964050 in provider astro proc_lsst.multi INFO: found job 16964050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d78580b6a9d8c5cd79e004d78e683f63 with result from task 4035 parsl.dataflow.dflow DEBUG: Task 4035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/41f8ffb7-53fe-4e44-b678-cc172325746f_isrForCrosstalkSources_855404_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/41f8ffb7-53fe-4e44-b678-cc172325746f_isrForCrosstalkSources_855404_57.stderr parsl.dataflow.dflow INFO: Task 3904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 906ae49d0c16b99bd9d42e236b49edc3 with result from task 3904 parsl.dataflow.dflow DEBUG: Task 3904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fbea2a0d-7d59-4e99-98ae-665dff156e3d_isrForCrosstalkSources_855412_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/fbea2a0d-7d59-4e99-98ae-665dff156e3d_isrForCrosstalkSources_855412_37.stderr parsl.dataflow.dflow INFO: Task 3993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ede12d30acb1917e9f65a8f20c14325 with result from task 3993 parsl.dataflow.dflow DEBUG: Task 3993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1668ffa8-dbba-449d-865e-e95ce7814081_isrForCrosstalkSources_855450_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/1668ffa8-dbba-449d-865e-e95ce7814081_isrForCrosstalkSources_855450_52.stderr parsl.dataflow.dflow INFO: Task 4218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d953c3165e7b50f2ef242ae3755ea879 with result from task 4218 parsl.dataflow.dflow DEBUG: Task 4218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3eb043b7-890a-405c-985f-8a68641f6862_isrForCrosstalkSources_855448_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/3eb043b7-890a-405c-985f-8a68641f6862_isrForCrosstalkSources_855448_1.stderr parsl.dataflow.dflow INFO: Task 3889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01f9adda23399c05398bd2c4eb453552 with result from task 3889 parsl.dataflow.dflow DEBUG: Task 3889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3018ff7f-951d-4219-96a1-9ebca82b5200_isrForCrosstalkSources_855427_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/3018ff7f-951d-4219-96a1-9ebca82b5200_isrForCrosstalkSources_855427_22.stderr parsl.dataflow.dflow INFO: Task 3976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b3a09db1dc2b48e05ce677b6c1c63f with result from task 3976 parsl.dataflow.dflow DEBUG: Task 3976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e729e89a-ab47-426f-a879-d857fe595146_isrForCrosstalkSources_855458_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/e729e89a-ab47-426f-a879-d857fe595146_isrForCrosstalkSources_855458_17.stderr parsl.dataflow.dflow INFO: Task 3628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd4d396e6fed55ea6b4ace4e42ebb230 with result from task 3628 parsl.dataflow.dflow DEBUG: Task 3628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f96a2162-9319-4e8f-ad6e-58435e03679f_isrForCrosstalkSources_855396_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f96a2162-9319-4e8f-ad6e-58435e03679f_isrForCrosstalkSources_855396_53.stderr parsl.dataflow.dflow INFO: Task 3894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c5b5d023a2af1ba925158fe500b9f06 with result from task 3894 parsl.dataflow.dflow DEBUG: Task 3894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68f5efea-c1e6-4555-ab5b-ae610ec8b837_isrForCrosstalkSources_855399_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/68f5efea-c1e6-4555-ab5b-ae610ec8b837_isrForCrosstalkSources_855399_33.stderr parsl.dataflow.dflow INFO: Task 4277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c42dea64ebc2f9cf00dd649db3abb005 with result from task 4277 parsl.dataflow.dflow DEBUG: Task 4277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/50784c65-be87-47eb-96af-56624b822d11_isrForCrosstalkSources_855414_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/50784c65-be87-47eb-96af-56624b822d11_isrForCrosstalkSources_855414_44.stderr parsl.dataflow.dflow INFO: Task 3967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe8780798b0c1e13482e0bdfd84f3005 with result from task 3967 parsl.dataflow.dflow DEBUG: Task 3967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/740bda14-1af6-417d-9acf-e7cd23b8d8d3_isrForCrosstalkSources_855432_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/740bda14-1af6-417d-9acf-e7cd23b8d8d3_isrForCrosstalkSources_855432_38.stderr parsl.dataflow.dflow INFO: Task 4249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40399a961d03d26c9862a04c4c0faf05 with result from task 4249 parsl.dataflow.dflow DEBUG: Task 4249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2604275-4513-41b1-b236-0f8ac4965e43_isrForCrosstalkSources_855385_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/c2604275-4513-41b1-b236-0f8ac4965e43_isrForCrosstalkSources_855385_57.stderr parsl.dataflow.dflow INFO: Task 3881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ede5b1c27536e4bb5c5880fe126d1ed with result from task 3881 parsl.dataflow.dflow DEBUG: Task 3881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/5224d7b6-7970-4221-a67c-29b9acda5192_isrForCrosstalkSources_855420_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/5224d7b6-7970-4221-a67c-29b9acda5192_isrForCrosstalkSources_855420_34.stderr parsl.dataflow.dflow INFO: Task 4245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 827e69b61b20462734252eaf9024cbc8 with result from task 4245 parsl.dataflow.dflow DEBUG: Task 4245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d66aef15-cce8-4c5f-a80c-c9b8c8e308af_isrForCrosstalkSources_855414_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/d66aef15-cce8-4c5f-a80c-c9b8c8e308af_isrForCrosstalkSources_855414_43.stderr parsl.dataflow.dflow INFO: Task 3761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f23e2d018f29b1ec361f0d5086392e60 with result from task 3761 parsl.dataflow.dflow DEBUG: Task 3761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5038f7bc-b53d-4687-995b-f63c6661aae3_isrForCrosstalkSources_855439_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5038f7bc-b53d-4687-995b-f63c6661aae3_isrForCrosstalkSources_855439_54.stderr parsl.dataflow.dflow INFO: Task 3951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86e76907f39b2717e6dba53cceb4b0a5 with result from task 3951 parsl.dataflow.dflow DEBUG: Task 3951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f2ad1495-1740-4bf0-8d16-710de03704a4_isrForCrosstalkSources_855384_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f2ad1495-1740-4bf0-8d16-710de03704a4_isrForCrosstalkSources_855384_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:16 proc_lsst.multi:146 [INFO] found job 16964051 in provider astro proc_lsst.multi INFO: found job 16964051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 3931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f518dd73a524ce6d2b7d524093e82d99 with result from task 3931 parsl.dataflow.dflow DEBUG: Task 3931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/af878195-5c79-4506-b1be-0512e77d3b8b_isrForCrosstalkSources_855458_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/af878195-5c79-4506-b1be-0512e77d3b8b_isrForCrosstalkSources_855458_57.stderr parsl.dataflow.dflow INFO: Task 4112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b60ade5a50a20d507c8263dc0159bcd8 with result from task 4112 parsl.dataflow.dflow DEBUG: Task 4112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e7b33a49-3281-4ec2-b0e3-4b9b5334f290_isrForCrosstalkSources_855443_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/e7b33a49-3281-4ec2-b0e3-4b9b5334f290_isrForCrosstalkSources_855443_50.stderr parsl.dataflow.dflow INFO: Task 4726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 513448cd69ec377ca720db42c1e78010 with result from task 4726 parsl.dataflow.dflow DEBUG: Task 4726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f0f73b3-c3a3-4219-9859-f4018dd2e78a_isrForCrosstalkSources_855387_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/1f0f73b3-c3a3-4219-9859-f4018dd2e78a_isrForCrosstalkSources_855387_17.stderr parsl.dataflow.dflow INFO: Task 4687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afaf81e87a48332954d176d688bd7a82 with result from task 4687 parsl.dataflow.dflow DEBUG: Task 4687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e983a939-0e9a-4b2f-8938-3ce3ee439dc6_isrForCrosstalkSources_855392_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/e983a939-0e9a-4b2f-8938-3ce3ee439dc6_isrForCrosstalkSources_855392_20.stderr parsl.dataflow.dflow INFO: Task 4696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7591941f4130ee554637e9a958c78e69 with result from task 4696 parsl.dataflow.dflow DEBUG: Task 4696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/241b26a0-31bf-4f37-bedf-7c5e2d857ddf_isrForCrosstalkSources_855428_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/241b26a0-31bf-4f37-bedf-7c5e2d857ddf_isrForCrosstalkSources_855428_42.stderr parsl.dataflow.dflow INFO: Task 4628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72334edd11872489336b903b4fb29f05 with result from task 4628 parsl.dataflow.dflow DEBUG: Task 4628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a20fa69e-5cfb-4b8f-9565-2ad4861e19b3_isrForCrosstalkSources_855385_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/a20fa69e-5cfb-4b8f-9565-2ad4861e19b3_isrForCrosstalkSources_855385_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:16 proc_lsst.multi:146 [INFO] found job 16964052 in provider astro proc_lsst.multi INFO: found job 16964052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aa0f4f4389b08d2486fd38ee59ecaf0 with result from task 4722 parsl.dataflow.dflow DEBUG: Task 4722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8848a791-b2b5-489c-b0b8-76c0b62db33a_isrForCrosstalkSources_855419_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/8848a791-b2b5-489c-b0b8-76c0b62db33a_isrForCrosstalkSources_855419_8.stderr parsl.dataflow.dflow INFO: Task 4417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 157650aac1da472a188f35ed1acba636 with result from task 4417 parsl.dataflow.dflow DEBUG: Task 4417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b4127af3-f4b9-406a-803f-ed35a360ed06_isrForCrosstalkSources_855451_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/b4127af3-f4b9-406a-803f-ed35a360ed06_isrForCrosstalkSources_855451_34.stderr parsl.dataflow.dflow INFO: Task 4629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 851a09ff74b36d32f826451a2f2b1e32 with result from task 4629 parsl.dataflow.dflow DEBUG: Task 4629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d40a58-ca35-4ebc-8b00-a3274f998057_isrForCrosstalkSources_855448_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/94d40a58-ca35-4ebc-8b00-a3274f998057_isrForCrosstalkSources_855448_41.stderr parsl.dataflow.dflow INFO: Task 4665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb3c36644a90ec66195aa82339cb1ce with result from task 4665 parsl.dataflow.dflow DEBUG: Task 4665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0fdeae7d-ce3d-4a99-9bda-8b8c90e8224c_isrForCrosstalkSources_855407_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/0fdeae7d-ce3d-4a99-9bda-8b8c90e8224c_isrForCrosstalkSources_855407_22.stderr parsl.dataflow.dflow INFO: Task 4518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6965d28e59a1e780dc33abe2693c0df6 with result from task 4518 parsl.dataflow.dflow DEBUG: Task 4518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5d0c7eaa-a236-4c1d-855f-d77470889218_isrForCrosstalkSources_855381_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/5d0c7eaa-a236-4c1d-855f-d77470889218_isrForCrosstalkSources_855381_28.stderr parsl.dataflow.dflow INFO: Task 4719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6749b33d49e166fdec8e87dd64b17c7e with result from task 4719 parsl.dataflow.dflow DEBUG: Task 4719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ffba4a8e-a2c3-4d99-a407-ea00caab0b1f_isrForCrosstalkSources_855416_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/ffba4a8e-a2c3-4d99-a407-ea00caab0b1f_isrForCrosstalkSources_855416_26.stderr parsl.dataflow.dflow INFO: Task 4470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb96732c5f3c4d79c0ae85696724f8d1 with result from task 4470 parsl.dataflow.dflow DEBUG: Task 4470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/01bf9351-9e61-4559-8d42-6fe20690474f_isrForCrosstalkSources_855427_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/01bf9351-9e61-4559-8d42-6fe20690474f_isrForCrosstalkSources_855427_34.stderr parsl.dataflow.dflow INFO: Task 4507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf1918bbdcaa5e0e324dcb99704d53ca with result from task 4507 parsl.dataflow.dflow DEBUG: Task 4507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/179d0d87-ad03-4995-a241-0d4b99a52631_isrForCrosstalkSources_855410_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/179d0d87-ad03-4995-a241-0d4b99a52631_isrForCrosstalkSources_855410_35.stderr parsl.dataflow.dflow INFO: Task 4666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e36c3a2ee6767e5c81c78dc782411b80 with result from task 4666 parsl.dataflow.dflow DEBUG: Task 4666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/da224c52-22bc-4ad5-9efb-88c836e82046_isrForCrosstalkSources_855444_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/da224c52-22bc-4ad5-9efb-88c836e82046_isrForCrosstalkSources_855444_21.stderr parsl.dataflow.dflow INFO: Task 4529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 561e955efafb6e24d66fc9e3413dd0fd with result from task 4529 parsl.dataflow.dflow DEBUG: Task 4529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08707dd7-ee1e-4e6d-8247-50a0baf0fffc_isrForCrosstalkSources_855437_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/08707dd7-ee1e-4e6d-8247-50a0baf0fffc_isrForCrosstalkSources_855437_35.stderr parsl.dataflow.dflow INFO: Task 4733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6b89012b40dc5bbfd422e355fc99d15 with result from task 4733 parsl.dataflow.dflow DEBUG: Task 4733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/71281858-46c9-4f4a-a7ad-5b599c44fe0e_isrForCrosstalkSources_855449_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/71281858-46c9-4f4a-a7ad-5b599c44fe0e_isrForCrosstalkSources_855449_2.stderr parsl.dataflow.dflow INFO: Task 3949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dfe85b5221ef7a41d7f7401450ccffa with result from task 3949 parsl.dataflow.dflow DEBUG: Task 3949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a814d647-4bb8-4abd-98b4-7dbee10e0e38_isrForCrosstalkSources_855448_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a814d647-4bb8-4abd-98b4-7dbee10e0e38_isrForCrosstalkSources_855448_48.stderr parsl.dataflow.dflow INFO: Task 4624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5773373abdef39125401192637742c8f with result from task 4624 parsl.dataflow.dflow DEBUG: Task 4624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f5a3b77a-168e-4a5c-bfaf-dccb4b5e230e_isrForCrosstalkSources_855419_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/f5a3b77a-168e-4a5c-bfaf-dccb4b5e230e_isrForCrosstalkSources_855419_43.stderr parsl.dataflow.dflow INFO: Task 4675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10739b68fffa8a97a43b1ee80dd902a5 with result from task 4675 parsl.dataflow.dflow DEBUG: Task 4675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/05feaf69-a778-40e2-90ef-abcfb13f7046_isrForCrosstalkSources_855441_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/05feaf69-a778-40e2-90ef-abcfb13f7046_isrForCrosstalkSources_855441_33.stderr parsl.dataflow.dflow INFO: Task 4707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d38ea6953937860df991baa162b90f9 with result from task 4707 parsl.dataflow.dflow DEBUG: Task 4707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/99dcd15e-00ea-46b2-b55a-0f1154404228_isrForCrosstalkSources_855380_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/99dcd15e-00ea-46b2-b55a-0f1154404228_isrForCrosstalkSources_855380_27.stderr parsl.dataflow.dflow INFO: Task 4410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2eb32a486a95c4a3b801a18bfeaf99e with result from task 4410 parsl.dataflow.dflow DEBUG: Task 4410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2bbfffd9-4733-4c15-9097-67b14fc20fb2_isrForCrosstalkSources_855396_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/2bbfffd9-4733-4c15-9097-67b14fc20fb2_isrForCrosstalkSources_855396_38.stderr parsl.dataflow.dflow INFO: Task 4557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 590799232c51e858cf34ec25b505fbb4 with result from task 4557 parsl.dataflow.dflow DEBUG: Task 4557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/379573f2-1734-43a2-88c9-286e13715725_isrForCrosstalkSources_855424_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/379573f2-1734-43a2-88c9-286e13715725_isrForCrosstalkSources_855424_16.stderr parsl.dataflow.dflow INFO: Task 4618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 756becf97054e186a63ba338ab9e03d6 with result from task 4618 parsl.dataflow.dflow DEBUG: Task 4618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/338482d3-5572-431e-a120-690be1155d19_isrForCrosstalkSources_855413_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/338482d3-5572-431e-a120-690be1155d19_isrForCrosstalkSources_855413_11.stderr parsl.dataflow.dflow INFO: Task 4554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94bc628a431cbac3807f2e3810470791 with result from task 4554 parsl.dataflow.dflow DEBUG: Task 4554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0277f124-b1af-434f-8296-22dea8ef59b2_isrForCrosstalkSources_855424_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0277f124-b1af-434f-8296-22dea8ef59b2_isrForCrosstalkSources_855424_47.stderr parsl.dataflow.dflow INFO: Task 4616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbdae70e44ee0a0950070932399b3d6f with result from task 4616 parsl.dataflow.dflow DEBUG: Task 4616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67687c00-209b-43e5-af49-847c4f4ed688_isrForCrosstalkSources_855382_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/67687c00-209b-43e5-af49-847c4f4ed688_isrForCrosstalkSources_855382_59.stderr parsl.dataflow.dflow INFO: Task 4457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8e2b400311ea0b8ce5e3cf0a0f031c with result from task 4457 parsl.dataflow.dflow DEBUG: Task 4457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e3faf19-b4c5-432a-8788-8f38fe27d848_isrForCrosstalkSources_855429_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/1e3faf19-b4c5-432a-8788-8f38fe27d848_isrForCrosstalkSources_855429_8.stderr parsl.dataflow.dflow INFO: Task 4617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 572dbc9f5ae840a0cb20a698b92ddc1e with result from task 4617 parsl.dataflow.dflow DEBUG: Task 4617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0205d0b1-0773-484c-8fb7-245438dfb393_isrForCrosstalkSources_855441_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/0205d0b1-0773-484c-8fb7-245438dfb393_isrForCrosstalkSources_855441_55.stderr parsl.dataflow.dflow INFO: Task 4510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91ad367b51ce750c1d94599baf0769a with result from task 4510 parsl.dataflow.dflow DEBUG: Task 4510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/348a20a7-d472-417a-b962-2a95dc0c6718_isrForCrosstalkSources_855429_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/348a20a7-d472-417a-b962-2a95dc0c6718_isrForCrosstalkSources_855429_13.stderr parsl.dataflow.dflow INFO: Task 4427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe320c5d31c804e76530820a6c10a6f with result from task 4427 parsl.dataflow.dflow DEBUG: Task 4427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6a8c3c2c-a8b3-44c9-9d2e-4ce98b12ca60_isrForCrosstalkSources_855456_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/6a8c3c2c-a8b3-44c9-9d2e-4ce98b12ca60_isrForCrosstalkSources_855456_49.stderr parsl.dataflow.dflow INFO: Task 4508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72274e0707f8d0125e206c881c04c1ca with result from task 4508 parsl.dataflow.dflow DEBUG: Task 4508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/14602b86-f19f-47ee-824f-a91ddab2b374_isrForCrosstalkSources_855413_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/14602b86-f19f-47ee-824f-a91ddab2b374_isrForCrosstalkSources_855413_6.stderr parsl.dataflow.dflow INFO: Task 4540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57987db0a383302fcf6d20b58dcedd16 with result from task 4540 parsl.dataflow.dflow DEBUG: Task 4540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/49adf696-6f05-4fac-a402-54428a1c762a_isrForCrosstalkSources_855385_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/49adf696-6f05-4fac-a402-54428a1c762a_isrForCrosstalkSources_855385_32.stderr parsl.dataflow.dflow INFO: Task 4420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 625aecfbbafe6de46319d975291a6c4d with result from task 4420 parsl.dataflow.dflow DEBUG: Task 4420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/cdc3b09a-2c1a-49d8-bf6b-8b5134b921a2_isrForCrosstalkSources_855451_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/cdc3b09a-2c1a-49d8-bf6b-8b5134b921a2_isrForCrosstalkSources_855451_19.stderr parsl.dataflow.dflow INFO: Task 4758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f017becdd21d5be72e6ebf0aca3d21d2 with result from task 4758 parsl.dataflow.dflow DEBUG: Task 4758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/31bbaf32-8e2e-4757-9ced-f89713e33e87_isrForCrosstalkSources_855394_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/31bbaf32-8e2e-4757-9ced-f89713e33e87_isrForCrosstalkSources_855394_10.stderr parsl.dataflow.dflow INFO: Task 4677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c164d70eb331a162b792e5e05d8b499 with result from task 4677 parsl.dataflow.dflow DEBUG: Task 4677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f9601865-ee64-41a9-a6d5-05aaf00c69d7_isrForCrosstalkSources_855437_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/f9601865-ee64-41a9-a6d5-05aaf00c69d7_isrForCrosstalkSources_855437_42.stderr parsl.dataflow.dflow INFO: Task 4452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eb6aede96b4462d5ddc18c45b7d95ea with result from task 4452 parsl.dataflow.dflow DEBUG: Task 4452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/7e2883a7-4bbe-4932-90a4-4e830790a7b8_isrForCrosstalkSources_855430_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/7e2883a7-4bbe-4932-90a4-4e830790a7b8_isrForCrosstalkSources_855430_60.stderr parsl.dataflow.dflow INFO: Task 4334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ca2fc84cb85916278cd5a3daac2c637 with result from task 4334 parsl.dataflow.dflow DEBUG: Task 4334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11f54cca-4f8c-41cf-b818-df17022ddfa3_isrForCrosstalkSources_855426_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/11f54cca-4f8c-41cf-b818-df17022ddfa3_isrForCrosstalkSources_855426_31.stderr parsl.dataflow.dflow INFO: Task 4558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b97fc5b33b251c643c188b3ee52aae7 with result from task 4558 parsl.dataflow.dflow DEBUG: Task 4558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/bb90ec88-90bc-4fcf-972a-a5b13f2275a8_isrForCrosstalkSources_855418_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/bb90ec88-90bc-4fcf-972a-a5b13f2275a8_isrForCrosstalkSources_855418_11.stderr parsl.dataflow.dflow INFO: Task 4699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a4e5206519553b3f13498f605c299b8 with result from task 4699 parsl.dataflow.dflow DEBUG: Task 4699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/6b3aeefd-e049-49eb-9779-86b8ac5ea840_isrForCrosstalkSources_855414_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/6b3aeefd-e049-49eb-9779-86b8ac5ea840_isrForCrosstalkSources_855414_48.stderr parsl.dataflow.dflow INFO: Task 4370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f466e704b5650ae2d9a8de3c82a5e4 with result from task 4370 parsl.dataflow.dflow DEBUG: Task 4370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/28bad6e2-57fc-4502-8f27-725571624724_isrForCrosstalkSources_855421_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/28bad6e2-57fc-4502-8f27-725571624724_isrForCrosstalkSources_855421_41.stderr parsl.dataflow.dflow INFO: Task 4387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57b8de7bec4e8241f8d4e2172707bc5a with result from task 4387 parsl.dataflow.dflow DEBUG: Task 4387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/780a115a-890a-470b-849a-9b53fa0db762_isrForCrosstalkSources_855435_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/780a115a-890a-470b-849a-9b53fa0db762_isrForCrosstalkSources_855435_58.stderr parsl.dataflow.dflow INFO: Task 4700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3e8a589bfb7c6256753fd0677c7296f with result from task 4700 parsl.dataflow.dflow DEBUG: Task 4700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/be1784b3-a219-4766-adce-83ee325bbc56_isrForCrosstalkSources_855427_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/be1784b3-a219-4766-adce-83ee325bbc56_isrForCrosstalkSources_855427_56.stderr parsl.dataflow.dflow INFO: Task 4552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac0cdd6112f8001a737097aee0ba113c with result from task 4552 parsl.dataflow.dflow DEBUG: Task 4552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/37761017-bf10-4181-be73-646fb58367ff_isrForCrosstalkSources_855430_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/37761017-bf10-4181-be73-646fb58367ff_isrForCrosstalkSources_855430_24.stderr parsl.dataflow.dflow INFO: Task 4505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2546d159fea4188ed0b609a156252e6f with result from task 4505 parsl.dataflow.dflow DEBUG: Task 4505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6f65f311-e3a7-471e-9eb4-173f1bfe39c8_isrForCrosstalkSources_855458_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6f65f311-e3a7-471e-9eb4-173f1bfe39c8_isrForCrosstalkSources_855458_53.stderr parsl.dataflow.dflow INFO: Task 4369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c073412bac669d86e68de239ef07a7e with result from task 4369 parsl.dataflow.dflow DEBUG: Task 4369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23588e2d-f71e-43cc-9fd2-df5fc89f50c6_isrForCrosstalkSources_855396_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/23588e2d-f71e-43cc-9fd2-df5fc89f50c6_isrForCrosstalkSources_855396_42.stderr parsl.dataflow.dflow INFO: Task 4682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efde83b38e30fd4eb46be695ff21ae2 with result from task 4682 parsl.dataflow.dflow DEBUG: Task 4682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9df66c54-108d-411f-b6c6-26c2ad2d6e4a_isrForCrosstalkSources_855406_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/9df66c54-108d-411f-b6c6-26c2ad2d6e4a_isrForCrosstalkSources_855406_3.stderr parsl.dataflow.dflow INFO: Task 4715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52dc2c7e4e70507aaa4afa1beb31d24a with result from task 4715 parsl.dataflow.dflow DEBUG: Task 4715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cc600854-fb9a-41ea-b53b-fb6d96aa689c_isrForCrosstalkSources_855387_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/cc600854-fb9a-41ea-b53b-fb6d96aa689c_isrForCrosstalkSources_855387_36.stderr parsl.dataflow.dflow INFO: Task 4448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49241e116990e177c5e6d3bfb403478d with result from task 4448 parsl.dataflow.dflow DEBUG: Task 4448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bc9ec2c9-54c7-47eb-a645-3acc633ab1d0_isrForCrosstalkSources_855426_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/bc9ec2c9-54c7-47eb-a645-3acc633ab1d0_isrForCrosstalkSources_855426_45.stderr parsl.dataflow.dflow INFO: Task 4509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb067fa8cc359a6bc8d12e95faae19d with result from task 4509 parsl.dataflow.dflow DEBUG: Task 4509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a919adc5-8501-45d6-9ac8-f85bd3edc40f_isrForCrosstalkSources_855406_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/a919adc5-8501-45d6-9ac8-f85bd3edc40f_isrForCrosstalkSources_855406_9.stderr parsl.dataflow.dflow INFO: Task 4649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 843e3bc1ac9d07e8afd145c304793011 with result from task 4649 parsl.dataflow.dflow DEBUG: Task 4649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cb0f553a-f21a-43d0-b3c1-712134abd80e_isrForCrosstalkSources_855449_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/cb0f553a-f21a-43d0-b3c1-712134abd80e_isrForCrosstalkSources_855449_9.stderr parsl.dataflow.dflow INFO: Task 4478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2ca1459d719138872916d0b65f90b53 with result from task 4478 parsl.dataflow.dflow DEBUG: Task 4478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/162faa76-1de6-4ab5-acc1-5226d6571c85_isrForCrosstalkSources_855416_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/162faa76-1de6-4ab5-acc1-5226d6571c85_isrForCrosstalkSources_855416_7.stderr parsl.dataflow.dflow INFO: Task 4362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cddcab48801d96241b59789740fda6a5 with result from task 4362 parsl.dataflow.dflow DEBUG: Task 4362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/eece1abb-9683-44bd-aeea-aa2442abc8d9_isrForCrosstalkSources_855415_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/eece1abb-9683-44bd-aeea-aa2442abc8d9_isrForCrosstalkSources_855415_50.stderr parsl.dataflow.dflow INFO: Task 4367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ab13e8f3f0cbff3671ae748f862358b with result from task 4367 parsl.dataflow.dflow DEBUG: Task 4367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7a6daf8f-eed1-48de-b47f-be567fb22812_isrForCrosstalkSources_855393_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/7a6daf8f-eed1-48de-b47f-be567fb22812_isrForCrosstalkSources_855393_35.stderr parsl.dataflow.dflow INFO: Task 4684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1936792f40e6f35066a1f452f1f3fd with result from task 4684 parsl.dataflow.dflow DEBUG: Task 4684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/252c70bf-fb90-4147-9d04-5f35b051ed2d_isrForCrosstalkSources_855422_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/252c70bf-fb90-4147-9d04-5f35b051ed2d_isrForCrosstalkSources_855422_41.stderr parsl.dataflow.dflow INFO: Task 4520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d35fec0c0c6d0e2586a5bc098ddfe49 with result from task 4520 parsl.dataflow.dflow DEBUG: Task 4520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c8194955-1fca-4235-89cc-b87077ed6158_isrForCrosstalkSources_855419_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c8194955-1fca-4235-89cc-b87077ed6158_isrForCrosstalkSources_855419_32.stderr parsl.dataflow.dflow INFO: Task 4363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 390865dd61f4675f14193d00f83e635e with result from task 4363 parsl.dataflow.dflow DEBUG: Task 4363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/32d12c6e-64d8-45b3-8241-5acd6d32c16c_isrForCrosstalkSources_855445_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/32d12c6e-64d8-45b3-8241-5acd6d32c16c_isrForCrosstalkSources_855445_59.stderr parsl.dataflow.dflow INFO: Task 4359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 275701ac1c43b979930051671d9ef932 with result from task 4359 parsl.dataflow.dflow DEBUG: Task 4359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b5e28533-b472-4df0-b82f-29c193e87f2a_isrForCrosstalkSources_855388_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/b5e28533-b472-4df0-b82f-29c193e87f2a_isrForCrosstalkSources_855388_30.stderr parsl.dataflow.dflow INFO: Task 4514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33769302d5bcc569245bd30190029bf8 with result from task 4514 parsl.dataflow.dflow DEBUG: Task 4514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7812401e-16fb-495d-9ac4-e4d93629a803_isrForCrosstalkSources_855437_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7812401e-16fb-495d-9ac4-e4d93629a803_isrForCrosstalkSources_855437_29.stderr parsl.dataflow.dflow INFO: Task 4766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 086113f440b1229179421d78db0a3f2d with result from task 4766 parsl.dataflow.dflow DEBUG: Task 4766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/af74e4a4-452e-408e-be5c-e29a770bf4db_isrForCrosstalkSources_855416_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/af74e4a4-452e-408e-be5c-e29a770bf4db_isrForCrosstalkSources_855416_45.stderr parsl.dataflow.dflow INFO: Task 4511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ba46e55675eb14c2f499977fc3eaba0 with result from task 4511 parsl.dataflow.dflow DEBUG: Task 4511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4da6316c-ab9a-4bc6-9f26-4b9f13a594be_isrForCrosstalkSources_855404_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/4da6316c-ab9a-4bc6-9f26-4b9f13a594be_isrForCrosstalkSources_855404_27.stderr parsl.dataflow.dflow INFO: Task 4392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a51384d9a8b9e28043db1e36bdc2b1 with result from task 4392 parsl.dataflow.dflow DEBUG: Task 4392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce10fdcb-7614-4c64-8926-364ea6de0ca4_isrForCrosstalkSources_855445_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/ce10fdcb-7614-4c64-8926-364ea6de0ca4_isrForCrosstalkSources_855445_45.stderr parsl.dataflow.dflow INFO: Task 4398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f7526ed6f9a0e4387c34ab7b6148ce5 with result from task 4398 parsl.dataflow.dflow DEBUG: Task 4398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/20a7d027-d24c-4738-a788-0e93bc9fb656_isrForCrosstalkSources_855457_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/20a7d027-d24c-4738-a788-0e93bc9fb656_isrForCrosstalkSources_855457_62.stderr parsl.dataflow.dflow INFO: Task 4496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee79ec3ae716b860d7ac812cbc0a37f4 with result from task 4496 parsl.dataflow.dflow DEBUG: Task 4496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/e463a854-12d1-4c2b-ac2e-d702ef48110b_isrForCrosstalkSources_855389_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/e463a854-12d1-4c2b-ac2e-d702ef48110b_isrForCrosstalkSources_855389_25.stderr parsl.dataflow.dflow INFO: Task 4537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd388507ef713280dac66d51f89728a2 with result from task 4537 parsl.dataflow.dflow DEBUG: Task 4537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/af094aaf-2b9e-4e81-86de-012b7359ff63_isrForCrosstalkSources_855404_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/af094aaf-2b9e-4e81-86de-012b7359ff63_isrForCrosstalkSources_855404_23.stderr parsl.dataflow.dflow INFO: Task 4506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23fbc4b1691fcd5641b35f4ddbf53baf with result from task 4506 parsl.dataflow.dflow DEBUG: Task 4506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f94fe58d-00dc-411a-9661-5c9cc574e9ce_isrForCrosstalkSources_855381_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f94fe58d-00dc-411a-9661-5c9cc574e9ce_isrForCrosstalkSources_855381_4.stderr parsl.dataflow.dflow INFO: Task 4516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6719473c2d91b46ea7abf8e60dcbc912 with result from task 4516 parsl.dataflow.dflow DEBUG: Task 4516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84c5cb95-00bc-493c-babc-acd1fe86a908_isrForCrosstalkSources_855416_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/84c5cb95-00bc-493c-babc-acd1fe86a908_isrForCrosstalkSources_855416_50.stderr parsl.dataflow.dflow INFO: Task 4386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec2047f31863743997855a39730d8031 with result from task 4386 parsl.dataflow.dflow DEBUG: Task 4386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/96e1848b-76ac-4ed4-a6a9-0cabca0f790b_isrForCrosstalkSources_855394_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/96e1848b-76ac-4ed4-a6a9-0cabca0f790b_isrForCrosstalkSources_855394_23.stderr parsl.dataflow.dflow INFO: Task 4492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca06d4ecddebb639a21d95790c358a4c with result from task 4492 parsl.dataflow.dflow DEBUG: Task 4492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f31e918a-0ce4-497d-a6b2-3996cb4aa655_isrForCrosstalkSources_855390_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/f31e918a-0ce4-497d-a6b2-3996cb4aa655_isrForCrosstalkSources_855390_59.stderr parsl.dataflow.dflow INFO: Task 4523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8afd943f2589d0776b3b087a148c8ed1 with result from task 4523 parsl.dataflow.dflow DEBUG: Task 4523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8bdf65ab-c4b5-42ec-81d2-36b1aef0dbb2_isrForCrosstalkSources_855456_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/8bdf65ab-c4b5-42ec-81d2-36b1aef0dbb2_isrForCrosstalkSources_855456_53.stderr parsl.dataflow.dflow INFO: Task 4622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e58bf96ed78ccc27cdae1818512ced05 with result from task 4622 parsl.dataflow.dflow DEBUG: Task 4622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2427d7dd-ac23-4277-b6ca-ad97cbb3625c_isrForCrosstalkSources_855417_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/2427d7dd-ac23-4277-b6ca-ad97cbb3625c_isrForCrosstalkSources_855417_7.stderr parsl.dataflow.dflow INFO: Task 4337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcc439a10d044757c9c2cba77f8649a0 with result from task 4337 parsl.dataflow.dflow DEBUG: Task 4337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b2b5fabe-4c6a-469d-87a6-96e1bf44b926_isrForCrosstalkSources_855424_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/b2b5fabe-4c6a-469d-87a6-96e1bf44b926_isrForCrosstalkSources_855424_1.stderr parsl.dataflow.dflow INFO: Task 4627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b22ec8d2c2a90a69b92390f4d003f24f with result from task 4627 parsl.dataflow.dflow DEBUG: Task 4627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/180e1f38-9be3-4c1d-afe0-bfdb079b955f_isrForCrosstalkSources_855452_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/180e1f38-9be3-4c1d-afe0-bfdb079b955f_isrForCrosstalkSources_855452_37.stderr parsl.dataflow.dflow INFO: Task 4390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc5d6fcf88ba10a6bc80cc48a8500f1 with result from task 4390 parsl.dataflow.dflow DEBUG: Task 4390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4480618f-6af2-40ab-a3d7-c576baa2ea76_isrForCrosstalkSources_855401_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/4480618f-6af2-40ab-a3d7-c576baa2ea76_isrForCrosstalkSources_855401_62.stderr parsl.dataflow.dflow INFO: Task 4435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d5e5899a4b7d9385ab4711f57938377 with result from task 4435 parsl.dataflow.dflow DEBUG: Task 4435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d70a2e-149e-444b-9602-5168ed804e78_isrForCrosstalkSources_855426_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/96d70a2e-149e-444b-9602-5168ed804e78_isrForCrosstalkSources_855426_57.stderr parsl.dataflow.dflow INFO: Task 4648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce69af9c3306809a3acfb82c4a3fa1a1 with result from task 4648 parsl.dataflow.dflow DEBUG: Task 4648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8c13d640-3618-4fa8-a9af-a0fec33cf956_isrForCrosstalkSources_855447_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/8c13d640-3618-4fa8-a9af-a0fec33cf956_isrForCrosstalkSources_855447_40.stderr parsl.dataflow.dflow INFO: Task 4360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 649bd2506b37e6a0d6ee0abaf3b8a5b5 with result from task 4360 parsl.dataflow.dflow DEBUG: Task 4360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/fe706ac7-5dd1-479c-8544-46e4c090b1c0_isrForCrosstalkSources_855386_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/fe706ac7-5dd1-479c-8544-46e4c090b1c0_isrForCrosstalkSources_855386_53.stderr parsl.dataflow.dflow INFO: Task 4483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15fa53643c4c51c1df26c349eee8fd51 with result from task 4483 parsl.dataflow.dflow DEBUG: Task 4483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/70adbb9e-200b-4af9-b408-02fe3d229a6d_isrForCrosstalkSources_855439_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/70adbb9e-200b-4af9-b408-02fe3d229a6d_isrForCrosstalkSources_855439_57.stderr parsl.dataflow.dflow INFO: Task 4559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aceb33b041c78dc8f07f0906ecaff21 with result from task 4559 parsl.dataflow.dflow DEBUG: Task 4559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/030d3933-905e-4548-aa19-54cc2b1dca0d_isrForCrosstalkSources_855422_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/030d3933-905e-4548-aa19-54cc2b1dca0d_isrForCrosstalkSources_855422_39.stderr parsl.dataflow.dflow INFO: Task 4623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a33ef817168cc366348dbcfbb4711319 with result from task 4623 parsl.dataflow.dflow DEBUG: Task 4623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0a5f2a0e-7a87-4ebc-a204-4606ff1c8040_isrForCrosstalkSources_855385_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/0a5f2a0e-7a87-4ebc-a204-4606ff1c8040_isrForCrosstalkSources_855385_11.stderr parsl.dataflow.dflow INFO: Task 4431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1549330d0b20e12d600aeb4e3bbae55 with result from task 4431 parsl.dataflow.dflow DEBUG: Task 4431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a2fcbb1-964d-4f30-ac1a-1066b33f7747_isrForCrosstalkSources_855379_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/5a2fcbb1-964d-4f30-ac1a-1066b33f7747_isrForCrosstalkSources_855379_6.stderr parsl.dataflow.dflow INFO: Task 4633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08bba0a81540cde44183154e24325e79 with result from task 4633 parsl.dataflow.dflow DEBUG: Task 4633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e3bca7a3-d3eb-4763-938c-8ebaf1e9bb22_isrForCrosstalkSources_855420_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e3bca7a3-d3eb-4763-938c-8ebaf1e9bb22_isrForCrosstalkSources_855420_20.stderr parsl.dataflow.dflow INFO: Task 4374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6956d27926b8887ce206a5cf91eb77a with result from task 4374 parsl.dataflow.dflow DEBUG: Task 4374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/609fd9c9-974a-4b7a-b802-68f1f95c936f_isrForCrosstalkSources_855424_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/609fd9c9-974a-4b7a-b802-68f1f95c936f_isrForCrosstalkSources_855424_50.stderr parsl.dataflow.dflow INFO: Task 4551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c33a243665de0410216f0a3123fd88f6 with result from task 4551 parsl.dataflow.dflow DEBUG: Task 4551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/44d5fc74-52c2-4afb-9952-1a36222f1c2a_isrForCrosstalkSources_855438_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/44d5fc74-52c2-4afb-9952-1a36222f1c2a_isrForCrosstalkSources_855438_25.stderr parsl.dataflow.dflow INFO: Task 4749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6714796f0a3a4214903f9169769e629 with result from task 4749 parsl.dataflow.dflow DEBUG: Task 4749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e1c0b867-453c-414f-862d-6bc7e09eee42_isrForCrosstalkSources_855382_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/e1c0b867-453c-414f-862d-6bc7e09eee42_isrForCrosstalkSources_855382_58.stderr parsl.dataflow.dflow INFO: Task 4763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 880456f5a5f8da328f88d923489de5b6 with result from task 4763 parsl.dataflow.dflow DEBUG: Task 4763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/190a900f-d531-41c7-9c42-8a8c95676b2a_isrForCrosstalkSources_855409_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/190a900f-d531-41c7-9c42-8a8c95676b2a_isrForCrosstalkSources_855409_57.stderr parsl.dataflow.dflow INFO: Task 4467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305bc049731a3fd553d36b0b792a0863 with result from task 4467 parsl.dataflow.dflow DEBUG: Task 4467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/33eca51f-c423-4fc7-94c4-8e59192ff3a8_isrForCrosstalkSources_855400_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/33eca51f-c423-4fc7-94c4-8e59192ff3a8_isrForCrosstalkSources_855400_61.stderr parsl.dataflow.dflow INFO: Task 4528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8d0ff160453bc5566b7245d5e716c14 with result from task 4528 parsl.dataflow.dflow DEBUG: Task 4528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cc65b729-f3e0-4285-af6e-37f100185f09_isrForCrosstalkSources_855382_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/cc65b729-f3e0-4285-af6e-37f100185f09_isrForCrosstalkSources_855382_40.stderr parsl.dataflow.dflow INFO: Task 4630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 990738aa77db087b9639088d2da2164a with result from task 4630 parsl.dataflow.dflow DEBUG: Task 4630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a7eaac77-fa83-4774-a94b-f246c79cba97_isrForCrosstalkSources_855441_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a7eaac77-fa83-4774-a94b-f246c79cba97_isrForCrosstalkSources_855441_41.stderr parsl.dataflow.dflow INFO: Task 4759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 509780587eb74d1c07b88df572cba669 with result from task 4759 parsl.dataflow.dflow DEBUG: Task 4759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/21dae5d6-2ffc-4960-b0fe-f310dbaa2e4d_isrForCrosstalkSources_855407_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/21dae5d6-2ffc-4960-b0fe-f310dbaa2e4d_isrForCrosstalkSources_855407_48.stderr parsl.dataflow.dflow INFO: Task 4640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a50ebbc46be66966d43e2a5e5f0a232 with result from task 4640 parsl.dataflow.dflow DEBUG: Task 4640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d153609-df4e-48f0-b222-45336088aee0_isrForCrosstalkSources_855410_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/9d153609-df4e-48f0-b222-45336088aee0_isrForCrosstalkSources_855410_54.stderr parsl.dataflow.dflow INFO: Task 4465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25c7be26eb062e8124cd471d78d6266 with result from task 4465 parsl.dataflow.dflow DEBUG: Task 4465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fc4565d0-62d7-4ad1-8b26-5881202c1b04_isrForCrosstalkSources_855450_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fc4565d0-62d7-4ad1-8b26-5881202c1b04_isrForCrosstalkSources_855450_50.stderr parsl.dataflow.dflow INFO: Task 4553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46a3d3bb6b4dc2b828c3f371278dbaf8 with result from task 4553 parsl.dataflow.dflow DEBUG: Task 4553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bc5f7542-5d00-436d-966d-fa413087588d_isrForCrosstalkSources_855417_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/bc5f7542-5d00-436d-966d-fa413087588d_isrForCrosstalkSources_855417_33.stderr parsl.dataflow.dflow INFO: Task 4620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0440fa4c91c5a30df6dde7a17dabb2d with result from task 4620 parsl.dataflow.dflow DEBUG: Task 4620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee332a93-8319-4f0d-84f9-656c2bae7d78_isrForCrosstalkSources_855443_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/ee332a93-8319-4f0d-84f9-656c2bae7d78_isrForCrosstalkSources_855443_3.stderr parsl.dataflow.dflow INFO: Task 4737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c23fa846e1db051242930ebf3c0af11a with result from task 4737 parsl.dataflow.dflow DEBUG: Task 4737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/cc35d258-e342-493a-aed0-50b178452e89_isrForCrosstalkSources_855426_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/cc35d258-e342-493a-aed0-50b178452e89_isrForCrosstalkSources_855426_4.stderr parsl.dataflow.dflow INFO: Task 4515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cced7f4caf350474ff0a83e3c2e12182 with result from task 4515 parsl.dataflow.dflow DEBUG: Task 4515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0a6ccec8-80ca-45a9-96f7-ee22c1bfc5bf_isrForCrosstalkSources_855424_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/0a6ccec8-80ca-45a9-96f7-ee22c1bfc5bf_isrForCrosstalkSources_855424_24.stderr parsl.dataflow.dflow INFO: Task 4502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f71c83aab8be4df76ae7394a681999eb with result from task 4502 parsl.dataflow.dflow DEBUG: Task 4502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9222778-802d-4ee6-9ffb-bc914c3efb40_isrForCrosstalkSources_855449_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/f9222778-802d-4ee6-9ffb-bc914c3efb40_isrForCrosstalkSources_855449_5.stderr parsl.dataflow.dflow INFO: Task 4443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccc4ef244e3cb292990e0709acfbb971 with result from task 4443 parsl.dataflow.dflow DEBUG: Task 4443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/59ffcacf-5b23-4747-b2a0-f41a7dfc36ea_isrForCrosstalkSources_855458_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/59ffcacf-5b23-4747-b2a0-f41a7dfc36ea_isrForCrosstalkSources_855458_31.stderr parsl.dataflow.dflow INFO: Task 4634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec11524aabe0f0c076f0242ac94a9bd4 with result from task 4634 parsl.dataflow.dflow DEBUG: Task 4634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6bda1318-5235-4922-8d6c-e3d68d72ad49_isrForCrosstalkSources_855385_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/6bda1318-5235-4922-8d6c-e3d68d72ad49_isrForCrosstalkSources_855385_35.stderr parsl.dataflow.dflow INFO: Task 4426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63c4a81e5effd7f4e692b88fd2364acd with result from task 4426 parsl.dataflow.dflow DEBUG: Task 4426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1bd01b9a-96dd-46ac-bb93-a85bde693b5a_isrForCrosstalkSources_855459_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/1bd01b9a-96dd-46ac-bb93-a85bde693b5a_isrForCrosstalkSources_855459_35.stderr parsl.dataflow.dflow INFO: Task 4543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19de98207408c169bfd6c2460790099a with result from task 4543 parsl.dataflow.dflow DEBUG: Task 4543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/39667fa6-7e0b-44d8-8a39-42e072df4d09_isrForCrosstalkSources_855399_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/39667fa6-7e0b-44d8-8a39-42e072df4d09_isrForCrosstalkSources_855399_34.stderr parsl.dataflow.dflow INFO: Task 4425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9abe4f48b58329d30ca277f7965236aa with result from task 4425 parsl.dataflow.dflow DEBUG: Task 4425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/684e13f8-0370-4cea-8c9a-0737f0ba0955_isrForCrosstalkSources_855442_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/684e13f8-0370-4cea-8c9a-0737f0ba0955_isrForCrosstalkSources_855442_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:18 proc_lsst.multi:146 [INFO] found job 16964053 in provider astro proc_lsst.multi INFO: found job 16964053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82933277d4be4eb7b619a6800e3f3c65 with result from task 4545 parsl.dataflow.dflow DEBUG: Task 4545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/24bd0894-b463-4317-b8d8-dcd4bb2b5b30_isrForCrosstalkSources_855407_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/24bd0894-b463-4317-b8d8-dcd4bb2b5b30_isrForCrosstalkSources_855407_1.stderr parsl.dataflow.dflow INFO: Task 4717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc2607beb0df2558976cab77351ed5e with result from task 4717 parsl.dataflow.dflow DEBUG: Task 4717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/061c5f0c-4b92-4eb8-81ec-073f530c0263_isrForCrosstalkSources_855454_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/061c5f0c-4b92-4eb8-81ec-073f530c0263_isrForCrosstalkSources_855454_17.stderr parsl.dataflow.dflow INFO: Task 4718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8589f9bc2e153d25f3c3f90defe64fea with result from task 4718 parsl.dataflow.dflow DEBUG: Task 4718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4cfb2e70-fdf5-469b-9a15-1a4cefda433b_isrForCrosstalkSources_855415_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/4cfb2e70-fdf5-469b-9a15-1a4cefda433b_isrForCrosstalkSources_855415_15.stderr parsl.dataflow.dflow INFO: Task 4664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fd9043adfb24cfabe7717920de7c1a6 with result from task 4664 parsl.dataflow.dflow DEBUG: Task 4664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a1769384-4dd2-423b-94e5-1902a6bce515_isrForCrosstalkSources_855408_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/a1769384-4dd2-423b-94e5-1902a6bce515_isrForCrosstalkSources_855408_13.stderr parsl.dataflow.dflow INFO: Task 4406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd178c3673c569f0f3504e4b75eec7d3 with result from task 4406 parsl.dataflow.dflow DEBUG: Task 4406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0c91b94d-1bda-4abc-b501-00c8b03c9a6e_isrForCrosstalkSources_855389_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/0c91b94d-1bda-4abc-b501-00c8b03c9a6e_isrForCrosstalkSources_855389_49.stderr parsl.dataflow.dflow INFO: Task 4746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69910a116cc026d513deb96faf41d341 with result from task 4746 parsl.dataflow.dflow DEBUG: Task 4746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7b3b8cd8-fd4f-488d-bd6f-d426f756b448_isrForCrosstalkSources_855455_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/7b3b8cd8-fd4f-488d-bd6f-d426f756b448_isrForCrosstalkSources_855455_46.stderr parsl.dataflow.dflow INFO: Task 4776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db711e9f7e027f538c2b771c01add49f with result from task 4776 parsl.dataflow.dflow DEBUG: Task 4776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/eb0c52a0-7724-4bc1-b44f-ac1559c51b69_isrForCrosstalkSources_855426_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/eb0c52a0-7724-4bc1-b44f-ac1559c51b69_isrForCrosstalkSources_855426_19.stderr parsl.dataflow.dflow INFO: Task 4731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e301c03348a621ad6988ecad42dcd2ec with result from task 4731 parsl.dataflow.dflow DEBUG: Task 4731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2ec98e4e-2a68-4d50-834f-7a2cf96a8dee_isrForCrosstalkSources_855453_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/2ec98e4e-2a68-4d50-834f-7a2cf96a8dee_isrForCrosstalkSources_855453_35.stderr parsl.dataflow.dflow INFO: Task 4512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1af4ee64534c1e0466cab25647b172e with result from task 4512 parsl.dataflow.dflow DEBUG: Task 4512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2802d345-7937-41b4-9bab-6a103ac4e55a_isrForCrosstalkSources_855403_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/2802d345-7937-41b4-9bab-6a103ac4e55a_isrForCrosstalkSources_855403_42.stderr parsl.dataflow.dflow INFO: Task 4486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7af0934870566dc6f1f62cfe7ff6dd97 with result from task 4486 parsl.dataflow.dflow DEBUG: Task 4486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dd2fc06b-d15a-4081-b9ab-d5b924376e19_isrForCrosstalkSources_855447_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/dd2fc06b-d15a-4081-b9ab-d5b924376e19_isrForCrosstalkSources_855447_34.stderr parsl.dataflow.dflow INFO: Task 4800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cec6506f101e5f8b29f492272deb269 with result from task 4800 parsl.dataflow.dflow DEBUG: Task 4800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/36363978-ae72-4c81-9799-9567cb9e964b_isrForCrosstalkSources_855404_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/36363978-ae72-4c81-9799-9567cb9e964b_isrForCrosstalkSources_855404_11.stderr parsl.dataflow.dflow INFO: Task 4679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc5d4eaa377e27162cd7f94b6212f7dd with result from task 4679 parsl.dataflow.dflow DEBUG: Task 4679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/bf2d83a2-61af-4383-ac9c-fe2f00645b17_isrForCrosstalkSources_855414_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/bf2d83a2-61af-4383-ac9c-fe2f00645b17_isrForCrosstalkSources_855414_6.stderr parsl.dataflow.dflow INFO: Task 4273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70c4eb512ef245d0f69ed995c40041b9 with result from task 4273 parsl.dataflow.dflow DEBUG: Task 4273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a34cc138-b2e7-44b2-a77a-d6b4bd2446af_isrForCrosstalkSources_855394_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a34cc138-b2e7-44b2-a77a-d6b4bd2446af_isrForCrosstalkSources_855394_36.stderr parsl.dataflow.dflow INFO: Task 4671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a20b69142b1ebbe74af839c1c35c63b with result from task 4671 parsl.dataflow.dflow DEBUG: Task 4671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fac24a84-df82-473a-92c4-36082f90bd29_isrForCrosstalkSources_855418_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/fac24a84-df82-473a-92c4-36082f90bd29_isrForCrosstalkSources_855418_30.stderr parsl.dataflow.dflow INFO: Task 4680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8b99d1d0d338c5b5cbddc930be8741c with result from task 4680 parsl.dataflow.dflow DEBUG: Task 4680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7c386a6f-33fc-49ab-b50c-c9c7ddb929f1_isrForCrosstalkSources_855419_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/7c386a6f-33fc-49ab-b50c-c9c7ddb929f1_isrForCrosstalkSources_855419_12.stderr parsl.dataflow.dflow INFO: Task 4673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71fea54d348a3499914f74b2f8b09912 with result from task 4673 parsl.dataflow.dflow DEBUG: Task 4673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/4bb17698-05cf-43f0-9241-51e655f3d7b8_isrForCrosstalkSources_855414_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/4bb17698-05cf-43f0-9241-51e655f3d7b8_isrForCrosstalkSources_855414_62.stderr parsl.dataflow.dflow INFO: Task 4674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21dfe75dd009ce88f222ec4e06531059 with result from task 4674 parsl.dataflow.dflow DEBUG: Task 4674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/8d70b013-3289-4cc8-a630-cf0d960f90eb_isrForCrosstalkSources_855418_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/8d70b013-3289-4cc8-a630-cf0d960f90eb_isrForCrosstalkSources_855418_19.stderr parsl.dataflow.dflow INFO: Task 4479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37934079d3d45837df3ef0542f795c1f with result from task 4479 parsl.dataflow.dflow DEBUG: Task 4479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/427325b0-8d01-45d8-b5e6-0fb9867e4367_isrForCrosstalkSources_855389_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/427325b0-8d01-45d8-b5e6-0fb9867e4367_isrForCrosstalkSources_855389_48.stderr parsl.dataflow.dflow INFO: Task 4615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aaec4b2003f64eaf82672c6f84e55c5 with result from task 4615 parsl.dataflow.dflow DEBUG: Task 4615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/3183bc89-790e-4c26-b92f-017d11f29485_isrForCrosstalkSources_855399_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/3183bc89-790e-4c26-b92f-017d11f29485_isrForCrosstalkSources_855399_30.stderr parsl.dataflow.dflow INFO: Task 4329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e2318cb479d815cd3bbda5411f0a197 with result from task 4329 parsl.dataflow.dflow DEBUG: Task 4329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a8f2aef3-8a7e-46f9-9e7d-c5d07ddf3619_isrForCrosstalkSources_855450_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/a8f2aef3-8a7e-46f9-9e7d-c5d07ddf3619_isrForCrosstalkSources_855450_30.stderr parsl.dataflow.dflow INFO: Task 4535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa7eaa837ab486c7a9306e70204ee2e with result from task 4535 parsl.dataflow.dflow DEBUG: Task 4535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/668db0d8-3fe9-4f64-b11b-d2960f02783b_isrForCrosstalkSources_855399_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/668db0d8-3fe9-4f64-b11b-d2960f02783b_isrForCrosstalkSources_855399_40.stderr parsl.dataflow.dflow INFO: Task 4519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee2141935d1da50f05fe35fc029f0296 with result from task 4519 parsl.dataflow.dflow DEBUG: Task 4519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fa77dbad-ab6b-4026-9cc9-3f56d74a71b6_isrForCrosstalkSources_855440_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/fa77dbad-ab6b-4026-9cc9-3f56d74a71b6_isrForCrosstalkSources_855440_7.stderr parsl.dataflow.dflow INFO: Task 4542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a183ea301ad2e6cf7070cd9735bf2f44 with result from task 4542 parsl.dataflow.dflow DEBUG: Task 4542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e7e89990-288c-47c0-8f15-68057804b588_isrForCrosstalkSources_855420_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/e7e89990-288c-47c0-8f15-68057804b588_isrForCrosstalkSources_855420_35.stderr parsl.dataflow.dflow INFO: Task 4550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e6666793521a42b11f545f18b9a5e7d with result from task 4550 parsl.dataflow.dflow DEBUG: Task 4550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0826e4c9-4a6e-4a27-9a1c-9ea3371ba2d3_isrForCrosstalkSources_855429_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/0826e4c9-4a6e-4a27-9a1c-9ea3371ba2d3_isrForCrosstalkSources_855429_27.stderr parsl.dataflow.dflow INFO: Task 4816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca172671459280deedadcbaa795144ad with result from task 4816 parsl.dataflow.dflow DEBUG: Task 4816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e1d8431d-b5eb-424a-81c9-ac9f0cfb6839_isrForCrosstalkSources_855418_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/e1d8431d-b5eb-424a-81c9-ac9f0cfb6839_isrForCrosstalkSources_855418_56.stderr parsl.dataflow.dflow INFO: Task 4782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17b3fbd4df7d99e7715effbbe4ddcdc8 with result from task 4782 parsl.dataflow.dflow DEBUG: Task 4782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/660cca55-23e4-4498-91d0-983b941e550b_isrForCrosstalkSources_855446_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/660cca55-23e4-4498-91d0-983b941e550b_isrForCrosstalkSources_855446_28.stderr parsl.dataflow.dflow INFO: Task 4810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2c67c15507e96d115908fbe3b10b9e with result from task 4810 parsl.dataflow.dflow DEBUG: Task 4810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2d5a2dd8-c4a3-4555-8c17-ac8ccd474ff6_isrForCrosstalkSources_855434_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/2d5a2dd8-c4a3-4555-8c17-ac8ccd474ff6_isrForCrosstalkSources_855434_33.stderr parsl.dataflow.dflow INFO: Task 4459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b89c3a152d0d07ff065a287eedd2ab87 with result from task 4459 parsl.dataflow.dflow DEBUG: Task 4459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/8facd480-b791-4113-b0ec-0291825fe8e4_isrForCrosstalkSources_855437_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/8facd480-b791-4113-b0ec-0291825fe8e4_isrForCrosstalkSources_855437_55.stderr parsl.dataflow.dflow INFO: Task 4546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 341383a350c92269c52bf82b45ac6fcb with result from task 4546 parsl.dataflow.dflow DEBUG: Task 4546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1f0e8c8a-ad7b-41a1-941d-95072133e6e9_isrForCrosstalkSources_855452_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1f0e8c8a-ad7b-41a1-941d-95072133e6e9_isrForCrosstalkSources_855452_46.stderr parsl.dataflow.dflow INFO: Task 4764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee337b398482d6e3f3a13e9c0faf724b with result from task 4764 parsl.dataflow.dflow DEBUG: Task 4764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/829c0872-38a7-472f-a9bb-14983b603fd7_isrForCrosstalkSources_855412_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/829c0872-38a7-472f-a9bb-14983b603fd7_isrForCrosstalkSources_855412_22.stderr parsl.dataflow.dflow INFO: Task 4487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec33f7955f5642d940d860600b5b1488 with result from task 4487 parsl.dataflow.dflow DEBUG: Task 4487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b646b5c3-1777-45f5-b117-f2208b9b428a_isrForCrosstalkSources_855453_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/b646b5c3-1777-45f5-b117-f2208b9b428a_isrForCrosstalkSources_855453_42.stderr parsl.dataflow.dflow INFO: Task 4688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9ac0f2526934d6c612c352f8488c82 with result from task 4688 parsl.dataflow.dflow DEBUG: Task 4688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/df1f1c32-02ad-444b-90e1-bdfff70ad4f0_isrForCrosstalkSources_855442_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/df1f1c32-02ad-444b-90e1-bdfff70ad4f0_isrForCrosstalkSources_855442_5.stderr parsl.dataflow.dflow INFO: Task 4827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b5bbdbaee867783e411b5915529f86b with result from task 4827 parsl.dataflow.dflow DEBUG: Task 4827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c940364b-3037-4412-b7d1-441e72cb5a46_isrForCrosstalkSources_855452_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/c940364b-3037-4412-b7d1-441e72cb5a46_isrForCrosstalkSources_855452_54.stderr parsl.dataflow.dflow INFO: Task 4613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e610a29b01d93373eb8697fc406e5555 with result from task 4613 parsl.dataflow.dflow DEBUG: Task 4613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d206a2f5-4eb4-48f6-802f-8fe1bb812dd3_isrForCrosstalkSources_855459_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d206a2f5-4eb4-48f6-802f-8fe1bb812dd3_isrForCrosstalkSources_855459_29.stderr parsl.dataflow.dflow INFO: Task 4621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 957d0bfcdd1f0b4491c75ac90c75b320 with result from task 4621 parsl.dataflow.dflow DEBUG: Task 4621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d6ae350-ab05-4998-b357-95a5359d2943_isrForCrosstalkSources_855439_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/5d6ae350-ab05-4998-b357-95a5359d2943_isrForCrosstalkSources_855439_49.stderr parsl.dataflow.dflow INFO: Task 4491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c469a13ad601641de162d6713e6ecd23 with result from task 4491 parsl.dataflow.dflow DEBUG: Task 4491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/651a31e2-3d61-43b4-bf90-5402d59ed4bd_isrForCrosstalkSources_855436_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/651a31e2-3d61-43b4-bf90-5402d59ed4bd_isrForCrosstalkSources_855436_24.stderr parsl.dataflow.dflow INFO: Task 4488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb4368937044a6bb4dc166c852cd832 with result from task 4488 parsl.dataflow.dflow DEBUG: Task 4488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca178794-770c-4820-9cbf-2c84abbab139_isrForCrosstalkSources_855437_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/ca178794-770c-4820-9cbf-2c84abbab139_isrForCrosstalkSources_855437_9.stderr parsl.dataflow.dflow INFO: Task 4761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 237921cae171f846d16e2e2cbd5163c3 with result from task 4761 parsl.dataflow.dflow DEBUG: Task 4761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2614a38f-42fa-4f0b-926f-a36f8f9c80fd_isrForCrosstalkSources_855414_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2614a38f-42fa-4f0b-926f-a36f8f9c80fd_isrForCrosstalkSources_855414_19.stderr parsl.dataflow.dflow INFO: Task 4499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3168b8a10b3af2212953e4b0c02c17f with result from task 4499 parsl.dataflow.dflow DEBUG: Task 4499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5f866e33-691d-4075-8401-b1f4d92d6ee5_isrForCrosstalkSources_855425_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/5f866e33-691d-4075-8401-b1f4d92d6ee5_isrForCrosstalkSources_855425_45.stderr parsl.dataflow.dflow INFO: Task 4835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f49fea3c0dda14cc4fbb5afd816f6e21 with result from task 4835 parsl.dataflow.dflow DEBUG: Task 4835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/cf627fa5-ff18-4652-a719-63a5d9f7486b_isrForCrosstalkSources_855436_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/cf627fa5-ff18-4652-a719-63a5d9f7486b_isrForCrosstalkSources_855436_6.stderr parsl.dataflow.dflow INFO: Task 4636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e96482ee10b2cafde164768ab0fbd9c4 with result from task 4636 parsl.dataflow.dflow DEBUG: Task 4636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/e72966fd-a5cf-4376-befa-a9ec756f5a92_isrForCrosstalkSources_855427_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/e72966fd-a5cf-4376-befa-a9ec756f5a92_isrForCrosstalkSources_855427_30.stderr parsl.dataflow.dflow INFO: Task 4828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c044122cca16b7873aede63523d5625 with result from task 4828 parsl.dataflow.dflow DEBUG: Task 4828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a812ead1-b7e3-4e0a-aa1d-d64367628a9a_isrForCrosstalkSources_855439_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a812ead1-b7e3-4e0a-aa1d-d64367628a9a_isrForCrosstalkSources_855439_15.stderr parsl.dataflow.dflow INFO: Task 4829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dcb17d7d18681ba7b3dbdc5b3e33c2f with result from task 4829 parsl.dataflow.dflow DEBUG: Task 4829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c70c0653-6907-4bdc-aa3c-512a23d59c07_isrForCrosstalkSources_855394_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/c70c0653-6907-4bdc-aa3c-512a23d59c07_isrForCrosstalkSources_855394_38.stderr parsl.dataflow.dflow INFO: Task 4831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b767d980bafbc144e45d0dbe19418529 with result from task 4831 parsl.dataflow.dflow DEBUG: Task 4831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/875b51c6-17fb-47c3-bed4-77b7185e3cea_isrForCrosstalkSources_855386_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/875b51c6-17fb-47c3-bed4-77b7185e3cea_isrForCrosstalkSources_855386_9.stderr parsl.dataflow.dflow INFO: Task 4411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4f1474f03dfcd3c943092b4319d4d08 with result from task 4411 parsl.dataflow.dflow DEBUG: Task 4411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c34e469b-65d1-402b-a621-1c5016386edb_isrForCrosstalkSources_855387_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/c34e469b-65d1-402b-a621-1c5016386edb_isrForCrosstalkSources_855387_37.stderr parsl.dataflow.dflow INFO: Task 4468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f6d27cc8b84f7a11079e876c561ed0 with result from task 4468 parsl.dataflow.dflow DEBUG: Task 4468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2df8a800-6bee-4bef-aec5-730fb69d127a_isrForCrosstalkSources_855456_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/2df8a800-6bee-4bef-aec5-730fb69d127a_isrForCrosstalkSources_855456_3.stderr parsl.dataflow.dflow INFO: Task 4730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e595a7259cb7ea33a9f4c23efd412887 with result from task 4730 parsl.dataflow.dflow DEBUG: Task 4730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7feff4b2-cd21-4385-9076-3c8e8b22dc30_isrForCrosstalkSources_855441_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/7feff4b2-cd21-4385-9076-3c8e8b22dc30_isrForCrosstalkSources_855441_7.stderr parsl.dataflow.dflow INFO: Task 4626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02829b740d8c26e5a7c8eb60882a1aff with result from task 4626 parsl.dataflow.dflow DEBUG: Task 4626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/07686f91-0a62-4690-b9c5-54fb8da241be_isrForCrosstalkSources_855426_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/07686f91-0a62-4690-b9c5-54fb8da241be_isrForCrosstalkSources_855426_35.stderr parsl.dataflow.dflow INFO: Task 4498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e2d64819c709d3dde24ce04076c18c5 with result from task 4498 parsl.dataflow.dflow DEBUG: Task 4498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ff71dee4-b287-4f71-838d-98d488dff6dc_isrForCrosstalkSources_855403_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ff71dee4-b287-4f71-838d-98d488dff6dc_isrForCrosstalkSources_855403_53.stderr parsl.dataflow.dflow INFO: Task 4635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b8a0c2a953b96a7c69533db18973293 with result from task 4635 parsl.dataflow.dflow DEBUG: Task 4635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14b0a0c1-1714-4f62-b60a-fd1209b87b62_isrForCrosstalkSources_855442_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/14b0a0c1-1714-4f62-b60a-fd1209b87b62_isrForCrosstalkSources_855442_54.stderr parsl.dataflow.dflow INFO: Task 4439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6083b5c2bcc3fbd130545762f602f5c2 with result from task 4439 parsl.dataflow.dflow DEBUG: Task 4439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ec17e1d7-4c7c-4593-bcf2-561fb3d4c432_isrForCrosstalkSources_855403_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/ec17e1d7-4c7c-4593-bcf2-561fb3d4c432_isrForCrosstalkSources_855403_47.stderr parsl.dataflow.dflow INFO: Task 4501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f17cd3818cc40c74056f1550161175d with result from task 4501 parsl.dataflow.dflow DEBUG: Task 4501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/516ee66c-9a98-4a17-b211-9363a2da21de_isrForCrosstalkSources_855422_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/516ee66c-9a98-4a17-b211-9363a2da21de_isrForCrosstalkSources_855422_29.stderr parsl.dataflow.dflow INFO: Task 4408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d84f756ce0b55e7c7471edd838ea4b5c with result from task 4408 parsl.dataflow.dflow DEBUG: Task 4408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2222936d-d76a-47bd-9983-220a8690326c_isrForCrosstalkSources_855419_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/2222936d-d76a-47bd-9983-220a8690326c_isrForCrosstalkSources_855419_1.stderr parsl.dataflow.dflow INFO: Task 4525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86ec2e7690f4eb9f6ac4e21be99f5e9a with result from task 4525 parsl.dataflow.dflow DEBUG: Task 4525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d7d7d656-22e8-4eee-bfee-8ec2a9cc7bb0_isrForCrosstalkSources_855386_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/d7d7d656-22e8-4eee-bfee-8ec2a9cc7bb0_isrForCrosstalkSources_855386_16.stderr parsl.dataflow.dflow INFO: Task 4721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ced4423349061975548a5484a831fb83 with result from task 4721 parsl.dataflow.dflow DEBUG: Task 4721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe41ad-7979-4797-a922-0da316fb3b73_isrForCrosstalkSources_855423_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/3bbe41ad-7979-4797-a922-0da316fb3b73_isrForCrosstalkSources_855423_36.stderr parsl.dataflow.dflow INFO: Task 4681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210cba202884e5872e527633f9ebd76f with result from task 4681 parsl.dataflow.dflow DEBUG: Task 4681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/832e078f-9688-4230-b477-bf3980fbe5c8_isrForCrosstalkSources_855411_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/832e078f-9688-4230-b477-bf3980fbe5c8_isrForCrosstalkSources_855411_50.stderr parsl.dataflow.dflow INFO: Task 4706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5658383138b306ee739d08664a1d5a89 with result from task 4706 parsl.dataflow.dflow DEBUG: Task 4706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/36735455-6f54-4a30-b95c-f0dcf7d12650_isrForCrosstalkSources_855444_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/36735455-6f54-4a30-b95c-f0dcf7d12650_isrForCrosstalkSources_855444_38.stderr parsl.dataflow.dflow INFO: Task 4669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 519848c36dc000d75cf31f977afb4fb5 with result from task 4669 parsl.dataflow.dflow DEBUG: Task 4669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/51b70413-5885-46b3-80cf-52fd8b9e418a_isrForCrosstalkSources_855416_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/51b70413-5885-46b3-80cf-52fd8b9e418a_isrForCrosstalkSources_855416_1.stderr parsl.dataflow.dflow INFO: Task 4642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7beb446cb57f9f6cbd9f4d1d42f46b with result from task 4642 parsl.dataflow.dflow DEBUG: Task 4642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5f10b2d-c0fd-4fb9-9f32-d8c56e79789e_isrForCrosstalkSources_855456_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b5f10b2d-c0fd-4fb9-9f32-d8c56e79789e_isrForCrosstalkSources_855456_32.stderr parsl.dataflow.dflow INFO: Task 4821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 073a145876e944b413aaec4a57ecd9e9 with result from task 4821 parsl.dataflow.dflow DEBUG: Task 4821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a1ba808c-4de6-4676-a279-b49deebe4717_isrForCrosstalkSources_855455_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a1ba808c-4de6-4676-a279-b49deebe4717_isrForCrosstalkSources_855455_59.stderr parsl.dataflow.dflow INFO: Task 4701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8588b252159f0fa4b71bf7a81bf60f24 with result from task 4701 parsl.dataflow.dflow DEBUG: Task 4701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/83c220ad-b89f-4d43-9453-05ea87660d85_isrForCrosstalkSources_855414_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/83c220ad-b89f-4d43-9453-05ea87660d85_isrForCrosstalkSources_855414_38.stderr parsl.dataflow.dflow INFO: Task 4469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca8b5343086e105699d786f0542cdafa with result from task 4469 parsl.dataflow.dflow DEBUG: Task 4469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/06f791a9-fa7c-4400-9c34-e13d5460520a_isrForCrosstalkSources_855453_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/06f791a9-fa7c-4400-9c34-e13d5460520a_isrForCrosstalkSources_855453_23.stderr parsl.dataflow.dflow INFO: Task 4826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c82385ea35c3b6f571092f3db0635181 with result from task 4826 parsl.dataflow.dflow DEBUG: Task 4826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/27dc8b4f-d9ff-4bec-b111-fc496e4a1b9b_isrForCrosstalkSources_855417_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/27dc8b4f-d9ff-4bec-b111-fc496e4a1b9b_isrForCrosstalkSources_855417_30.stderr parsl.dataflow.dflow INFO: Task 4497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 443334c6f6b06fd2803e8ad09efb9145 with result from task 4497 parsl.dataflow.dflow DEBUG: Task 4497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/f8a8d2e5-1d9e-419a-9e93-23b5637d6593_isrForCrosstalkSources_855410_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/f8a8d2e5-1d9e-419a-9e93-23b5637d6593_isrForCrosstalkSources_855410_38.stderr parsl.dataflow.dflow INFO: Task 4522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90d4316df10372f0e7fd5528c5f8d80c with result from task 4522 parsl.dataflow.dflow DEBUG: Task 4522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f6f4dadf-30c5-4d61-b644-342c3fc087ee_isrForCrosstalkSources_855446_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/f6f4dadf-30c5-4d61-b644-342c3fc087ee_isrForCrosstalkSources_855446_10.stderr parsl.dataflow.dflow INFO: Task 4667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1098b1864046ae68961bd888e2d26c3c with result from task 4667 parsl.dataflow.dflow DEBUG: Task 4667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/38c64183-2ca3-413c-88f0-c563c5e46d3e_isrForCrosstalkSources_855385_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/38c64183-2ca3-413c-88f0-c563c5e46d3e_isrForCrosstalkSources_855385_62.stderr parsl.dataflow.dflow INFO: Task 4672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e7808c0846ba92362d673a854853e0 with result from task 4672 parsl.dataflow.dflow DEBUG: Task 4672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7f0832db-fedb-48e7-a1d4-c6260f30afe0_isrForCrosstalkSources_855431_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/7f0832db-fedb-48e7-a1d4-c6260f30afe0_isrForCrosstalkSources_855431_29.stderr parsl.dataflow.dflow INFO: Task 4527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1160e647bb4383dc541a6a7c2f274165 with result from task 4527 parsl.dataflow.dflow DEBUG: Task 4527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1c2da685-0384-4fa5-a1cf-bfd224f47633_isrForCrosstalkSources_855452_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/1c2da685-0384-4fa5-a1cf-bfd224f47633_isrForCrosstalkSources_855452_11.stderr parsl.dataflow.dflow INFO: Task 4549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb9d757b0161a9dbe044a17dc9b702c7 with result from task 4549 parsl.dataflow.dflow DEBUG: Task 4549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0acfad0e-7d3b-4780-829e-df3be8d63809_isrForCrosstalkSources_855439_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/0acfad0e-7d3b-4780-829e-df3be8d63809_isrForCrosstalkSources_855439_40.stderr parsl.dataflow.dflow INFO: Task 4668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 713fd18b3aec2dfe2c9c46c14530dc99 with result from task 4668 parsl.dataflow.dflow DEBUG: Task 4668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/70c124d7-b15e-41ba-8de7-73232c81e948_isrForCrosstalkSources_855458_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/70c124d7-b15e-41ba-8de7-73232c81e948_isrForCrosstalkSources_855458_10.stderr parsl.dataflow.dflow INFO: Task 4691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6611c9f4bca197c1a5e7b69751000bf with result from task 4691 parsl.dataflow.dflow DEBUG: Task 4691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d5bb1139-187f-489e-a274-0b13fd6f326f_isrForCrosstalkSources_855429_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/d5bb1139-187f-489e-a274-0b13fd6f326f_isrForCrosstalkSources_855429_5.stderr parsl.dataflow.dflow INFO: Task 4670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 824173e64606a6bf2376c89e2a04dbbf with result from task 4670 parsl.dataflow.dflow DEBUG: Task 4670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b49ea62-ec8f-42d9-b837-05bda88f605b_isrForCrosstalkSources_855406_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/4b49ea62-ec8f-42d9-b837-05bda88f605b_isrForCrosstalkSources_855406_10.stderr parsl.dataflow.dflow INFO: Task 4695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe8516bfe0df0e3f53262abbbeb9a6d8 with result from task 4695 parsl.dataflow.dflow DEBUG: Task 4695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/678baf93-a650-4796-9b98-65a77b533ad6_isrForCrosstalkSources_855417_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/678baf93-a650-4796-9b98-65a77b533ad6_isrForCrosstalkSources_855417_46.stderr parsl.dataflow.dflow INFO: Task 4517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc74a189023a036d8d591bf3df3dd458 with result from task 4517 parsl.dataflow.dflow DEBUG: Task 4517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/59a66a81-7668-46d6-9b6e-cae5f2d1bc48_isrForCrosstalkSources_855426_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/59a66a81-7668-46d6-9b6e-cae5f2d1bc48_isrForCrosstalkSources_855426_9.stderr parsl.dataflow.dflow INFO: Task 4632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72266f80bc57d1e3e87e0364e453a449 with result from task 4632 parsl.dataflow.dflow DEBUG: Task 4632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/334be8ba-3ab3-4e6e-8944-793c746f82ef_isrForCrosstalkSources_855396_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/334be8ba-3ab3-4e6e-8944-793c746f82ef_isrForCrosstalkSources_855396_19.stderr parsl.dataflow.dflow INFO: Task 4541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a669646a1fb8fad12924e4519937d1b7 with result from task 4541 parsl.dataflow.dflow DEBUG: Task 4541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e4632eea-7cbc-4b66-9a21-2becfa4a983a_isrForCrosstalkSources_855402_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/e4632eea-7cbc-4b66-9a21-2becfa4a983a_isrForCrosstalkSources_855402_20.stderr parsl.dataflow.dflow INFO: Task 4538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4989dc53b0ec02dffa02a0e8467fd8cf with result from task 4538 parsl.dataflow.dflow DEBUG: Task 4538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a6ecba16-8701-4430-8234-99e4bb7cd1f0_isrForCrosstalkSources_855410_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/a6ecba16-8701-4430-8234-99e4bb7cd1f0_isrForCrosstalkSources_855410_15.stderr parsl.dataflow.dflow INFO: Task 4823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ce9b82fb9be83284b77d2acfa22c33 with result from task 4823 parsl.dataflow.dflow DEBUG: Task 4823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5726f0f1-13c1-431e-884b-824331e161f5_isrForCrosstalkSources_855405_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5726f0f1-13c1-431e-884b-824331e161f5_isrForCrosstalkSources_855405_51.stderr parsl.dataflow.dflow INFO: Task 4503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f3e66002cd9b02e3afc905dde5f446a with result from task 4503 parsl.dataflow.dflow DEBUG: Task 4503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a42060d-bc4d-4263-9e2a-76fdaa105fa0_isrForCrosstalkSources_855447_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3a42060d-bc4d-4263-9e2a-76fdaa105fa0_isrForCrosstalkSources_855447_3.stderr parsl.dataflow.dflow INFO: Task 4450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddfa42fa2d6d0941646c69632b24f0a4 with result from task 4450 parsl.dataflow.dflow DEBUG: Task 4450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/064bfd6b-aecd-41f7-bf67-ca7e708861a5_isrForCrosstalkSources_855422_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/064bfd6b-aecd-41f7-bf67-ca7e708861a5_isrForCrosstalkSources_855422_38.stderr parsl.dataflow.dflow INFO: Task 4356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 899306a43e0c9aa0b6a04107d9dd71f3 with result from task 4356 parsl.dataflow.dflow DEBUG: Task 4356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8ccb4dd0-6455-4105-9c84-3200ef1d6669_isrForCrosstalkSources_855442_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8ccb4dd0-6455-4105-9c84-3200ef1d6669_isrForCrosstalkSources_855442_26.stderr parsl.dataflow.dflow INFO: Task 4484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6a51768b4c0f523300d5dc80c26a0e1 with result from task 4484 parsl.dataflow.dflow DEBUG: Task 4484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f4694f4a-d31c-4d5c-9e30-e42bd0a20736_isrForCrosstalkSources_855398_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f4694f4a-d31c-4d5c-9e30-e42bd0a20736_isrForCrosstalkSources_855398_44.stderr parsl.dataflow.dflow INFO: Task 4532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a033e2643093bab08a252b8cf428b69b with result from task 4532 parsl.dataflow.dflow DEBUG: Task 4532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35a4c009-91cc-4f75-87dc-288c1501f7b5_isrForCrosstalkSources_855385_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35a4c009-91cc-4f75-87dc-288c1501f7b5_isrForCrosstalkSources_855385_20.stderr parsl.dataflow.dflow INFO: Task 4455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41c9df3e65300a2f8b85d434eed8014b with result from task 4455 parsl.dataflow.dflow DEBUG: Task 4455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7730ad6f-78f8-43e6-9e21-cb8b0c7b5247_isrForCrosstalkSources_855446_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/7730ad6f-78f8-43e6-9e21-cb8b0c7b5247_isrForCrosstalkSources_855446_61.stderr parsl.dataflow.dflow INFO: Task 4732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38f9abf2ba349480c4b320c9e6b816f9 with result from task 4732 parsl.dataflow.dflow DEBUG: Task 4732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/9b781c53-dc37-4758-b61d-4c27c00c3d93_isrForCrosstalkSources_855425_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/9b781c53-dc37-4758-b61d-4c27c00c3d93_isrForCrosstalkSources_855425_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:21 proc_lsst.multi:146 [INFO] found job 16964054 in provider astro proc_lsst.multi INFO: found job 16964054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9371846c6d0339a8415acb3a034fffa with result from task 4481 parsl.dataflow.dflow DEBUG: Task 4481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/f07e2776-9c7d-468a-be8c-143392393dbb_isrForCrosstalkSources_855403_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/f07e2776-9c7d-468a-be8c-143392393dbb_isrForCrosstalkSources_855403_48.stderr parsl.dataflow.dflow INFO: Task 4631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98b23605be244eb24a60cf695ad62f09 with result from task 4631 parsl.dataflow.dflow DEBUG: Task 4631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ce66ad49-af30-494f-8f75-5e097fd5abb7_isrForCrosstalkSources_855459_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/ce66ad49-af30-494f-8f75-5e097fd5abb7_isrForCrosstalkSources_855459_41.stderr parsl.dataflow.dflow INFO: Task 4328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe1b4daee445af4359c5eb01339136f9 with result from task 4328 parsl.dataflow.dflow DEBUG: Task 4328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a8fdbe8-868e-489f-bd54-d7e86d65a8f0_isrForCrosstalkSources_855442_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/8a8fdbe8-868e-489f-bd54-d7e86d65a8f0_isrForCrosstalkSources_855442_19.stderr parsl.dataflow.dflow INFO: Task 4489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fcfb39b926301bf249a1deaf76be665 with result from task 4489 parsl.dataflow.dflow DEBUG: Task 4489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b526bc0d-2076-41ea-a098-2da54dd9e2f0_isrForCrosstalkSources_855456_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/b526bc0d-2076-41ea-a098-2da54dd9e2f0_isrForCrosstalkSources_855456_55.stderr parsl.dataflow.dflow INFO: Task 4440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45bbf2082d1b98b16a3fe34f4cdcf6c0 with result from task 4440 parsl.dataflow.dflow DEBUG: Task 4440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b5ff75ff-bbbe-4034-84e4-6760b6066f3c_isrForCrosstalkSources_855446_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/b5ff75ff-bbbe-4034-84e4-6760b6066f3c_isrForCrosstalkSources_855446_3.stderr parsl.dataflow.dflow INFO: Task 4655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 163bc6f5e6455ca896d9a5dec54f74ef with result from task 4655 parsl.dataflow.dflow DEBUG: Task 4655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/268e6567-ff5b-499d-9a8b-56e65bdea9e0_isrForCrosstalkSources_855412_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/268e6567-ff5b-499d-9a8b-56e65bdea9e0_isrForCrosstalkSources_855412_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:21 proc_lsst.multi:146 [INFO] found job 16964055 in provider astro proc_lsst.multi INFO: found job 16964055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 4500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6639cf57c72399a8b282e6a723c9aec3 with result from task 4500 parsl.dataflow.dflow DEBUG: Task 4500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5b0ae83b-d399-4226-994e-93599832e162_isrForCrosstalkSources_855397_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/5b0ae83b-d399-4226-994e-93599832e162_isrForCrosstalkSources_855397_50.stderr parsl.dataflow.dflow INFO: Task 4765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f75313ce1884eb12489aae5f45a58e with result from task 4765 parsl.dataflow.dflow DEBUG: Task 4765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d2a07bff-6cf7-4f86-8ee9-4dbc41e9fd11_isrForCrosstalkSources_855438_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/d2a07bff-6cf7-4f86-8ee9-4dbc41e9fd11_isrForCrosstalkSources_855438_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:21 proc_lsst.multi:146 [INFO] found job 16964056 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964056 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 4641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d7a14d7f4b214c1a6993bc16fafa506 with result from task 4641 parsl.dataflow.dflow DEBUG: Task 4641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0279d00e-146b-4758-8de1-72df7d54bbca_isrForCrosstalkSources_855403_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/0279d00e-146b-4758-8de1-72df7d54bbca_isrForCrosstalkSources_855403_20.stderr parsl.dataflow.dflow INFO: Task 4556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8402ddaa26625e7f4831d6f4919a8d19 with result from task 4556 parsl.dataflow.dflow DEBUG: Task 4556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0626d095-4c61-46ea-9594-b5a50908bcea_isrForCrosstalkSources_855426_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/0626d095-4c61-46ea-9594-b5a50908bcea_isrForCrosstalkSources_855426_41.stderr parsl.dataflow.dflow INFO: Task 4366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b7e265b75c3593496906817e00ed3b2 with result from task 4366 parsl.dataflow.dflow DEBUG: Task 4366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/07e94a09-8c5a-49b5-8322-c97a8195a171_isrForCrosstalkSources_855394_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/07e94a09-8c5a-49b5-8322-c97a8195a171_isrForCrosstalkSources_855394_19.stderr parsl.dataflow.dflow INFO: Task 4825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7454db621d0bea1b8bd994a468752556 with result from task 4825 parsl.dataflow.dflow DEBUG: Task 4825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2ce9003c-676a-4620-b262-112e74ec49f8_isrForCrosstalkSources_855450_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/2ce9003c-676a-4620-b262-112e74ec49f8_isrForCrosstalkSources_855450_46.stderr parsl.dataflow.dflow INFO: Task 4724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0519538a01375299f64b9a8e0591fd6b with result from task 4724 parsl.dataflow.dflow DEBUG: Task 4724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/66b01b7d-8924-45e6-9e4e-c5884a25c90a_isrForCrosstalkSources_855392_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/66b01b7d-8924-45e6-9e4e-c5884a25c90a_isrForCrosstalkSources_855392_59.stderr parsl.dataflow.dflow INFO: Task 4480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4389e3825b8267e37d0a071089087a with result from task 4480 parsl.dataflow.dflow DEBUG: Task 4480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac8ba7bc-85a4-44b4-b953-76bb1cafcc77_isrForCrosstalkSources_855394_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/ac8ba7bc-85a4-44b4-b953-76bb1cafcc77_isrForCrosstalkSources_855394_37.stderr parsl.dataflow.dflow INFO: Task 4530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b90e024d140e7d6f06bf41e8d57db268 with result from task 4530 parsl.dataflow.dflow DEBUG: Task 4530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f586b0f1-8748-4989-b208-52f361fe26e2_isrForCrosstalkSources_855384_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f586b0f1-8748-4989-b208-52f361fe26e2_isrForCrosstalkSources_855384_35.stderr parsl.dataflow.dflow INFO: Task 4513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb88feb59c7855498a307856a4b25519 with result from task 4513 parsl.dataflow.dflow DEBUG: Task 4513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3a3b6ce6-1676-4123-b447-cb557157e9df_isrForCrosstalkSources_855445_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3a3b6ce6-1676-4123-b447-cb557157e9df_isrForCrosstalkSources_855445_2.stderr parsl.dataflow.dflow INFO: Task 4692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9ea3d661ba2e3eb6d1c3004f068a20e with result from task 4692 parsl.dataflow.dflow DEBUG: Task 4692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a08f2727-8580-404e-b6ce-35d4a5d974fd_isrForCrosstalkSources_855455_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/a08f2727-8580-404e-b6ce-35d4a5d974fd_isrForCrosstalkSources_855455_44.stderr parsl.dataflow.dflow INFO: Task 4793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db34a857d5531ceff790df0f7d656d5b with result from task 4793 parsl.dataflow.dflow DEBUG: Task 4793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3fe92d09-98e7-4039-b317-65458893f857_isrForCrosstalkSources_855452_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/3fe92d09-98e7-4039-b317-65458893f857_isrForCrosstalkSources_855452_56.stderr parsl.dataflow.dflow INFO: Task 4795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a93487ee39940123ebfbd54f3a030372 with result from task 4795 parsl.dataflow.dflow DEBUG: Task 4795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/34c2406b-43ac-4d48-b26c-3fa0c42c8ff1_isrForCrosstalkSources_855420_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/34c2406b-43ac-4d48-b26c-3fa0c42c8ff1_isrForCrosstalkSources_855420_4.stderr parsl.dataflow.dflow INFO: Task 4784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0453f75f0cc2fd416801858cd1c9173e with result from task 4784 parsl.dataflow.dflow DEBUG: Task 4784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/bc527e5f-79be-44af-9eea-a487d36bd447_isrForCrosstalkSources_855429_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/bc527e5f-79be-44af-9eea-a487d36bd447_isrForCrosstalkSources_855429_1.stderr parsl.dataflow.dflow INFO: Task 4524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae54a4736aa2f1b7d0bfbab7e54ec14 with result from task 4524 parsl.dataflow.dflow DEBUG: Task 4524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/67713071-13a8-48b4-b25b-667b5c3f2d8d_isrForCrosstalkSources_855436_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/67713071-13a8-48b4-b25b-667b5c3f2d8d_isrForCrosstalkSources_855436_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:22 proc_lsst.multi:146 [INFO] found job 16964057 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964057 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 4645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc10ff9d1bf5dae397dcb29c4fad8ff2 with result from task 4645 parsl.dataflow.dflow DEBUG: Task 4645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4fefa4b9-503d-4fe0-a8fb-f9510222779e_isrForCrosstalkSources_855430_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/4fefa4b9-503d-4fe0-a8fb-f9510222779e_isrForCrosstalkSources_855430_30.stderr parsl.dataflow.dflow INFO: Task 4660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 700db3b095075dc70152241c738f149d with result from task 4660 parsl.dataflow.dflow DEBUG: Task 4660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/bc6c7ae3-a516-4680-bffa-594acf1042c5_isrForCrosstalkSources_855422_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/bc6c7ae3-a516-4680-bffa-594acf1042c5_isrForCrosstalkSources_855422_46.stderr parsl.dataflow.dflow INFO: Task 4661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 738f355ac844611e864f7a05fc8b6b90 with result from task 4661 parsl.dataflow.dflow DEBUG: Task 4661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d1f6e0bf-b155-4070-b0bb-d06251b9f348_isrForCrosstalkSources_855436_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/d1f6e0bf-b155-4070-b0bb-d06251b9f348_isrForCrosstalkSources_855436_20.stderr parsl.dataflow.dflow INFO: Task 4521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4df7cc4461cc99e89c6d800117ad0fa with result from task 4521 parsl.dataflow.dflow DEBUG: Task 4521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3768d226-8eaf-4db7-99f4-63b69989856d_isrForCrosstalkSources_855447_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/3768d226-8eaf-4db7-99f4-63b69989856d_isrForCrosstalkSources_855447_38.stderr parsl.dataflow.dflow INFO: Task 4809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ec84ef5ae3ca5673d2ef7fb8e9460ae with result from task 4809 parsl.dataflow.dflow DEBUG: Task 4809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f240ea4d-5bc7-4265-9c92-2b2a5916e739_isrForCrosstalkSources_855429_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/f240ea4d-5bc7-4265-9c92-2b2a5916e739_isrForCrosstalkSources_855429_36.stderr parsl.dataflow.dflow INFO: Task 4562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f169deb66cda3b9b0ef9ff5b16516c1d with result from task 4562 parsl.dataflow.dflow DEBUG: Task 4562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9e5c00be-0e54-49e5-a2bc-02558fb52530_isrForCrosstalkSources_855428_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/9e5c00be-0e54-49e5-a2bc-02558fb52530_isrForCrosstalkSources_855428_48.stderr parsl.dataflow.dflow INFO: Task 4547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f6f9aacac7b16f9b12554eac26ba75b with result from task 4547 parsl.dataflow.dflow DEBUG: Task 4547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c111500c-792d-4f85-a59d-b26efc2777a4_isrForCrosstalkSources_855413_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/c111500c-792d-4f85-a59d-b26efc2777a4_isrForCrosstalkSources_855413_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:22 proc_lsst.multi:146 [INFO] found job 16964058 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964058 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 4326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef71501c5e43ace9d537b208e7cadce3 with result from task 4326 parsl.dataflow.dflow DEBUG: Task 4326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e121a2e2-674f-4a18-bf37-64c9f3821d06_isrForCrosstalkSources_855394_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/e121a2e2-674f-4a18-bf37-64c9f3821d06_isrForCrosstalkSources_855394_58.stderr parsl.dataflow.dflow INFO: Task 4647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3c88110cb8cdd37fc9c22597a119b23 with result from task 4647 parsl.dataflow.dflow DEBUG: Task 4647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c91ed46f-ec78-4234-9e6a-83771758e4e5_isrForCrosstalkSources_855401_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c91ed46f-ec78-4234-9e6a-83771758e4e5_isrForCrosstalkSources_855401_9.stderr parsl.dataflow.dflow INFO: Task 4533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a39d123b7b7de2a90973e4a68f5d4842 with result from task 4533 parsl.dataflow.dflow DEBUG: Task 4533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6bd9e5c1-843d-47a0-9a21-7ab0cc433650_isrForCrosstalkSources_855395_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6bd9e5c1-843d-47a0-9a21-7ab0cc433650_isrForCrosstalkSources_855395_38.stderr parsl.dataflow.dflow INFO: Task 4531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d534730b4818527d0da6ab56540ecbeb with result from task 4531 parsl.dataflow.dflow DEBUG: Task 4531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/acf2dd10-58aa-47ac-8050-ba913d6635df_isrForCrosstalkSources_855398_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/acf2dd10-58aa-47ac-8050-ba913d6635df_isrForCrosstalkSources_855398_12.stderr parsl.dataflow.dflow INFO: Task 4812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcd0102e7afe1db46e8b74456bab2b71 with result from task 4812 parsl.dataflow.dflow DEBUG: Task 4812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9eb8efc5-954d-4bbb-a3d9-6b76be3c9caf_isrForCrosstalkSources_855411_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/9eb8efc5-954d-4bbb-a3d9-6b76be3c9caf_isrForCrosstalkSources_855411_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:23 proc_lsst.multi:146 [INFO] found job 16964059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 4564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76d9bf575c30607b04e48e4db2762574 with result from task 4564 parsl.dataflow.dflow DEBUG: Task 4564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fb600c-7f77-4a27-95c8-6e993e49233a_isrForCrosstalkSources_855420_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/52fb600c-7f77-4a27-95c8-6e993e49233a_isrForCrosstalkSources_855420_1.stderr parsl.dataflow.dflow INFO: Task 4563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 719a7fb3f7f6c3eee3d439533625a79c with result from task 4563 parsl.dataflow.dflow DEBUG: Task 4563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/98ce37d5-bc3b-441b-870f-0c2c8938d386_isrForCrosstalkSources_855455_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/98ce37d5-bc3b-441b-870f-0c2c8938d386_isrForCrosstalkSources_855455_38.stderr parsl.dataflow.dflow INFO: Task 4725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6b44739438ca7592027a4c9fab2dc4b with result from task 4725 parsl.dataflow.dflow DEBUG: Task 4725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7e4cd012-abff-414e-9fa9-d66907d7edc9_isrForCrosstalkSources_855449_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/7e4cd012-abff-414e-9fa9-d66907d7edc9_isrForCrosstalkSources_855449_62.stderr parsl.dataflow.dflow INFO: Task 4767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39f5eabefbeef134be4803360d84ae2b with result from task 4767 parsl.dataflow.dflow DEBUG: Task 4767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e2ef4678-8fc9-4441-8244-460e41055738_isrForCrosstalkSources_855396_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/e2ef4678-8fc9-4441-8244-460e41055738_isrForCrosstalkSources_855396_54.stderr parsl.dataflow.dflow INFO: Task 4728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5652b263882c947646fd1a2905349256 with result from task 4728 parsl.dataflow.dflow DEBUG: Task 4728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bbefc63f-bd27-46e4-865b-9015114ec10e_isrForCrosstalkSources_855450_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/bbefc63f-bd27-46e4-865b-9015114ec10e_isrForCrosstalkSources_855450_58.stderr parsl.dataflow.dflow INFO: Task 4560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aa12507a52fe2184e3d6b5b633f1ccc with result from task 4560 parsl.dataflow.dflow DEBUG: Task 4560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/bbed5fad-0dc9-4390-aa69-7f331e08d7a4_isrForCrosstalkSources_855444_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/bbed5fad-0dc9-4390-aa69-7f331e08d7a4_isrForCrosstalkSources_855444_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:23 proc_lsst.multi:146 [INFO] found job 16964060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.dataflow.dflow INFO: Task 4561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7de3d82a53f43f212c820903d4247f1f with result from task 4561 parsl.dataflow.dflow DEBUG: Task 4561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/32284a33-f9ed-4b6e-ba12-72cf5ac718ff_isrForCrosstalkSources_855440_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/32284a33-f9ed-4b6e-ba12-72cf5ac718ff_isrForCrosstalkSources_855440_41.stderr parsl.dataflow.dflow INFO: Task 4536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 566bdc673e44b9c85541bcf32e9dbdc4 with result from task 4536 parsl.dataflow.dflow DEBUG: Task 4536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e4f34cde-7b74-4d88-90c7-35aa8b2a8705_isrForCrosstalkSources_855421_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e4f34cde-7b74-4d88-90c7-35aa8b2a8705_isrForCrosstalkSources_855421_60.stderr parsl.dataflow.dflow INFO: Task 4845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dbd4f7b36bbd025aecf1c042982e368 with result from task 4845 parsl.dataflow.dflow DEBUG: Task 4845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/72105a25-69dd-496d-8010-ac0e01663008_isrForCrosstalkSources_855427_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/72105a25-69dd-496d-8010-ac0e01663008_isrForCrosstalkSources_855427_15.stderr parsl.dataflow.dflow INFO: Task 4490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b455f56b5c44d895735cba68706ad74d with result from task 4490 parsl.dataflow.dflow DEBUG: Task 4490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d0627ec3-bc67-4c88-9d99-0572e83d0e02_isrForCrosstalkSources_855431_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/d0627ec3-bc67-4c88-9d99-0572e83d0e02_isrForCrosstalkSources_855431_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:23 proc_lsst.multi:146 [INFO] found job 16964061 in provider escience_a40 proc_lsst.multi INFO: found job 16964061 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca6cd41f3c839e41584b60648d99485b with result from task 4934 parsl.dataflow.dflow DEBUG: Task 4934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b6f802ed-1426-40fb-b6b3-735eaf9675b6_isrForCrosstalkSources_855398_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/b6f802ed-1426-40fb-b6b3-735eaf9675b6_isrForCrosstalkSources_855398_27.stderr parsl.dataflow.dflow INFO: Task 4946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4cb362881ce3ad81c910d2da54b353d with result from task 4946 parsl.dataflow.dflow DEBUG: Task 4946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/50710850-64b7-4a36-b7b6-a98eac832c56_isrForCrosstalkSources_855435_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/50710850-64b7-4a36-b7b6-a98eac832c56_isrForCrosstalkSources_855435_27.stderr parsl.dataflow.dflow INFO: Task 4743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd821ac029ec0c7c60c3c38ae22fe94 with result from task 4743 parsl.dataflow.dflow DEBUG: Task 4743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/eeb9b7b9-edc5-419f-8f93-dc8188d06b93_isrForCrosstalkSources_855448_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/eeb9b7b9-edc5-419f-8f93-dc8188d06b93_isrForCrosstalkSources_855448_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:24 proc_lsst.multi:146 [INFO] found job 16964062 in provider escience_a40 proc_lsst.multi INFO: found job 16964062 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27994ecf67b20d0b1d57ba4a4d6011df with result from task 4643 parsl.dataflow.dflow DEBUG: Task 4643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9ed0745b-960a-49f1-bfa1-4a4e258326fc_isrForCrosstalkSources_855426_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/9ed0745b-960a-49f1-bfa1-4a4e258326fc_isrForCrosstalkSources_855426_2.stderr parsl.dataflow.dflow INFO: Task 4646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c90332f424d0d3952dd930fce5bce2d8 with result from task 4646 parsl.dataflow.dflow DEBUG: Task 4646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/959b09ba-ebd3-46bc-9eed-04450cd2c0b1_isrForCrosstalkSources_855425_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/959b09ba-ebd3-46bc-9eed-04450cd2c0b1_isrForCrosstalkSources_855425_56.stderr parsl.dataflow.dflow INFO: Task 4355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3874b679bb11ee05d633e529ec7628e0 with result from task 4355 parsl.dataflow.dflow DEBUG: Task 4355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/888f5f1c-e305-470e-a352-91a28cd5361f_isrForCrosstalkSources_855382_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/888f5f1c-e305-470e-a352-91a28cd5361f_isrForCrosstalkSources_855382_1.stderr parsl.dataflow.dflow INFO: Task 4709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24d10f82be4b4a65f65529cec60a05bc with result from task 4709 parsl.dataflow.dflow DEBUG: Task 4709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1e4bf6b-e35c-4c9a-9f8e-ed07582d12e6_isrForCrosstalkSources_855458_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/c1e4bf6b-e35c-4c9a-9f8e-ed07582d12e6_isrForCrosstalkSources_855458_61.stderr parsl.dataflow.dflow INFO: Task 4727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 179e8b2e6bc99bab6b726560fead9366 with result from task 4727 parsl.dataflow.dflow DEBUG: Task 4727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/974c4d6a-06f5-4c8f-b592-4cf1c8a913ba_isrForCrosstalkSources_855381_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/974c4d6a-06f5-4c8f-b592-4cf1c8a913ba_isrForCrosstalkSources_855381_50.stderr parsl.dataflow.dflow INFO: Task 4555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d04db9fc8869b30bb8b462a9db0c266 with result from task 4555 parsl.dataflow.dflow DEBUG: Task 4555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/9960c492-95cd-477f-8ca7-fb3a0e45381c_isrForCrosstalkSources_855412_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/9960c492-95cd-477f-8ca7-fb3a0e45381c_isrForCrosstalkSources_855412_60.stderr parsl.dataflow.dflow INFO: Task 4789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b662ba134eb836b0493f9ec13a5792d2 with result from task 4789 parsl.dataflow.dflow DEBUG: Task 4789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/33a3a14c-ba4e-414a-90bb-e8b410ddefbe_isrForCrosstalkSources_855392_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/33a3a14c-ba4e-414a-90bb-e8b410ddefbe_isrForCrosstalkSources_855392_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:24 proc_lsst.multi:146 [INFO] found job 16964063 in provider escience_a40 proc_lsst.multi INFO: found job 16964063 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8c9e397fdffe78747591937837d8eaa with result from task 4768 parsl.dataflow.dflow DEBUG: Task 4768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/337463f6-4feb-40c1-89f0-e48dad5ef36b_isrForCrosstalkSources_855453_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/337463f6-4feb-40c1-89f0-e48dad5ef36b_isrForCrosstalkSources_855453_31.stderr parsl.dataflow.dflow INFO: Task 4923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b95936935ca02fe5bc9bc740609a7804 with result from task 4923 parsl.dataflow.dflow DEBUG: Task 4923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/71e62f60-3261-4c5c-b3f9-afbb73dc9a81_isrForCrosstalkSources_855381_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/71e62f60-3261-4c5c-b3f9-afbb73dc9a81_isrForCrosstalkSources_855381_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:24 proc_lsst.multi:146 [INFO] found job 16964064 in provider escience_a40 proc_lsst.multi INFO: found job 16964064 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:24 proc_lsst.multi:146 [INFO] found job 16964065 in provider escience_a40 proc_lsst.multi INFO: found job 16964065 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3117ce37b022bad5476e7f937647df78 with result from task 4819 parsl.dataflow.dflow DEBUG: Task 4819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/147226d3-087e-42fb-9435-96360186e2fe_isrForCrosstalkSources_855395_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/147226d3-087e-42fb-9435-96360186e2fe_isrForCrosstalkSources_855395_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:25 proc_lsst.multi:146 [INFO] found job 16964066 in provider escience_a40 proc_lsst.multi INFO: found job 16964066 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bafe8396e4c6e29bfdb95812a6af11f2 with result from task 4548 parsl.dataflow.dflow DEBUG: Task 4548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b292ff7-d5db-4b92-9f28-23e6274f1752_isrForCrosstalkSources_855449_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/9b292ff7-d5db-4b92-9f28-23e6274f1752_isrForCrosstalkSources_855449_49.stderr parsl.dataflow.dflow INFO: Task 4659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d52d954aa458317a40ad6ed5cde087f with result from task 4659 parsl.dataflow.dflow DEBUG: Task 4659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/75121019-7b23-4568-bd6b-9ca86b344b89_isrForCrosstalkSources_855389_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/75121019-7b23-4568-bd6b-9ca86b344b89_isrForCrosstalkSources_855389_1.stderr parsl.dataflow.dflow INFO: Task 4824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de208e6739d76d32be06e0f22d87760b with result from task 4824 parsl.dataflow.dflow DEBUG: Task 4824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/43875a40-dbe6-4d86-9627-3ad999ec8ebf_isrForCrosstalkSources_855399_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/43875a40-dbe6-4d86-9627-3ad999ec8ebf_isrForCrosstalkSources_855399_3.stderr parsl.dataflow.dflow INFO: Task 4927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09b17e1db10d5782e70bf87e80199ab9 with result from task 4927 parsl.dataflow.dflow DEBUG: Task 4927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c0090916-f123-4eac-8c1a-c07c3ea70807_isrForCrosstalkSources_855401_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/c0090916-f123-4eac-8c1a-c07c3ea70807_isrForCrosstalkSources_855401_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:25 proc_lsst.multi:146 [INFO] found job 16964067 in provider escience_a40 proc_lsst.multi INFO: found job 16964067 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1ee657b57e50dec85ae2770d2d1122e with result from task 4913 parsl.dataflow.dflow DEBUG: Task 4913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ea0449e-5d4a-4ed6-ba3e-fcd9a52c8683_isrForCrosstalkSources_855418_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/5ea0449e-5d4a-4ed6-ba3e-fcd9a52c8683_isrForCrosstalkSources_855418_20.stderr parsl.dataflow.dflow INFO: Task 4352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf80e760d16128dde3b96fc4ff074f17 with result from task 4352 parsl.dataflow.dflow DEBUG: Task 4352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2530b659-1ad2-418c-bbb0-c7cabc85e22b_isrForCrosstalkSources_855406_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/2530b659-1ad2-418c-bbb0-c7cabc85e22b_isrForCrosstalkSources_855406_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:25 proc_lsst.multi:146 [INFO] found job 16964068 in provider escience_a40 proc_lsst.multi INFO: found job 16964068 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20d33a325893a01914a4f59cf1cf9f34 with result from task 4818 parsl.dataflow.dflow DEBUG: Task 4818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b37a09ef-adb9-4dfe-9720-3c1e15440aba_isrForCrosstalkSources_855425_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/b37a09ef-adb9-4dfe-9720-3c1e15440aba_isrForCrosstalkSources_855425_28.stderr parsl.dataflow.dflow INFO: Task 4416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8af07ce41e6a22543a94e1b1345466dc with result from task 4416 parsl.dataflow.dflow DEBUG: Task 4416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/763de7f7-9a3b-4641-83a6-3ccd826bbe54_isrForCrosstalkSources_855394_60.stdout parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.dataflow.dflow INFO: Standard error for task 4416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/763de7f7-9a3b-4641-83a6-3ccd826bbe54_isrForCrosstalkSources_855394_60.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:26 proc_lsst.multi:146 [INFO] found job 16964069 in provider escience_a40 proc_lsst.multi INFO: found job 16964069 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:26 proc_lsst.multi:146 [INFO] found job 16964070 in provider escience_a40 proc_lsst.multi INFO: found job 16964070 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 068b77cfb910eefa38904c42631398b5 with result from task 4654 parsl.dataflow.dflow DEBUG: Task 4654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/29de2ba7-8342-4625-bb4d-5f50226f459d_isrForCrosstalkSources_855429_15.stdout parsl.dataflow.dflow INFO: Standard error for task 4654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/29de2ba7-8342-4625-bb4d-5f50226f459d_isrForCrosstalkSources_855429_15.stderr parsl.dataflow.dflow INFO: Task 4650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 566717c6387a6461dcf6f45845128623 with result from task 4650 parsl.dataflow.dflow DEBUG: Task 4650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/80a659b8-934b-4bd4-aaea-d74688cb1802_isrForCrosstalkSources_855414_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/80a659b8-934b-4bd4-aaea-d74688cb1802_isrForCrosstalkSources_855414_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:26 proc_lsst.multi:146 [INFO] found job 16964071 in provider escience_a40 proc_lsst.multi INFO: found job 16964071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.dataflow.dflow INFO: Task 4534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae814f112b628c2bdb223e5068f23b9a with result from task 4534 parsl.dataflow.dflow DEBUG: Task 4534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0e06a9af-7847-4a80-ba09-b01a22a29f69_isrForCrosstalkSources_855406_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/0e06a9af-7847-4a80-ba09-b01a22a29f69_isrForCrosstalkSources_855406_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:26 proc_lsst.multi:146 [INFO] found job 16964072 in provider escience_a40 proc_lsst.multi INFO: found job 16964072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:26 proc_lsst.multi:146 [INFO] found job 16964073 in provider escience_a40 proc_lsst.multi INFO: found job 16964073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:27 proc_lsst.multi:146 [INFO] found job 16964074 in provider escience_a40 proc_lsst.multi INFO: found job 16964074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:27 proc_lsst.multi:146 [INFO] found job 16964075 in provider escience_a40 proc_lsst.multi INFO: found job 16964075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:27 proc_lsst.multi:146 [INFO] found job 16964076 in provider escience_a40 proc_lsst.multi INFO: found job 16964076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:28 proc_lsst.multi:146 [INFO] found job 16964077 in provider escience_a40 proc_lsst.multi INFO: found job 16964077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:28 proc_lsst.multi:146 [INFO] found job 16964078 in provider escience_a40 proc_lsst.multi INFO: found job 16964078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:28 proc_lsst.multi:146 [INFO] found job 16964079 in provider escience_a40 proc_lsst.multi INFO: found job 16964079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:28 proc_lsst.multi:146 [INFO] found job 16964080 in provider escience_a40 proc_lsst.multi INFO: found job 16964080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:29 proc_lsst.multi:146 [INFO] found job 16964081 in provider checkpoint proc_lsst.multi INFO: found job 16964081 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:29 proc_lsst.multi:146 [INFO] found job 16964082 in provider checkpoint proc_lsst.multi INFO: found job 16964082 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:29 proc_lsst.multi:146 [INFO] found job 16964083 in provider checkpoint proc_lsst.multi INFO: found job 16964083 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:29 proc_lsst.multi:146 [INFO] found job 16964084 in provider checkpoint proc_lsst.multi INFO: found job 16964084 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:30 proc_lsst.multi:146 [INFO] found job 16964085 in provider checkpoint proc_lsst.multi INFO: found job 16964085 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:30 proc_lsst.multi:146 [INFO] found job 16964086 in provider checkpoint proc_lsst.multi INFO: found job 16964086 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:30 proc_lsst.multi:146 [INFO] found job 16964087 in provider checkpoint proc_lsst.multi INFO: found job 16964087 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:30 proc_lsst.multi:146 [INFO] found job 16964088 in provider checkpoint proc_lsst.multi INFO: found job 16964088 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d79111b04aa296ab87eae85d7771a1b with result from task 4658 parsl.dataflow.dflow DEBUG: Task 4658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7e26897a-50cd-4b6a-8b0a-1b2973bb9b78_isrForCrosstalkSources_855392_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/7e26897a-50cd-4b6a-8b0a-1b2973bb9b78_isrForCrosstalkSources_855392_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:31 proc_lsst.multi:146 [INFO] found job 16964089 in provider checkpoint proc_lsst.multi INFO: found job 16964089 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:31 proc_lsst.multi:146 [INFO] found job 16964090 in provider checkpoint proc_lsst.multi INFO: found job 16964090 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fd21c7cfd7343ff346ac3a35ed96aec with result from task 4817 parsl.dataflow.dflow DEBUG: Task 4817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/26b4dc17-608f-41c2-8f7e-efa103706caa_isrForCrosstalkSources_855446_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/26b4dc17-608f-41c2-8f7e-efa103706caa_isrForCrosstalkSources_855446_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:31 proc_lsst.multi:146 [INFO] found job 16964091 in provider checkpoint proc_lsst.multi INFO: found job 16964091 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:32 proc_lsst.multi:146 [INFO] found job 16964092 in provider checkpoint proc_lsst.multi INFO: found job 16964092 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:32 proc_lsst.multi:146 [INFO] found job 16964093 in provider checkpoint proc_lsst.multi INFO: found job 16964093 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:32 proc_lsst.multi:146 [INFO] found job 16964094 in provider checkpoint proc_lsst.multi INFO: found job 16964094 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:32 proc_lsst.multi:146 [INFO] found job 16964095 in provider checkpoint proc_lsst.multi INFO: found job 16964095 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:32 proc_lsst.multi:146 [INFO] found job 16964096 in provider checkpoint proc_lsst.multi INFO: found job 16964096 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:33 proc_lsst.multi:146 [INFO] found job 16964097 in provider checkpoint proc_lsst.multi INFO: found job 16964097 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 3801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7b09b319f1911b03d778459aa22b49 with result from task 3801 parsl.dataflow.dflow DEBUG: Task 3801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e283ec3a-5306-483e-b9f9-282f664c9e23_isrForCrosstalkSources_855432_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e283ec3a-5306-483e-b9f9-282f664c9e23_isrForCrosstalkSources_855432_13.stderr parsl.dataflow.dflow INFO: Task 3804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b0bd970a8314224c2e18677f85d76cf with result from task 3804 parsl.dataflow.dflow DEBUG: Task 3804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e62ce6ff-1418-4c3e-a1f6-8c4b5b3efe2d_isrForCrosstalkSources_855444_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e62ce6ff-1418-4c3e-a1f6-8c4b5b3efe2d_isrForCrosstalkSources_855444_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:33 proc_lsst.multi:146 [INFO] found job 16964099 in provider checkpoint proc_lsst.multi INFO: found job 16964099 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60991185d3caca69d46d25fd29e34bc7 with result from task 4876 parsl.dataflow.dflow DEBUG: Task 4876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0c110788-8b45-4650-922b-1f152f62c9f6_isrForCrosstalkSources_855382_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/0c110788-8b45-4650-922b-1f152f62c9f6_isrForCrosstalkSources_855382_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:33 proc_lsst.multi:146 [INFO] found job 16964100 in provider checkpoint proc_lsst.multi INFO: found job 16964100 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c700a1bb8ca2a86533667491652b772 with result from task 4752 parsl.dataflow.dflow DEBUG: Task 4752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/afedd47d-5de8-4ea4-95c1-f778caf483d6_isrForCrosstalkSources_855428_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855428/afedd47d-5de8-4ea4-95c1-f778caf483d6_isrForCrosstalkSources_855428_55.stderr parsl.dataflow.dflow INFO: Task 4785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08e7cde3bda05d1ed8cd58057d840811 with result from task 4785 parsl.dataflow.dflow DEBUG: Task 4785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7b05a3d4-2fcf-4203-ac27-b8a0153769b8_isrForCrosstalkSources_855443_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7b05a3d4-2fcf-4203-ac27-b8a0153769b8_isrForCrosstalkSources_855443_47.stderr parsl.dataflow.dflow INFO: Task 4708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe7fc3789a73ad7941b5bff63b8ed8d6 with result from task 4708 parsl.dataflow.dflow DEBUG: Task 4708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/4b39c764-a09d-4d15-b392-858bc6668f2a_isrForCrosstalkSources_855448_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/4b39c764-a09d-4d15-b392-858bc6668f2a_isrForCrosstalkSources_855448_24.stderr parsl.dataflow.dflow INFO: Task 4702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e37d68effe300bfca1741ae4c0338470 with result from task 4702 parsl.dataflow.dflow DEBUG: Task 4702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/998ccdad-ec3c-4b01-8ea3-56b7d6ab9f95_isrForCrosstalkSources_855420_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/998ccdad-ec3c-4b01-8ea3-56b7d6ab9f95_isrForCrosstalkSources_855420_27.stderr parsl.dataflow.dflow INFO: Task 4777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0aee951340e2c9cdda9ec9585440bf with result from task 4777 parsl.dataflow.dflow DEBUG: Task 4777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bf47f90c-68d8-4d9f-b7ae-e1cb250e0c8b_isrForCrosstalkSources_855436_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/bf47f90c-68d8-4d9f-b7ae-e1cb250e0c8b_isrForCrosstalkSources_855436_47.stderr parsl.dataflow.dflow INFO: Task 4779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23d75ba69a1ba38ab439064bdd18290f with result from task 4779 parsl.dataflow.dflow DEBUG: Task 4779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cddf2464-9061-4127-8335-dec59197f070_isrForCrosstalkSources_855446_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/cddf2464-9061-4127-8335-dec59197f070_isrForCrosstalkSources_855446_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4639 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry bc62c25786846383baa7265c1e77640d with result from task 4639 parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4639 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/efe796fa-a243-411e-8c6d-18861b7a6d58_isrForCrosstalkSources_855413_30.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/efe796fa-a243-411e-8c6d-18861b7a6d58_isrForCrosstalkSources_855413_30.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:33 proc_lsst.multi:146 [INFO] found job 16964101 in provider checkpoint proc_lsst.multi INFO: found job 16964101 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 646fbd1cb24371a6cc315b602b1c1c02 with result from task 4820 parsl.dataflow.dflow DEBUG: Task 4820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/bb6b3c68-78e9-4060-b099-c671f27b77af_isrForCrosstalkSources_855413_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/bb6b3c68-78e9-4060-b099-c671f27b77af_isrForCrosstalkSources_855413_41.stderr parsl.dataflow.dflow INFO: Task 4751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2a5e97f6ced35db25b48a2b52045f1 with result from task 4751 parsl.dataflow.dflow DEBUG: Task 4751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/78fafebb-433b-47ef-9245-188cbf2a3976_isrForCrosstalkSources_855440_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/78fafebb-433b-47ef-9245-188cbf2a3976_isrForCrosstalkSources_855440_1.stderr parsl.dataflow.dflow INFO: Task 4808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae20abf0768e6d64cbeb8c9d5127b83 with result from task 4808 parsl.dataflow.dflow DEBUG: Task 4808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7057e2b2-f1bf-4b38-886f-536a148da0c1_isrForCrosstalkSources_855443_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/7057e2b2-f1bf-4b38-886f-536a148da0c1_isrForCrosstalkSources_855443_54.stderr parsl.dataflow.dflow INFO: Task 4723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 909d3957f564c310d0833fd16d1b8f2f with result from task 4723 parsl.dataflow.dflow DEBUG: Task 4723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/153a7c8d-9dfb-4d39-8a7b-240db1ca0e73_isrForCrosstalkSources_855397_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/153a7c8d-9dfb-4d39-8a7b-240db1ca0e73_isrForCrosstalkSources_855397_13.stderr parsl.dataflow.dflow INFO: Task 4748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5049b50e1542d5c93cb0a9a461cd2891 with result from task 4748 parsl.dataflow.dflow DEBUG: Task 4748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/284c6dcb-de7a-47f6-94c9-cb50dbb4392d_isrForCrosstalkSources_855429_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/284c6dcb-de7a-47f6-94c9-cb50dbb4392d_isrForCrosstalkSources_855429_53.stderr parsl.dataflow.dflow INFO: Task 4760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7fd97dd2c43fc2a694a10e6aeb8a9da with result from task 4760 parsl.dataflow.dflow DEBUG: Task 4760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e29968fd-342f-4607-aa7c-4eb5fe4e48e5_isrForCrosstalkSources_855403_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/e29968fd-342f-4607-aa7c-4eb5fe4e48e5_isrForCrosstalkSources_855403_41.stderr parsl.dataflow.dflow INFO: Task 4937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f2e342eba714b91c9046fdc49f199b1 with result from task 4937 parsl.dataflow.dflow DEBUG: Task 4937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/98bf6c0c-c2fc-438e-a3f1-bd9ab9fae140_isrForCrosstalkSources_855396_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/98bf6c0c-c2fc-438e-a3f1-bd9ab9fae140_isrForCrosstalkSources_855396_4.stderr parsl.dataflow.dflow INFO: Task 4918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68600accc86407b3582d5b0d8c0b6185 with result from task 4918 parsl.dataflow.dflow DEBUG: Task 4918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0c3bf50a-a500-4289-9e0f-fc3b579ae90e_isrForCrosstalkSources_855458_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/0c3bf50a-a500-4289-9e0f-fc3b579ae90e_isrForCrosstalkSources_855458_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:34 proc_lsst.multi:146 [INFO] found job 16964102 in provider checkpoint proc_lsst.multi INFO: found job 16964102 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 3800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfef7d89c75d2ed091b909dcbf780f94 with result from task 3800 parsl.dataflow.dflow DEBUG: Task 3800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/07114559-394e-46d1-bfc2-998b30af29b2_isrForCrosstalkSources_855411_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/07114559-394e-46d1-bfc2-998b30af29b2_isrForCrosstalkSources_855411_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:34 proc_lsst.multi:146 [INFO] found job 16964103 in provider checkpoint proc_lsst.multi INFO: found job 16964103 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 3799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e96b0adbbd554b7fb77e44b763f5c6 with result from task 3799 parsl.dataflow.dflow DEBUG: Task 3799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f78e8ad2-4cc4-4f9f-bf7a-c2da06d9e304_isrForCrosstalkSources_855431_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/f78e8ad2-4cc4-4f9f-bf7a-c2da06d9e304_isrForCrosstalkSources_855431_18.stderr parsl.dataflow.dflow INFO: Task 3806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a03252983ab240d44fd85e5d1706b535 with result from task 3806 parsl.dataflow.dflow DEBUG: Task 3806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e923f0e8-97df-4366-82b5-94043665c4f5_isrForCrosstalkSources_855431_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e923f0e8-97df-4366-82b5-94043665c4f5_isrForCrosstalkSources_855431_37.stderr parsl.dataflow.dflow INFO: Task 4922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc3454d478c22082eec6d22d3917c9a with result from task 4922 parsl.dataflow.dflow DEBUG: Task 4922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8fed1919-3876-472f-935c-a4862782c3df_isrForCrosstalkSources_855402_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8fed1919-3876-472f-935c-a4862782c3df_isrForCrosstalkSources_855402_22.stderr parsl.dataflow.dflow INFO: Task 4678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fc3a7ff3428cea6eb51c32ff29af3d7 with result from task 4678 parsl.dataflow.dflow DEBUG: Task 4678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6a635d25-faa5-42aa-b2d8-767caa22e1f3_isrForCrosstalkSources_855395_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855395/6a635d25-faa5-42aa-b2d8-767caa22e1f3_isrForCrosstalkSources_855395_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:34 proc_lsst.multi:146 [INFO] found job 16964104 in provider checkpoint proc_lsst.multi INFO: found job 16964104 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdce8b5024ad354905c685ce2563d8b5 with result from task 4930 parsl.dataflow.dflow DEBUG: Task 4930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5fc6a04c-60e0-4f4a-98fe-288705230346_isrForCrosstalkSources_855427_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/5fc6a04c-60e0-4f4a-98fe-288705230346_isrForCrosstalkSources_855427_41.stderr parsl.dataflow.dflow INFO: Task 4944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8040b158119134df9691f9998da48e4 with result from task 4944 parsl.dataflow.dflow DEBUG: Task 4944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b17f5560-fe2e-4474-b170-7dc2fdc1da3d_isrForCrosstalkSources_855438_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b17f5560-fe2e-4474-b170-7dc2fdc1da3d_isrForCrosstalkSources_855438_23.stderr parsl.dataflow.dflow INFO: Task 4928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf11202a07fb1c9d2f94fe5c1253c20 with result from task 4928 parsl.dataflow.dflow DEBUG: Task 4928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1e889bae-85e9-44ef-a4a1-aeac2e265628_isrForCrosstalkSources_855423_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/1e889bae-85e9-44ef-a4a1-aeac2e265628_isrForCrosstalkSources_855423_7.stderr parsl.dataflow.dflow INFO: Task 4925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63d6c7c04c0faef2c9984c71d5431770 with result from task 4925 parsl.dataflow.dflow DEBUG: Task 4925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9a1ef7a8-8cd7-467b-b79f-3d3addebc993_isrForCrosstalkSources_855393_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855393/9a1ef7a8-8cd7-467b-b79f-3d3addebc993_isrForCrosstalkSources_855393_37.stderr parsl.dataflow.dflow INFO: Task 4932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8a181c9559636233ad5ec70245fa35c with result from task 4932 parsl.dataflow.dflow DEBUG: Task 4932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35cd9a08-4763-42c3-93a0-00ec65a3e796_isrForCrosstalkSources_855385_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/35cd9a08-4763-42c3-93a0-00ec65a3e796_isrForCrosstalkSources_855385_28.stderr parsl.dataflow.dflow INFO: Task 4973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fed7f4ff9b9df7e8f23df8ed963726d with result from task 4973 parsl.dataflow.dflow DEBUG: Task 4973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eda1ee8d-432a-4a7e-b031-8887f3792a98_isrForCrosstalkSources_855446_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/eda1ee8d-432a-4a7e-b031-8887f3792a98_isrForCrosstalkSources_855446_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:34 proc_lsst.multi:146 [INFO] found job 16964105 in provider checkpoint proc_lsst.multi INFO: found job 16964105 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f77ca5eed3196ae593e0710a01e0ce3 with result from task 4931 parsl.dataflow.dflow DEBUG: Task 4931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1327d6fc-3bd0-4c49-8ba3-292432a91b64_isrForCrosstalkSources_855446_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/1327d6fc-3bd0-4c49-8ba3-292432a91b64_isrForCrosstalkSources_855446_30.stderr parsl.dataflow.dflow INFO: Task 4969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2888ff1e287824f1dc7ee4781c9ef9cd with result from task 4969 parsl.dataflow.dflow DEBUG: Task 4969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5d9f9848-060e-4325-b5db-65e656616570_isrForCrosstalkSources_855405_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/5d9f9848-060e-4325-b5db-65e656616570_isrForCrosstalkSources_855405_54.stderr parsl.dataflow.dflow INFO: Task 4964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267f9ad37ec0793ab061ab5878ab25ff with result from task 4964 parsl.dataflow.dflow DEBUG: Task 4964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1e09d5e2-7996-4e9e-9878-42fd0fce099b_isrForCrosstalkSources_855385_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/1e09d5e2-7996-4e9e-9878-42fd0fce099b_isrForCrosstalkSources_855385_12.stderr parsl.dataflow.dflow INFO: Task 4938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cfc59a8dfbcd48911199c8ca6415fd3 with result from task 4938 parsl.dataflow.dflow DEBUG: Task 4938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/81c58a58-5817-4556-9f6d-cbd55256530e_isrForCrosstalkSources_855456_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/81c58a58-5817-4556-9f6d-cbd55256530e_isrForCrosstalkSources_855456_35.stderr parsl.dataflow.dflow INFO: Task 4656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 348335a860e48a0f24a8ffd73fa671e7 with result from task 4656 parsl.dataflow.dflow DEBUG: Task 4656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c9fa249a-c0db-4e3a-875d-295fcfbb84d0_isrForCrosstalkSources_855412_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/c9fa249a-c0db-4e3a-875d-295fcfbb84d0_isrForCrosstalkSources_855412_27.stderr parsl.dataflow.dflow INFO: Task 4943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78ec3c8eff7e50e5d39f4eb88d70785c with result from task 4943 parsl.dataflow.dflow DEBUG: Task 4943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/33a047ff-dbff-4dc9-8048-a47535aa0fad_isrForCrosstalkSources_855409_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/33a047ff-dbff-4dc9-8048-a47535aa0fad_isrForCrosstalkSources_855409_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3802 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 4f176934ea529f0f454b7e5a61cace74 with result from task 3802 parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3802 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3a7851cd-8928-4942-9a38-4f6700939632_isrForCrosstalkSources_855394_51.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/3a7851cd-8928-4942-9a38-4f6700939632_isrForCrosstalkSources_855394_51.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:35 proc_lsst.multi:146 [INFO] found job 16964106 in provider checkpoint proc_lsst.multi INFO: found job 16964106 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:35 proc_lsst.multi:146 [INFO] found job 16964107 in provider checkpoint proc_lsst.multi INFO: found job 16964107 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:35 proc_lsst.multi:146 [INFO] found job 16964108 in provider checkpoint proc_lsst.multi INFO: found job 16964108 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:35 proc_lsst.multi:146 [INFO] found job 16964109 in provider checkpoint proc_lsst.multi INFO: found job 16964109 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:36 proc_lsst.multi:146 [INFO] found job 16964110 in provider checkpoint proc_lsst.multi INFO: found job 16964110 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:36 proc_lsst.multi:146 [INFO] found job 16964111 in provider checkpoint proc_lsst.multi INFO: found job 16964111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:36 proc_lsst.multi:146 [INFO] found job 16964112 in provider checkpoint proc_lsst.multi INFO: found job 16964112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:36 proc_lsst.multi:146 [INFO] found job 16964113 in provider checkpoint proc_lsst.multi INFO: found job 16964113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:37 proc_lsst.multi:146 [INFO] found job 16964114 in provider checkpoint proc_lsst.multi INFO: found job 16964114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f5ccb4fcc8223b9b5d55a440e1dbc11 with result from task 4846 parsl.dataflow.dflow DEBUG: Task 4846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/1c51dc82-fe7b-4985-b3b1-bfae6f7daf5d_isrForCrosstalkSources_855438_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/1c51dc82-fe7b-4985-b3b1-bfae6f7daf5d_isrForCrosstalkSources_855438_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:37 proc_lsst.multi:146 [INFO] found job 16964115 in provider checkpoint proc_lsst.multi INFO: found job 16964115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:37 proc_lsst.multi:146 [INFO] found job 16964116 in provider checkpoint proc_lsst.multi INFO: found job 16964116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:37 proc_lsst.multi:146 [INFO] found job 16964117 in provider checkpoint proc_lsst.multi INFO: found job 16964117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:38 proc_lsst.multi:146 [INFO] found job 16964118 in provider checkpoint proc_lsst.multi INFO: found job 16964118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b05c375e1eeae4587e65b3d0313bebb with result from task 4848 parsl.dataflow.dflow DEBUG: Task 4848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8b640d39-889d-4bf3-9ff0-c2c178a11bea_isrForCrosstalkSources_855454_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/8b640d39-889d-4bf3-9ff0-c2c178a11bea_isrForCrosstalkSources_855454_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:38 proc_lsst.multi:146 [INFO] found job 16964119 in provider checkpoint proc_lsst.multi INFO: found job 16964119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:38 proc_lsst.multi:146 [INFO] found job 16964120 in provider checkpoint proc_lsst.multi INFO: found job 16964120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:38 proc_lsst.multi:146 [INFO] found job 16964121 in provider checkpoint proc_lsst.multi INFO: found job 16964121 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:39 proc_lsst.multi:146 [INFO] found job 16964122 in provider checkpoint proc_lsst.multi INFO: found job 16964122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:39 proc_lsst.multi:146 [INFO] found job 16964123 in provider checkpoint proc_lsst.multi INFO: found job 16964123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:39 proc_lsst.multi:146 [INFO] found job 16964124 in provider checkpoint proc_lsst.multi INFO: found job 16964124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:39 proc_lsst.multi:146 [INFO] found job 16964125 in provider checkpoint proc_lsst.multi INFO: found job 16964125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed69b04c49a2a0d9e1731a7a2d086537 with result from task 4239 parsl.dataflow.dflow DEBUG: Task 4239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c66ca99f-56fa-461b-8c6e-8f867db9ba96_isrForCrosstalkSources_855419_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/c66ca99f-56fa-461b-8c6e-8f867db9ba96_isrForCrosstalkSources_855419_4.stderr parsl.dataflow.dflow INFO: Task 4236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65639e1414363aca36a9e9c8472d2c41 with result from task 4236 parsl.dataflow.dflow DEBUG: Task 4236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/042f088d-014e-4784-93dd-db56ffc2af7b_isrForCrosstalkSources_855442_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/042f088d-014e-4784-93dd-db56ffc2af7b_isrForCrosstalkSources_855442_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:40 proc_lsst.multi:146 [INFO] found job 16964126 in provider checkpoint proc_lsst.multi INFO: found job 16964126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:40 proc_lsst.multi:146 [INFO] found job 16964127 in provider checkpoint proc_lsst.multi INFO: found job 16964127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:40 proc_lsst.multi:146 [INFO] found job 16964128 in provider checkpoint proc_lsst.multi INFO: found job 16964128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:40 proc_lsst.multi:146 [INFO] found job 16964129 in provider checkpoint proc_lsst.multi INFO: found job 16964129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:41 proc_lsst.multi:146 [INFO] found job 16964130 in provider checkpoint proc_lsst.multi INFO: found job 16964130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:41 proc_lsst.multi:146 [INFO] found job 16964131 in provider checkpoint proc_lsst.multi INFO: found job 16964131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:41 proc_lsst.multi:146 [INFO] found job 16964132 in provider checkpoint proc_lsst.multi INFO: found job 16964132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:41 proc_lsst.multi:146 [INFO] found job 16964133 in provider checkpoint proc_lsst.multi INFO: found job 16964133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:42 proc_lsst.multi:146 [INFO] found job 16964134 in provider checkpoint proc_lsst.multi INFO: found job 16964134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 4762 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry c8aa9f3ca56d93ed6da29792cf986891 with result from task 4762 parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 4762 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 4762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/14375599-9e48-4ef1-b8cc-58c7e4cb01d5_isrForCrosstalkSources_855437_25.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 4762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/14375599-9e48-4ef1-b8cc-58c7e4cb01d5_isrForCrosstalkSources_855437_25.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:42 proc_lsst.multi:146 [INFO] found job 16964135 in provider checkpoint proc_lsst.multi INFO: found job 16964135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:42 proc_lsst.multi:146 [INFO] found job 16964136 in provider checkpoint proc_lsst.multi INFO: found job 16964136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:42 proc_lsst.multi:146 [INFO] found job 16964137 in provider checkpoint proc_lsst.multi INFO: found job 16964137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c4c2ce0c4ca4fd89db7fc8de66c364d with result from task 4619 parsl.dataflow.dflow DEBUG: Task 4619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cd7d818a-aa5f-48b0-bb4e-669077c6b061_isrForCrosstalkSources_855456_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/cd7d818a-aa5f-48b0-bb4e-669077c6b061_isrForCrosstalkSources_855456_11.stderr parsl.dataflow.dflow INFO: Task 4663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42c5460f44342d51fe100a2296d9a13e with result from task 4663 parsl.dataflow.dflow DEBUG: Task 4663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3cc05831-1bbb-49db-b63f-24f003d7281f_isrForCrosstalkSources_855453_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/3cc05831-1bbb-49db-b63f-24f003d7281f_isrForCrosstalkSources_855453_53.stderr parsl.dataflow.dflow INFO: Task 4685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80e872923dcc81da2a1d57c181bbb5e with result from task 4685 parsl.dataflow.dflow DEBUG: Task 4685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/27f8924c-c861-4c9e-8cba-7e9a70671276_isrForCrosstalkSources_855399_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/27f8924c-c861-4c9e-8cba-7e9a70671276_isrForCrosstalkSources_855399_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4662 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 934ff2a8a88348c828273307979e4c94 with result from task 4662 parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4662 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4f780375-ff6d-4384-9152-c721eb455acb_isrForCrosstalkSources_855383_19.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/4f780375-ff6d-4384-9152-c721eb455acb_isrForCrosstalkSources_855383_19.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4704 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 9180a7a3829cbce28b8eb106a8551a3e with result from task 4704 parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4704 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3ee6dcf3-deaf-4b11-8eb2-58e6ea911f34_isrForCrosstalkSources_855385_36.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3ee6dcf3-deaf-4b11-8eb2-58e6ea911f34_isrForCrosstalkSources_855385_36.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:43 proc_lsst.multi:146 [INFO] found job 16964138 in provider checkpoint proc_lsst.multi INFO: found job 16964138 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a04075c9c8b01d06bda124f371ec22 with result from task 4747 parsl.dataflow.dflow DEBUG: Task 4747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/655c6307-501a-464c-b168-dee4bf2bc661_isrForCrosstalkSources_855417_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/655c6307-501a-464c-b168-dee4bf2bc661_isrForCrosstalkSources_855417_17.stderr parsl.dataflow.dflow INFO: Task 4657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28ffec2de7ab61506bea05c1a57f828b with result from task 4657 parsl.dataflow.dflow DEBUG: Task 4657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4445a8fd-2e1c-49a7-97e4-3ae46aa6a991_isrForCrosstalkSources_855400_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/4445a8fd-2e1c-49a7-97e4-3ae46aa6a991_isrForCrosstalkSources_855400_31.stderr parsl.dataflow.dflow INFO: Task 4780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3be9829bfd54a6110a37149bb979778 with result from task 4780 parsl.dataflow.dflow DEBUG: Task 4780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9bdb76d8-abad-49fa-9ef0-ef3164f7a9ea_isrForCrosstalkSources_855401_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/9bdb76d8-abad-49fa-9ef0-ef3164f7a9ea_isrForCrosstalkSources_855401_10.stderr parsl.dataflow.dflow INFO: Task 4919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c7ae4c5a58d07fc701dbcccace2f94 with result from task 4919 parsl.dataflow.dflow DEBUG: Task 4919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/932b6ee8-0b58-490d-bc15-7338e41934cf_isrForCrosstalkSources_855417_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/932b6ee8-0b58-490d-bc15-7338e41934cf_isrForCrosstalkSources_855417_60.stderr parsl.dataflow.dflow INFO: Task 4788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57ae48299d8cb85501ad01a3ac67aa40 with result from task 4788 parsl.dataflow.dflow DEBUG: Task 4788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87a19571-fbfe-4e0f-ad6a-6196b898897c_isrForCrosstalkSources_855391_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/87a19571-fbfe-4e0f-ad6a-6196b898897c_isrForCrosstalkSources_855391_18.stderr parsl.dataflow.dflow INFO: Task 4847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bf20d67db78c17450a4c05e8fbfa5d4 with result from task 4847 parsl.dataflow.dflow DEBUG: Task 4847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/fb84c2e8-1e93-4a6a-889b-c31533122803_isrForCrosstalkSources_855431_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/fb84c2e8-1e93-4a6a-889b-c31533122803_isrForCrosstalkSources_855431_12.stderr parsl.dataflow.dflow INFO: Task 4309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 099356d4d2bb8e39a85c67d68ba13fcf with result from task 4309 parsl.dataflow.dflow DEBUG: Task 4309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d251d0f6-0c9b-45fc-a314-822d4fc3bd77_isrForCrosstalkSources_855432_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/d251d0f6-0c9b-45fc-a314-822d4fc3bd77_isrForCrosstalkSources_855432_33.stderr parsl.dataflow.dflow INFO: Task 4539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26e2a0acc319c53965351ad7793f19fe with result from task 4539 parsl.dataflow.dflow DEBUG: Task 4539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b7eab635-c02e-4026-a973-6ace5dc7f23d_isrForCrosstalkSources_855432_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/b7eab635-c02e-4026-a973-6ace5dc7f23d_isrForCrosstalkSources_855432_30.stderr parsl.dataflow.dflow INFO: Task 4783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95968c009789943876fe8c9e1df356cf with result from task 4783 parsl.dataflow.dflow DEBUG: Task 4783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/18b5910c-f001-4f5c-9740-eff1ce85ff04_isrForCrosstalkSources_855410_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/18b5910c-f001-4f5c-9740-eff1ce85ff04_isrForCrosstalkSources_855410_20.stderr parsl.dataflow.dflow INFO: Task 4778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1b430abc879973b106aa92f8b97460 with result from task 4778 parsl.dataflow.dflow DEBUG: Task 4778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e351af5e-9fde-465c-8134-0d855436c69f_isrForCrosstalkSources_855444_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855444/e351af5e-9fde-465c-8134-0d855436c69f_isrForCrosstalkSources_855444_60.stderr parsl.dataflow.dflow INFO: Task 4790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57963a6cb5f7f0ad789b08cb3c34e09a with result from task 4790 parsl.dataflow.dflow DEBUG: Task 4790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a920e287-17be-448e-b52d-617bf6e7d586_isrForCrosstalkSources_855415_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a920e287-17be-448e-b52d-617bf6e7d586_isrForCrosstalkSources_855415_49.stderr parsl.dataflow.dflow INFO: Task 4947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972a139eb64f8ce1c2b1df425846fb4f with result from task 4947 parsl.dataflow.dflow DEBUG: Task 4947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fc4473d2-a88d-4d81-a0a1-3d0801d811d6_isrForCrosstalkSources_855454_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/fc4473d2-a88d-4d81-a0a1-3d0801d811d6_isrForCrosstalkSources_855454_6.stderr parsl.dataflow.dflow INFO: Task 4802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b87e87bf9c8892cd53ff27daf398483 with result from task 4802 parsl.dataflow.dflow DEBUG: Task 4802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e84c2e37-784e-492b-ac6a-8b6e55c030d8_isrForCrosstalkSources_855413_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/e84c2e37-784e-492b-ac6a-8b6e55c030d8_isrForCrosstalkSources_855413_38.stderr parsl.dataflow.dflow INFO: Task 4924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b09062fde37b5802fda2f22ab3f7b780 with result from task 4924 parsl.dataflow.dflow DEBUG: Task 4924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/0ee8b3aa-c50c-485d-939e-117c975ebaa2_isrForCrosstalkSources_855415_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/0ee8b3aa-c50c-485d-939e-117c975ebaa2_isrForCrosstalkSources_855415_57.stderr parsl.dataflow.dflow INFO: Task 4830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feeb96b3e1474a900248ceddcdafe018 with result from task 4830 parsl.dataflow.dflow DEBUG: Task 4830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a943de96-93a7-40c5-87b7-7df9bc49f4d1_isrForCrosstalkSources_855421_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/a943de96-93a7-40c5-87b7-7df9bc49f4d1_isrForCrosstalkSources_855421_57.stderr parsl.dataflow.dflow INFO: Task 4771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b6846cd91764eb07d21d4abded05db with result from task 4771 parsl.dataflow.dflow DEBUG: Task 4771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/49c4f3a5-0e13-4773-b61a-6fb8f35cbdde_isrForCrosstalkSources_855446_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/49c4f3a5-0e13-4773-b61a-6fb8f35cbdde_isrForCrosstalkSources_855446_62.stderr parsl.dataflow.dflow INFO: Task 4071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a3ab87220fa77b4c86be2f6d64077f9 with result from task 4071 parsl.dataflow.dflow DEBUG: Task 4071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e7b4080b-5b7d-4539-906d-4c762b653d70_isrForCrosstalkSources_855445_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/e7b4080b-5b7d-4539-906d-4c762b653d70_isrForCrosstalkSources_855445_55.stderr parsl.dataflow.dflow INFO: Task 4886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59ba171e1328bba792ceb575b3a1e31d with result from task 4886 parsl.dataflow.dflow DEBUG: Task 4886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/af125d11-db53-4883-b147-ba67a946b86d_isrForCrosstalkSources_855383_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/af125d11-db53-4883-b147-ba67a946b86d_isrForCrosstalkSources_855383_25.stderr parsl.dataflow.dflow INFO: Task 4806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d22a48a021f8aee082b35d28f7ab612d with result from task 4806 parsl.dataflow.dflow DEBUG: Task 4806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8a77af0a-d7a2-42e7-9296-90f0ac83e32b_isrForCrosstalkSources_855402_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/8a77af0a-d7a2-42e7-9296-90f0ac83e32b_isrForCrosstalkSources_855402_43.stderr parsl.dataflow.dflow INFO: Task 4814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b91e2db83e4324378f8d9722464a5965 with result from task 4814 parsl.dataflow.dflow DEBUG: Task 4814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/fa33bbda-8026-4520-8f3d-8d536b745175_isrForCrosstalkSources_855389_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/fa33bbda-8026-4520-8f3d-8d536b745175_isrForCrosstalkSources_855389_45.stderr parsl.dataflow.dflow INFO: Task 4753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fe431d64d33e4b8dd740e23afe0b5f0 with result from task 4753 parsl.dataflow.dflow DEBUG: Task 4753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2216a23d-4240-4de7-9ac3-766f242e89e2_isrForCrosstalkSources_855404_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/2216a23d-4240-4de7-9ac3-766f242e89e2_isrForCrosstalkSources_855404_61.stderr parsl.dataflow.dflow INFO: Task 4805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aed7fb909af191b17606e05dd78803a with result from task 4805 parsl.dataflow.dflow DEBUG: Task 4805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b97a20-c821-4512-8428-15f4a60c51cf_isrForCrosstalkSources_855416_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/72b97a20-c821-4512-8428-15f4a60c51cf_isrForCrosstalkSources_855416_36.stderr parsl.dataflow.dflow INFO: Task 4072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67cacd64a69b15e1e780d19a654800fb with result from task 4072 parsl.dataflow.dflow DEBUG: Task 4072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/28efe684-2215-4b5c-bbb1-156541818ca3_isrForCrosstalkSources_855435_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/28efe684-2215-4b5c-bbb1-156541818ca3_isrForCrosstalkSources_855435_17.stderr parsl.dataflow.dflow INFO: Task 4076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af955014a0b5a37604c60a017b063018 with result from task 4076 parsl.dataflow.dflow DEBUG: Task 4076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a2301f6e-4bb2-4ea0-a4bb-a6c542937272_isrForCrosstalkSources_855442_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/a2301f6e-4bb2-4ea0-a4bb-a6c542937272_isrForCrosstalkSources_855442_28.stderr parsl.dataflow.dflow INFO: Task 4074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01be05dd176847e719a4a900bc52edf9 with result from task 4074 parsl.dataflow.dflow DEBUG: Task 4074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f68bef45-6c86-40c6-a4a9-bc07e16a6126_isrForCrosstalkSources_855407_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/f68bef45-6c86-40c6-a4a9-bc07e16a6126_isrForCrosstalkSources_855407_33.stderr parsl.dataflow.dflow INFO: Task 4912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c82ac4f89f01fc7f2d241cb99fda994 with result from task 4912 parsl.dataflow.dflow DEBUG: Task 4912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/795d3cf5-b8f6-4bf5-a8f8-b0362fe41d80_isrForCrosstalkSources_855417_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/795d3cf5-b8f6-4bf5-a8f8-b0362fe41d80_isrForCrosstalkSources_855417_44.stderr parsl.dataflow.dflow INFO: Task 4942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a1fe6e0ceae2ac30968eb40ad535c0 with result from task 4942 parsl.dataflow.dflow DEBUG: Task 4942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ac6928fe-2708-43c6-a90c-bde55426dfa1_isrForCrosstalkSources_855399_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/ac6928fe-2708-43c6-a90c-bde55426dfa1_isrForCrosstalkSources_855399_24.stderr parsl.dataflow.dflow INFO: Task 4078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfeae2c6982fcde2adc91e6f4fb3af7b with result from task 4078 parsl.dataflow.dflow DEBUG: Task 4078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b319fd85-a851-463f-99d5-98b1a6d9b596_isrForCrosstalkSources_855407_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/b319fd85-a851-463f-99d5-98b1a6d9b596_isrForCrosstalkSources_855407_13.stderr parsl.dataflow.dflow INFO: Task 4077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2aa5364d31e673e2842e53a51c5265 with result from task 4077 parsl.dataflow.dflow DEBUG: Task 4077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e33099c8-ad64-4886-84b8-8bc86df0233d_isrForCrosstalkSources_855438_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/e33099c8-ad64-4886-84b8-8bc86df0233d_isrForCrosstalkSources_855438_41.stderr parsl.dataflow.dflow INFO: Task 4756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19d9f80e2d9a16a0374a8a5763dfe8f6 with result from task 4756 parsl.dataflow.dflow DEBUG: Task 4756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/77ca3153-1aa2-4a7d-a9f8-b4dcc73f253f_isrForCrosstalkSources_855423_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/77ca3153-1aa2-4a7d-a9f8-b4dcc73f253f_isrForCrosstalkSources_855423_57.stderr parsl.dataflow.dflow INFO: Task 3803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 311b45230269869cf404c08ed410e901 with result from task 3803 parsl.dataflow.dflow DEBUG: Task 3803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/2e00787b-1950-4275-a669-3e90a156cc9f_isrForCrosstalkSources_855383_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/2e00787b-1950-4275-a669-3e90a156cc9f_isrForCrosstalkSources_855383_52.stderr parsl.dataflow.dflow INFO: Task 4075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a511acf4b34f774e70c8c085463ee22 with result from task 4075 parsl.dataflow.dflow DEBUG: Task 4075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5bed13c9-6bd7-4759-bdb0-38cd36362cb7_isrForCrosstalkSources_855455_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5bed13c9-6bd7-4759-bdb0-38cd36362cb7_isrForCrosstalkSources_855455_22.stderr parsl.dataflow.dflow INFO: Task 4941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcce2f13405da39f2606ffc5561202f5 with result from task 4941 parsl.dataflow.dflow DEBUG: Task 4941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fcbf673a-f943-4309-a91e-f1b16086e56a_isrForCrosstalkSources_855450_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/fcbf673a-f943-4309-a91e-f1b16086e56a_isrForCrosstalkSources_855450_17.stderr parsl.dataflow.dflow INFO: Task 4242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c896ad7eb86f77c3836a528cd4708823 with result from task 4242 parsl.dataflow.dflow DEBUG: Task 4242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fee46111-90bd-4f0f-b0bc-2e2bc73b72f8_isrForCrosstalkSources_855394_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/fee46111-90bd-4f0f-b0bc-2e2bc73b72f8_isrForCrosstalkSources_855394_45.stderr parsl.dataflow.dflow INFO: Task 4241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 857420df1793cf93c3239e276c3e6755 with result from task 4241 parsl.dataflow.dflow DEBUG: Task 4241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/86198d71-f86b-48cb-b2cd-de604c9e252a_isrForCrosstalkSources_855399_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855399/86198d71-f86b-48cb-b2cd-de604c9e252a_isrForCrosstalkSources_855399_21.stderr parsl.dataflow.dflow INFO: Task 4935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43755f9b110b66435e9e8f80c73c1f38 with result from task 4935 parsl.dataflow.dflow DEBUG: Task 4935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6a23d830-d0f9-4bee-93cf-3a15dd6a9e3b_isrForCrosstalkSources_855379_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/6a23d830-d0f9-4bee-93cf-3a15dd6a9e3b_isrForCrosstalkSources_855379_58.stderr parsl.dataflow.dflow INFO: Task 4240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb0309392c9b818b16cd0ce7ba1c3fbf with result from task 4240 parsl.dataflow.dflow DEBUG: Task 4240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/176a52eb-25d5-4120-b611-5e4b0cda629b_isrForCrosstalkSources_855411_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/176a52eb-25d5-4120-b611-5e4b0cda629b_isrForCrosstalkSources_855411_5.stderr parsl.dataflow.dflow INFO: Task 4936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea64453bc5977322af9b05bb60febac1 with result from task 4936 parsl.dataflow.dflow DEBUG: Task 4936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99c6f6d4-2617-42e0-a919-7e1db4efe6ae_isrForCrosstalkSources_855406_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/99c6f6d4-2617-42e0-a919-7e1db4efe6ae_isrForCrosstalkSources_855406_50.stderr parsl.dataflow.dflow INFO: Task 4745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5ca98115cb55560165088b2722b33ad with result from task 4745 parsl.dataflow.dflow DEBUG: Task 4745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f86193d9-17b6-4af9-bb84-40ce67dba0c6_isrForCrosstalkSources_855425_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f86193d9-17b6-4af9-bb84-40ce67dba0c6_isrForCrosstalkSources_855425_38.stderr parsl.dataflow.dflow INFO: Task 4703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ab97a28696e78cc4e1ae84406e304ee with result from task 4703 parsl.dataflow.dflow DEBUG: Task 4703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ef237ae4-174b-432e-92ce-7b7369325e1a_isrForCrosstalkSources_855396_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/ef237ae4-174b-432e-92ce-7b7369325e1a_isrForCrosstalkSources_855396_12.stderr parsl.dataflow.dflow INFO: Task 4815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a17958819d9bdb6581201030e4709d29 with result from task 4815 parsl.dataflow.dflow DEBUG: Task 4815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5d0a8f94-d95f-408b-957e-6912c5fbf651_isrForCrosstalkSources_855435_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/5d0a8f94-d95f-408b-957e-6912c5fbf651_isrForCrosstalkSources_855435_39.stderr parsl.dataflow.dflow INFO: Task 4797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb9bdf3f429e4eb6ead5cda636bb1130 with result from task 4797 parsl.dataflow.dflow DEBUG: Task 4797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a2578705-575d-462c-9156-06b6682148c1_isrForCrosstalkSources_855411_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/a2578705-575d-462c-9156-06b6682148c1_isrForCrosstalkSources_855411_29.stderr parsl.dataflow.dflow INFO: Task 4712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05809c3af84b2a68caccea5a319bc63f with result from task 4712 parsl.dataflow.dflow DEBUG: Task 4712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/72c02f0c-19de-4b6e-9e5b-33d5064a9c73_isrForCrosstalkSources_855415_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/72c02f0c-19de-4b6e-9e5b-33d5064a9c73_isrForCrosstalkSources_855415_18.stderr parsl.dataflow.dflow INFO: Task 4904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 833b492fcec8e92554980aaf96fc1a43 with result from task 4904 parsl.dataflow.dflow DEBUG: Task 4904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9721d82c-bc19-450e-85cc-70d01aebd479_isrForCrosstalkSources_855414_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/9721d82c-bc19-450e-85cc-70d01aebd479_isrForCrosstalkSources_855414_4.stderr parsl.dataflow.dflow INFO: Task 4900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11c3a8e3395c0e767aa5b65232b41dca with result from task 4900 parsl.dataflow.dflow DEBUG: Task 4900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8d9496db-a647-4b77-921f-38c731e9a11f_isrForCrosstalkSources_855425_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8d9496db-a647-4b77-921f-38c731e9a11f_isrForCrosstalkSources_855425_41.stderr parsl.dataflow.dflow INFO: Task 4625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a409c3e07c3257e55b8bdb8988222c65 with result from task 4625 parsl.dataflow.dflow DEBUG: Task 4625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6ee943e9-49f9-42c3-a5a5-16c14afe5ffd_isrForCrosstalkSources_855432_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/6ee943e9-49f9-42c3-a5a5-16c14afe5ffd_isrForCrosstalkSources_855432_14.stderr parsl.dataflow.dflow INFO: Task 4890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4225ba58d2d5d5d16af39fca00c5b51c with result from task 4890 parsl.dataflow.dflow DEBUG: Task 4890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7028b4f7-b823-446a-a26c-053aa2b8d961_isrForCrosstalkSources_855437_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/7028b4f7-b823-446a-a26c-053aa2b8d961_isrForCrosstalkSources_855437_49.stderr parsl.dataflow.dflow INFO: Task 4905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e74cd6e10303032cf385d1d1a400b64 with result from task 4905 parsl.dataflow.dflow DEBUG: Task 4905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/08157e45-c397-49bc-ba61-e2d7a71daf36_isrForCrosstalkSources_855409_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/08157e45-c397-49bc-ba61-e2d7a71daf36_isrForCrosstalkSources_855409_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:43 proc_lsst.multi:146 [INFO] found job 16964139 in provider checkpoint proc_lsst.multi INFO: found job 16964139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78cd1af0d8fc635292112f8b078b3b0c with result from task 4073 parsl.dataflow.dflow DEBUG: Task 4073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/547dec9d-29ef-41c9-ab31-7b21e92c5933_isrForCrosstalkSources_855384_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/547dec9d-29ef-41c9-ab31-7b21e92c5933_isrForCrosstalkSources_855384_29.stderr parsl.dataflow.dflow INFO: Task 4644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25b7f6c7bde9055fd9170a6bf0d645e with result from task 4644 parsl.dataflow.dflow DEBUG: Task 4644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/04002a52-aff6-420d-bbf0-090cb89fe646_isrForCrosstalkSources_855389_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/04002a52-aff6-420d-bbf0-090cb89fe646_isrForCrosstalkSources_855389_59.stderr parsl.dataflow.dflow INFO: Task 4804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54f5b4c96e629d7579a1d4abd0b6ac53 with result from task 4804 parsl.dataflow.dflow DEBUG: Task 4804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/904dc1d1-2aaa-47ab-8482-d78fafe7bcf6_isrForCrosstalkSources_855424_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/904dc1d1-2aaa-47ab-8482-d78fafe7bcf6_isrForCrosstalkSources_855424_28.stderr parsl.dataflow.dflow INFO: Task 4903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca083509a5215038def2b9b33babd532 with result from task 4903 parsl.dataflow.dflow DEBUG: Task 4903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/b37fcca0-4a69-4d0c-b5c4-361a4797147b_isrForCrosstalkSources_855404_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/b37fcca0-4a69-4d0c-b5c4-361a4797147b_isrForCrosstalkSources_855404_3.stderr parsl.dataflow.dflow INFO: Task 4652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48bcf6396fdd9d287c93d6d204048dea with result from task 4652 parsl.dataflow.dflow DEBUG: Task 4652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9e4056c3-822e-495b-b8c0-d3f482cdabc8_isrForCrosstalkSources_855400_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/9e4056c3-822e-495b-b8c0-d3f482cdabc8_isrForCrosstalkSources_855400_48.stderr parsl.dataflow.dflow INFO: Task 4238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 094e9c5f9ec078f2e039906496668bb0 with result from task 4238 parsl.dataflow.dflow DEBUG: Task 4238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/40ed2dca-8f20-4152-a44a-b46cd965e0fa_isrForCrosstalkSources_855406_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/40ed2dca-8f20-4152-a44a-b46cd965e0fa_isrForCrosstalkSources_855406_17.stderr parsl.dataflow.dflow INFO: Task 4243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7721c021b4ca5e83c0695387cb755f8 with result from task 4243 parsl.dataflow.dflow DEBUG: Task 4243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a1f2528-db5d-4431-93e2-06cd386d4a0f_isrForCrosstalkSources_855410_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/6a1f2528-db5d-4431-93e2-06cd386d4a0f_isrForCrosstalkSources_855410_41.stderr parsl.dataflow.dflow INFO: Task 4908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8845bae49298e2df5c9daa79fecbd466 with result from task 4908 parsl.dataflow.dflow DEBUG: Task 4908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/59fe27f0-7d4e-4967-b41d-4f42830831e3_isrForCrosstalkSources_855408_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/59fe27f0-7d4e-4967-b41d-4f42830831e3_isrForCrosstalkSources_855408_37.stderr parsl.dataflow.dflow INFO: Task 4754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a94265a49c39522d77c458ed68dc96b1 with result from task 4754 parsl.dataflow.dflow DEBUG: Task 4754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2f03f5ea-ef85-4e4e-8f1e-7a3405bf86c2_isrForCrosstalkSources_855414_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/2f03f5ea-ef85-4e4e-8f1e-7a3405bf86c2_isrForCrosstalkSources_855414_29.stderr parsl.dataflow.dflow INFO: Task 4729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28cbba24a5085b68982b79bef5b14852 with result from task 4729 parsl.dataflow.dflow DEBUG: Task 4729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6175362e-402f-441f-a610-4c3ddecfd751_isrForCrosstalkSources_855424_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6175362e-402f-441f-a610-4c3ddecfd751_isrForCrosstalkSources_855424_10.stderr parsl.dataflow.dflow INFO: Task 4786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b07457dee8279eaa237aea1c6b4144b with result from task 4786 parsl.dataflow.dflow DEBUG: Task 4786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bad94c10-60d9-43a1-a51b-839155a8dcd0_isrForCrosstalkSources_855397_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/bad94c10-60d9-43a1-a51b-839155a8dcd0_isrForCrosstalkSources_855397_55.stderr parsl.dataflow.dflow INFO: Task 4910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75eea4c6075a7bcc494f568d62cb49ab with result from task 4910 parsl.dataflow.dflow DEBUG: Task 4910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0ef66155-4149-4439-babf-57bfbf8fdb71_isrForCrosstalkSources_855450_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/0ef66155-4149-4439-babf-57bfbf8fdb71_isrForCrosstalkSources_855450_25.stderr parsl.dataflow.dflow INFO: Task 4854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7467cc20c012bd8780ac75190c746d8 with result from task 4854 parsl.dataflow.dflow DEBUG: Task 4854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ec0c2573-7ea1-4e51-919b-03d1074f090e_isrForCrosstalkSources_855397_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/ec0c2573-7ea1-4e51-919b-03d1074f090e_isrForCrosstalkSources_855397_27.stderr parsl.dataflow.dflow INFO: Task 4898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff63b87837728dac60297e75cd7e8c2a with result from task 4898 parsl.dataflow.dflow DEBUG: Task 4898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c4c85315-660a-4137-9570-2b3f26caf300_isrForCrosstalkSources_855434_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c4c85315-660a-4137-9570-2b3f26caf300_isrForCrosstalkSources_855434_58.stderr parsl.dataflow.dflow INFO: Task 4888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f73b355584d9a97cfc6e41a2719f1f1 with result from task 4888 parsl.dataflow.dflow DEBUG: Task 4888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9c7fb924-82fe-4742-a58c-470c8396c28f_isrForCrosstalkSources_855434_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/9c7fb924-82fe-4742-a58c-470c8396c28f_isrForCrosstalkSources_855434_42.stderr parsl.dataflow.dflow INFO: Task 4813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e11532200b6ee04cc5dbc1516aef5d32 with result from task 4813 parsl.dataflow.dflow DEBUG: Task 4813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/b1dfd0a1-d28d-4283-8970-50b9c4b1e66a_isrForCrosstalkSources_855405_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/b1dfd0a1-d28d-4283-8970-50b9c4b1e66a_isrForCrosstalkSources_855405_49.stderr parsl.dataflow.dflow INFO: Task 4614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9508e3ea3b3813af562ccfa61bd2d9f9 with result from task 4614 parsl.dataflow.dflow DEBUG: Task 4614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/82b7d3ab-c68a-4e2c-9997-b8e8d2d2843f_isrForCrosstalkSources_855432_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/82b7d3ab-c68a-4e2c-9997-b8e8d2d2843f_isrForCrosstalkSources_855432_42.stderr parsl.dataflow.dflow INFO: Task 4801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45fd60354a6b3cc070445793cf477b75 with result from task 4801 parsl.dataflow.dflow DEBUG: Task 4801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/928a2314-0805-4b14-ae43-21cdbcd6a19f_isrForCrosstalkSources_855385_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/928a2314-0805-4b14-ae43-21cdbcd6a19f_isrForCrosstalkSources_855385_48.stderr parsl.dataflow.dflow INFO: Task 4716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0857047cbe66430d32ebf92ab4e80322 with result from task 4716 parsl.dataflow.dflow DEBUG: Task 4716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/94d6c420-ca6f-4157-a234-0ff9caf76d45_isrForCrosstalkSources_855414_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/94d6c420-ca6f-4157-a234-0ff9caf76d45_isrForCrosstalkSources_855414_35.stderr parsl.dataflow.dflow INFO: Task 4792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aab04ff3ab1bff0677969fe12b39031 with result from task 4792 parsl.dataflow.dflow DEBUG: Task 4792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/56e3ed84-ac76-407f-a777-b1dd0b6ff621_isrForCrosstalkSources_855385_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/56e3ed84-ac76-407f-a777-b1dd0b6ff621_isrForCrosstalkSources_855385_9.stderr parsl.dataflow.dflow INFO: Task 4896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e2a6f4e5eb436ba8f5f2103061016de with result from task 4896 parsl.dataflow.dflow DEBUG: Task 4896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/540eccec-71cf-4c35-90be-403396901b25_isrForCrosstalkSources_855424_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/540eccec-71cf-4c35-90be-403396901b25_isrForCrosstalkSources_855424_20.stderr parsl.dataflow.dflow INFO: Task 4907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfba593a60b4bc4994394d0e61ff9677 with result from task 4907 parsl.dataflow.dflow DEBUG: Task 4907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8f2265f4-4007-4522-9f50-a00a3cf9d3fa_isrForCrosstalkSources_855379_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/8f2265f4-4007-4522-9f50-a00a3cf9d3fa_isrForCrosstalkSources_855379_55.stderr parsl.dataflow.dflow INFO: Task 4897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 243969f0d9d8a32a721e7e48435ea493 with result from task 4897 parsl.dataflow.dflow DEBUG: Task 4897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/56c4f8c4-1e40-4169-baee-55ef48617184_isrForCrosstalkSources_855421_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/56c4f8c4-1e40-4169-baee-55ef48617184_isrForCrosstalkSources_855421_36.stderr parsl.dataflow.dflow INFO: Task 4889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40bc729924b330ca066911f73644e14a with result from task 4889 parsl.dataflow.dflow DEBUG: Task 4889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3931c9cc-9e3d-4ad7-a5a9-2cb49c2aa602_isrForCrosstalkSources_855383_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/3931c9cc-9e3d-4ad7-a5a9-2cb49c2aa602_isrForCrosstalkSources_855383_62.stderr parsl.dataflow.dflow INFO: Task 4799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff38acfd1f56a18d2ea6dde2fe2b37c5 with result from task 4799 parsl.dataflow.dflow DEBUG: Task 4799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/268ebce8-6fd0-4ddc-9620-d90ef3550b6a_isrForCrosstalkSources_855379_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855379/268ebce8-6fd0-4ddc-9620-d90ef3550b6a_isrForCrosstalkSources_855379_61.stderr parsl.dataflow.dflow INFO: Task 4837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7e596efb51be165dfb8986d08ce6271 with result from task 4837 parsl.dataflow.dflow DEBUG: Task 4837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/675dd014-1231-4671-ab1e-1956ec2947aa_isrForCrosstalkSources_855416_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/675dd014-1231-4671-ab1e-1956ec2947aa_isrForCrosstalkSources_855416_19.stderr parsl.dataflow.dflow INFO: Task 4741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a16a0cccd2336cb49b5a9a0a3b85b00 with result from task 4741 parsl.dataflow.dflow DEBUG: Task 4741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/49ad6525-c57f-46b2-a3a5-16dbe8c0c31a_isrForCrosstalkSources_855432_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/49ad6525-c57f-46b2-a3a5-16dbe8c0c31a_isrForCrosstalkSources_855432_52.stderr parsl.dataflow.dflow INFO: Task 4901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d8d0dbc0f6c515cba2264729c2f4781 with result from task 4901 parsl.dataflow.dflow DEBUG: Task 4901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ec4ae99a-1710-450b-a0a1-b329c4ad5471_isrForCrosstalkSources_855405_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/ec4ae99a-1710-450b-a0a1-b329c4ad5471_isrForCrosstalkSources_855405_42.stderr parsl.dataflow.dflow INFO: Task 4237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7336aeb22891015279f4e68af00090a3 with result from task 4237 parsl.dataflow.dflow DEBUG: Task 4237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9e6132f6-076b-422e-8155-b1061863da06_isrForCrosstalkSources_855432_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9e6132f6-076b-422e-8155-b1061863da06_isrForCrosstalkSources_855432_45.stderr parsl.dataflow.dflow INFO: Task 4926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f43c5dc9c10a6d9a32bcfccee416adf8 with result from task 4926 parsl.dataflow.dflow DEBUG: Task 4926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2b3111ba-7bd3-4d5e-be5e-3eb560f11298_isrForCrosstalkSources_855418_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/2b3111ba-7bd3-4d5e-be5e-3eb560f11298_isrForCrosstalkSources_855418_29.stderr parsl.dataflow.dflow INFO: Task 4902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f1e953f54a9b33de59e9a22ca12be51 with result from task 4902 parsl.dataflow.dflow DEBUG: Task 4902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/176ab8d5-d3d2-4d74-abb3-ce4e33a41161_isrForCrosstalkSources_855385_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/176ab8d5-d3d2-4d74-abb3-ce4e33a41161_isrForCrosstalkSources_855385_24.stderr parsl.dataflow.dflow INFO: Task 4653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f78b3885fd77cc39dae4ccd41217cbb with result from task 4653 parsl.dataflow.dflow DEBUG: Task 4653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e4569980-a41c-43dd-8a46-b9aa216357af_isrForCrosstalkSources_855431_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e4569980-a41c-43dd-8a46-b9aa216357af_isrForCrosstalkSources_855431_44.stderr parsl.dataflow.dflow INFO: Task 4899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f05bc40ace9583764d3581ad33fde588 with result from task 4899 parsl.dataflow.dflow DEBUG: Task 4899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4f014aec-db58-4476-ab0b-371cd054d934_isrForCrosstalkSources_855385_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/4f014aec-db58-4476-ab0b-371cd054d934_isrForCrosstalkSources_855385_61.stderr parsl.dataflow.dflow INFO: Task 4906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e17af8509dfd06e1836eff4179ac3ad with result from task 4906 parsl.dataflow.dflow DEBUG: Task 4906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cb24dbdc-78ad-4c88-a7ce-fb6574be18ac_isrForCrosstalkSources_855432_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cb24dbdc-78ad-4c88-a7ce-fb6574be18ac_isrForCrosstalkSources_855432_62.stderr parsl.dataflow.dflow INFO: Task 4838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7097673ef3a4f7e233f28510f1f73777 with result from task 4838 parsl.dataflow.dflow DEBUG: Task 4838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/4dcd3b5b-ad80-47ed-bfa4-668c01913bdb_isrForCrosstalkSources_855410_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/4dcd3b5b-ad80-47ed-bfa4-668c01913bdb_isrForCrosstalkSources_855410_8.stderr parsl.dataflow.dflow INFO: Task 4791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c9ca24eb0669998e637c6bb25a47b37 with result from task 4791 parsl.dataflow.dflow DEBUG: Task 4791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9163333f-4a2e-486f-ab49-7ef3c150368e_isrForCrosstalkSources_855432_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/9163333f-4a2e-486f-ab49-7ef3c150368e_isrForCrosstalkSources_855432_44.stderr parsl.dataflow.dflow INFO: Task 4807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3e7c0be2452d220a169a6639e7f5607 with result from task 4807 parsl.dataflow.dflow DEBUG: Task 4807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8039ae8c-8188-48fa-8a44-82503d533298_isrForCrosstalkSources_855397_23.stdout parsl.dataflow.dflow INFO: Standard error for task 4807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/8039ae8c-8188-48fa-8a44-82503d533298_isrForCrosstalkSources_855397_23.stderr parsl.dataflow.dflow INFO: Task 4887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e11b0fbca208d18883c8728bbe3bcbf with result from task 4887 parsl.dataflow.dflow DEBUG: Task 4887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/f89faf4c-e467-4be4-b40f-f2d5bd891d56_isrForCrosstalkSources_855389_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/f89faf4c-e467-4be4-b40f-f2d5bd891d56_isrForCrosstalkSources_855389_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 4841 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 23ea09925c900a9140ef25a79ec5b731 with result from task 4841 parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 4841 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:43 proc_lsst.multi:146 [INFO] found job 16964140 in provider checkpoint proc_lsst.multi INFO: found job 16964140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Standard output for task 4841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/207dc7c7-cca6-413c-b01b-3efd3b677184_isrForCrosstalkSources_855397_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/207dc7c7-cca6-413c-b01b-3efd3b677184_isrForCrosstalkSources_855397_22.stderr parsl.dataflow.dflow INFO: Task 4840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f3b9e43e339107560f48fddc0ce357c with result from task 4840 parsl.dataflow.dflow DEBUG: Task 4840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/93236832-ade0-4d80-850e-921856358f43_isrForCrosstalkSources_855405_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/93236832-ade0-4d80-850e-921856358f43_isrForCrosstalkSources_855405_38.stderr parsl.dataflow.dflow INFO: Task 4409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0174d700b497a750ddfbd5c7ece765 with result from task 4409 parsl.dataflow.dflow DEBUG: Task 4409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e5176301-dd6f-4e0a-be3a-dd1d9506ffd6_isrForCrosstalkSources_855432_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/e5176301-dd6f-4e0a-be3a-dd1d9506ffd6_isrForCrosstalkSources_855432_32.stderr parsl.dataflow.dflow INFO: Task 4863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19259ff49e5276c0a0114b42c0715036 with result from task 4863 parsl.dataflow.dflow DEBUG: Task 4863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0387494d-ddb3-4e49-bbdb-d728533e09b0_isrForCrosstalkSources_855408_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0387494d-ddb3-4e49-bbdb-d728533e09b0_isrForCrosstalkSources_855408_8.stderr parsl.dataflow.dflow INFO: Task 4843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39f9eb95ae6a68b03dc0f4ac7e48e91d with result from task 4843 parsl.dataflow.dflow DEBUG: Task 4843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e42025e-5015-40f8-aab1-df67ac0a739c_isrForCrosstalkSources_855384_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/5e42025e-5015-40f8-aab1-df67ac0a739c_isrForCrosstalkSources_855384_33.stderr parsl.dataflow.dflow INFO: Task 4803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c3b3629b269d7c6ca85f8d680a4913 with result from task 4803 parsl.dataflow.dflow DEBUG: Task 4803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/14df86af-5a16-4424-a589-05448afa5946_isrForCrosstalkSources_855432_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/14df86af-5a16-4424-a589-05448afa5946_isrForCrosstalkSources_855432_5.stderr parsl.dataflow.dflow INFO: Task 4833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9464d26e3752c17cb1e89a2f0e91a8e with result from task 4833 parsl.dataflow.dflow DEBUG: Task 4833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3002315-b602-4730-992d-4ba28eaac179_isrForCrosstalkSources_855431_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/c3002315-b602-4730-992d-4ba28eaac179_isrForCrosstalkSources_855431_27.stderr parsl.dataflow.dflow INFO: Task 4832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 094370421f092208d20396d773d74766 with result from task 4832 parsl.dataflow.dflow DEBUG: Task 4832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e3f8cd83-2128-4d5e-a6e3-4655ccf68ffc_isrForCrosstalkSources_855405_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/e3f8cd83-2128-4d5e-a6e3-4655ccf68ffc_isrForCrosstalkSources_855405_13.stderr parsl.dataflow.dflow INFO: Task 4849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ba17af13e84ebf6107a3ef1af4c11cd with result from task 4849 parsl.dataflow.dflow DEBUG: Task 4849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/550d8b55-464c-4a63-8c5a-e65cf29d287c_isrForCrosstalkSources_855405_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/550d8b55-464c-4a63-8c5a-e65cf29d287c_isrForCrosstalkSources_855405_34.stderr parsl.dataflow.dflow INFO: Task 4933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef64a953ae40416114e2bcfa8a5613b with result from task 4933 parsl.dataflow.dflow DEBUG: Task 4933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9b0d3e02-628b-4efc-bafb-b2ef045305b4_isrForCrosstalkSources_855452_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/9b0d3e02-628b-4efc-bafb-b2ef045305b4_isrForCrosstalkSources_855452_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:43 proc_lsst.multi:146 [INFO] found job 16964141 in provider checkpoint proc_lsst.multi INFO: found job 16964141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f44f8def71617ae6af5264c6fb04735 with result from task 4736 parsl.dataflow.dflow DEBUG: Task 4736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/96ea71d6-fa6c-4c5c-9f4a-a02f77771581_isrForCrosstalkSources_855459_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/96ea71d6-fa6c-4c5c-9f4a-a02f77771581_isrForCrosstalkSources_855459_14.stderr parsl.dataflow.dflow INFO: Task 4939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d803974eac97cd386b3a707a43e978d5 with result from task 4939 parsl.dataflow.dflow DEBUG: Task 4939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/71f0216f-a407-40a9-93e3-2df632f27883_isrForCrosstalkSources_855418_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855418/71f0216f-a407-40a9-93e3-2df632f27883_isrForCrosstalkSources_855418_2.stderr parsl.dataflow.dflow INFO: Task 4774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 594ecc981aecf1cb4799c0de10e41077 with result from task 4774 parsl.dataflow.dflow DEBUG: Task 4774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3ffe4e7e-8e93-4281-b77b-13104e7d6586_isrForCrosstalkSources_855426_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/3ffe4e7e-8e93-4281-b77b-13104e7d6586_isrForCrosstalkSources_855426_37.stderr parsl.dataflow.dflow INFO: Task 4929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f51e3db88a1715037e17274e8e54cd7 with result from task 4929 parsl.dataflow.dflow DEBUG: Task 4929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/6b36c139-aeea-4981-aa02-48d61c31b189_isrForCrosstalkSources_855381_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/6b36c139-aeea-4981-aa02-48d61c31b189_isrForCrosstalkSources_855381_62.stderr parsl.dataflow.dflow INFO: Task 4740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65ccd3675103178f015c9817ac3a784a with result from task 4740 parsl.dataflow.dflow DEBUG: Task 4740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d28547bb-440c-4296-89ee-164a13a6e3d9_isrForCrosstalkSources_855385_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/d28547bb-440c-4296-89ee-164a13a6e3d9_isrForCrosstalkSources_855385_51.stderr parsl.dataflow.dflow INFO: Task 4811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d999aacb55c85f2fbcbee516c2dabbb1 with result from task 4811 parsl.dataflow.dflow DEBUG: Task 4811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2b0472c0-6964-4def-b744-dfce1b7d8b2a_isrForCrosstalkSources_855411_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/2b0472c0-6964-4def-b744-dfce1b7d8b2a_isrForCrosstalkSources_855411_46.stderr parsl.dataflow.dflow INFO: Task 4914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcc7acb95410f412e3ba0f11afe3ffef with result from task 4914 parsl.dataflow.dflow DEBUG: Task 4914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/97dda977-a30d-4e06-a339-3f4888a58cf8_isrForCrosstalkSources_855437_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/97dda977-a30d-4e06-a339-3f4888a58cf8_isrForCrosstalkSources_855437_7.stderr parsl.dataflow.dflow INFO: Task 4757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16f916f4c8b7527307b919117812bcd9 with result from task 4757 parsl.dataflow.dflow DEBUG: Task 4757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/64f01096-c70b-44cc-8b90-36afd6ced2d2_isrForCrosstalkSources_855441_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/64f01096-c70b-44cc-8b90-36afd6ced2d2_isrForCrosstalkSources_855441_26.stderr parsl.dataflow.dflow INFO: Task 4720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119ab10641d3affe705011285a660f2f with result from task 4720 parsl.dataflow.dflow DEBUG: Task 4720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0d4198be-f81e-4bfc-9157-073163a29853_isrForCrosstalkSources_855437_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/0d4198be-f81e-4bfc-9157-073163a29853_isrForCrosstalkSources_855437_47.stderr parsl.dataflow.dflow INFO: Task 4690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 279b71074a4073947518f4ce76d66743 with result from task 4690 parsl.dataflow.dflow DEBUG: Task 4690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6eabccf3-00f7-4b0e-8327-202e56e19f07_isrForCrosstalkSources_855424_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/6eabccf3-00f7-4b0e-8327-202e56e19f07_isrForCrosstalkSources_855424_13.stderr parsl.dataflow.dflow INFO: Task 4705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca1f3001553fdf11c34d4b214999de89 with result from task 4705 parsl.dataflow.dflow DEBUG: Task 4705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/160489f9-4340-470e-ba27-d8e1d6183f58_isrForCrosstalkSources_855401_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/160489f9-4340-470e-ba27-d8e1d6183f58_isrForCrosstalkSources_855401_18.stderr parsl.dataflow.dflow INFO: Task 4769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fbef40c8985788c3b32b57830d13ed8 with result from task 4769 parsl.dataflow.dflow DEBUG: Task 4769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a515206-5286-4be0-8911-e471a79f9a75_isrForCrosstalkSources_855425_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/3a515206-5286-4be0-8911-e471a79f9a75_isrForCrosstalkSources_855425_6.stderr parsl.dataflow.dflow INFO: Task 4698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f58bc25b8a729ea9d616530715976db5 with result from task 4698 parsl.dataflow.dflow DEBUG: Task 4698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/750ec061-4251-4a56-897d-9a53fdbea4e0_isrForCrosstalkSources_855391_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/750ec061-4251-4a56-897d-9a53fdbea4e0_isrForCrosstalkSources_855391_16.stderr parsl.dataflow.dflow INFO: Task 4713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd3f3e42cc1bfa3eda3b41ec1f937476 with result from task 4713 parsl.dataflow.dflow DEBUG: Task 4713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/2964a1dc-13ff-44d8-8716-28a9fcc8d138_isrForCrosstalkSources_855405_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/2964a1dc-13ff-44d8-8716-28a9fcc8d138_isrForCrosstalkSources_855405_30.stderr parsl.dataflow.dflow INFO: Task 4781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2f08a4ab4c711e40c867ada88dc6f55 with result from task 4781 parsl.dataflow.dflow DEBUG: Task 4781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06b807fe-d709-4da4-bf0d-335d1f2e8cc1_isrForCrosstalkSources_855454_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/06b807fe-d709-4da4-bf0d-335d1f2e8cc1_isrForCrosstalkSources_855454_48.stderr parsl.dataflow.dflow INFO: Task 4739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca0403ee34655d8895339fd446eb9a9 with result from task 4739 parsl.dataflow.dflow DEBUG: Task 4739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/bb357eef-6291-4336-9cd3-c1a57be7ded7_isrForCrosstalkSources_855407_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/bb357eef-6291-4336-9cd3-c1a57be7ded7_isrForCrosstalkSources_855407_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4735 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry d963d6dfd36e835fb30c5313a43f1a2e with result from task 4735 parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4735 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/24a12a1a-52ff-4537-a7d6-de91c4674ebf_isrForCrosstalkSources_855451_5.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/24a12a1a-52ff-4537-a7d6-de91c4674ebf_isrForCrosstalkSources_855451_5.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 4787 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry ab28fc9e691678b14c015adfd01a7618 with result from task 4787 parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 4787 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 4787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/02e1f6b7-3a17-4ea1-86fe-bd7eebae0504_isrForCrosstalkSources_855420_62.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 4787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/02e1f6b7-3a17-4ea1-86fe-bd7eebae0504_isrForCrosstalkSources_855420_62.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:44 proc_lsst.multi:146 [INFO] found job 16964142 in provider checkpoint proc_lsst.multi INFO: found job 16964142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d2f05f603a9842f97e5d1b35f39c178 with result from task 4755 parsl.dataflow.dflow DEBUG: Task 4755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f7cbe2fb-fc8b-4193-9207-26dd8416ec6b_isrForCrosstalkSources_855383_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/f7cbe2fb-fc8b-4193-9207-26dd8416ec6b_isrForCrosstalkSources_855383_34.stderr parsl.dataflow.dflow INFO: Task 4770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f800fc662b48177271f5d7e2a470eba with result from task 4770 parsl.dataflow.dflow DEBUG: Task 4770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/d15c433e-86db-4d45-8b55-c274e1450f2b_isrForCrosstalkSources_855391_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/d15c433e-86db-4d45-8b55-c274e1450f2b_isrForCrosstalkSources_855391_53.stderr parsl.dataflow.dflow INFO: Task 4796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b2aa7acd2dbd2b030b506f506433b33 with result from task 4796 parsl.dataflow.dflow DEBUG: Task 4796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9bf9aeb3-a653-485b-85fd-640220de3745_isrForCrosstalkSources_855409_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/9bf9aeb3-a653-485b-85fd-640220de3745_isrForCrosstalkSources_855409_7.stderr parsl.dataflow.dflow INFO: Task 4773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 533dda52002e33bc2ec9cd148a411bcd with result from task 4773 parsl.dataflow.dflow DEBUG: Task 4773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/39aebcfd-8410-4a65-8a80-789c2d54a661_isrForCrosstalkSources_855409_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/39aebcfd-8410-4a65-8a80-789c2d54a661_isrForCrosstalkSources_855409_29.stderr parsl.dataflow.dflow INFO: Task 4851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1eb86627aa89c12e0b2e6e9359b87c with result from task 4851 parsl.dataflow.dflow DEBUG: Task 4851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c6aede6-cb46-4ab4-8186-e0fb38d7a603_isrForCrosstalkSources_855458_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c6aede6-cb46-4ab4-8186-e0fb38d7a603_isrForCrosstalkSources_855458_43.stderr parsl.dataflow.dflow INFO: Task 4882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42df77d99efb55e917fe8eaa6d3b5b33 with result from task 4882 parsl.dataflow.dflow DEBUG: Task 4882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e95edb8-8136-49e5-9d40-1b7fea75ad5b_isrForCrosstalkSources_855449_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/5e95edb8-8136-49e5-9d40-1b7fea75ad5b_isrForCrosstalkSources_855449_38.stderr parsl.dataflow.dflow INFO: Task 4569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811f3997172e63e589348368588a2a6a with result from task 4569 parsl.dataflow.dflow DEBUG: Task 4569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ea90c68e-217b-4241-9df2-ffffa0a9e273_isrForCrosstalkSources_855431_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/ea90c68e-217b-4241-9df2-ffffa0a9e273_isrForCrosstalkSources_855431_38.stderr parsl.dataflow.dflow INFO: Task 4883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7efecde660bacbd1c86ad42cff0acf with result from task 4883 parsl.dataflow.dflow DEBUG: Task 4883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8bae7093-88f4-48f9-b56a-28fa786034b1_isrForCrosstalkSources_855443_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/8bae7093-88f4-48f9-b56a-28fa786034b1_isrForCrosstalkSources_855443_31.stderr parsl.dataflow.dflow INFO: Task 3805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10c48bedacfa66f2cf682f0b9de02b9f with result from task 3805 parsl.dataflow.dflow DEBUG: Task 3805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b930ed94-cf4a-42ff-af08-564080c44837_isrForCrosstalkSources_855427_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/b930ed94-cf4a-42ff-af08-564080c44837_isrForCrosstalkSources_855427_17.stderr parsl.dataflow.dflow INFO: Task 4866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 992a2bf9e12d8f81bed313182bcf4e99 with result from task 4866 parsl.dataflow.dflow DEBUG: Task 4866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/1a44e85d-8fa5-44a5-a1ab-fb9277d9d6d0_isrForCrosstalkSources_855436_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/1a44e85d-8fa5-44a5-a1ab-fb9277d9d6d0_isrForCrosstalkSources_855436_21.stderr parsl.dataflow.dflow INFO: Task 4750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cb604df777c37411411a96ab4a48778 with result from task 4750 parsl.dataflow.dflow DEBUG: Task 4750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/47ea28dd-9287-4e6e-8996-b493fd7e0580_isrForCrosstalkSources_855431_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/47ea28dd-9287-4e6e-8996-b493fd7e0580_isrForCrosstalkSources_855431_25.stderr parsl.dataflow.dflow INFO: Task 4855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f85b3d7d32bc4efefadd6844cae28d3 with result from task 4855 parsl.dataflow.dflow DEBUG: Task 4855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de607337-986f-49cd-a7ce-5fbaae50ad32_isrForCrosstalkSources_855459_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/de607337-986f-49cd-a7ce-5fbaae50ad32_isrForCrosstalkSources_855459_51.stderr parsl.dataflow.dflow INFO: Task 4948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f133cb89e550a7d0a3574cedb2e3bf4 with result from task 4948 parsl.dataflow.dflow DEBUG: Task 4948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f18f1e4b-6476-4cd9-b114-0d8710731725_isrForCrosstalkSources_855398_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/f18f1e4b-6476-4cd9-b114-0d8710731725_isrForCrosstalkSources_855398_38.stderr parsl.dataflow.dflow INFO: Task 4857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3f24f4d3964f91ddede3477ee9fa785 with result from task 4857 parsl.dataflow.dflow DEBUG: Task 4857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/174d128c-4467-41fd-918d-00fa8d50ebf4_isrForCrosstalkSources_855388_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/174d128c-4467-41fd-918d-00fa8d50ebf4_isrForCrosstalkSources_855388_48.stderr parsl.dataflow.dflow INFO: Task 4738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cacc6f1781f75fd46b86e85ef94c759a with result from task 4738 parsl.dataflow.dflow DEBUG: Task 4738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6b3240dd-080e-45a5-b6ab-3af53c3e4b71_isrForCrosstalkSources_855397_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/6b3240dd-080e-45a5-b6ab-3af53c3e4b71_isrForCrosstalkSources_855397_12.stderr parsl.dataflow.dflow INFO: Task 4861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c206e94af31b8d030ee0a426608ae2c with result from task 4861 parsl.dataflow.dflow DEBUG: Task 4861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3c248f39-57dd-42e5-bd29-3f8635ff64b8_isrForCrosstalkSources_855445_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/3c248f39-57dd-42e5-bd29-3f8635ff64b8_isrForCrosstalkSources_855445_50.stderr parsl.dataflow.dflow INFO: Task 4853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43d581412f6d093660b85b2e4dc18346 with result from task 4853 parsl.dataflow.dflow DEBUG: Task 4853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6228ccb3-903b-4604-9d62-c9decdf2cb97_isrForCrosstalkSources_855404_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855404/6228ccb3-903b-4604-9d62-c9decdf2cb97_isrForCrosstalkSources_855404_28.stderr parsl.dataflow.dflow INFO: Task 4856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a209e51954cc4d634a7074c2829bb1 with result from task 4856 parsl.dataflow.dflow DEBUG: Task 4856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/427d879f-a56b-4172-9e21-a81656117038_isrForCrosstalkSources_855453_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/427d879f-a56b-4172-9e21-a81656117038_isrForCrosstalkSources_855453_7.stderr parsl.dataflow.dflow INFO: Task 4850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0006c2cf965c34c051ad32cf50016da9 with result from task 4850 parsl.dataflow.dflow DEBUG: Task 4850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/dc915129-1a86-47c7-9826-aa200d4abd3b_isrForCrosstalkSources_855420_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/dc915129-1a86-47c7-9826-aa200d4abd3b_isrForCrosstalkSources_855420_21.stderr parsl.dataflow.dflow INFO: Task 4920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed0d665c2e6ef999deda36122abe586 with result from task 4920 parsl.dataflow.dflow DEBUG: Task 4920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fd04b6a5-98f5-4379-8e88-3c8e3a5ee15e_isrForCrosstalkSources_855459_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/fd04b6a5-98f5-4379-8e88-3c8e3a5ee15e_isrForCrosstalkSources_855459_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4869 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry b9c6f238ac3aabcc15a040d3383025e3 with result from task 4869 parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4869 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/62621562-39ec-4122-8d9f-4525b9e372ec_isrForCrosstalkSources_855448_37.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/62621562-39ec-4122-8d9f-4525b9e372ec_isrForCrosstalkSources_855448_37.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4868 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 07891d812bfabea49b88f993556afe98 with result from task 4868 parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4868 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b42ec553-19e3-4583-ba85-dca42432cfca_isrForCrosstalkSources_855440_37.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/b42ec553-19e3-4583-ba85-dca42432cfca_isrForCrosstalkSources_855440_37.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:44 proc_lsst.multi:146 [INFO] found job 16964143 in provider checkpoint proc_lsst.multi INFO: found job 16964143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1ccd23d73430e5e78f37eee8c59caac with result from task 4893 parsl.dataflow.dflow DEBUG: Task 4893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/48f7dc44-0468-4903-a31b-86cab03ab4af_isrForCrosstalkSources_855433_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/48f7dc44-0468-4903-a31b-86cab03ab4af_isrForCrosstalkSources_855433_17.stderr parsl.dataflow.dflow INFO: Task 4875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffdac0c12acfd20ad1fd4467f1bf51ed with result from task 4875 parsl.dataflow.dflow DEBUG: Task 4875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/656f748e-7e10-4bc8-95b7-f5b10f2b1e15_isrForCrosstalkSources_855447_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855447/656f748e-7e10-4bc8-95b7-f5b10f2b1e15_isrForCrosstalkSources_855447_36.stderr parsl.dataflow.dflow INFO: Task 4878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1659f06434a5eb931cc03dc5dee33a with result from task 4878 parsl.dataflow.dflow DEBUG: Task 4878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a48a9352-467b-47e5-a087-a916917bbd29_isrForCrosstalkSources_855441_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/a48a9352-467b-47e5-a087-a916917bbd29_isrForCrosstalkSources_855441_2.stderr parsl.dataflow.dflow INFO: Task 4917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32a202ce82269cb724a3fe9d18f4a1b7 with result from task 4917 parsl.dataflow.dflow DEBUG: Task 4917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c5cf71e6-8f99-448a-ae3f-a6c8879358d4_isrForCrosstalkSources_855390_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/c5cf71e6-8f99-448a-ae3f-a6c8879358d4_isrForCrosstalkSources_855390_50.stderr parsl.dataflow.dflow INFO: Task 4867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c92434d519611d85eca6eb2c3cd761 with result from task 4867 parsl.dataflow.dflow DEBUG: Task 4867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/6a591308-efb9-47f8-8b0b-8d8f98143c26_isrForCrosstalkSources_855386_6.stdout parsl.dataflow.dflow INFO: Standard error for task 4867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/6a591308-efb9-47f8-8b0b-8d8f98143c26_isrForCrosstalkSources_855386_6.stderr parsl.dataflow.dflow INFO: Task 4877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfd9062e8e681b4b642e8037a7f93463 with result from task 4877 parsl.dataflow.dflow DEBUG: Task 4877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e1b91ad9-d940-4c5a-92a1-d61468b99972_isrForCrosstalkSources_855454_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/e1b91ad9-d940-4c5a-92a1-d61468b99972_isrForCrosstalkSources_855454_44.stderr parsl.dataflow.dflow INFO: Task 4858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f8e6f29dcd2e121ba47b8831cd0b55 with result from task 4858 parsl.dataflow.dflow DEBUG: Task 4858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e9041643-3869-46a4-8843-7d544381c689_isrForCrosstalkSources_855431_4.stdout parsl.dataflow.dflow INFO: Standard error for task 4858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855431/e9041643-3869-46a4-8843-7d544381c689_isrForCrosstalkSources_855431_4.stderr parsl.dataflow.dflow INFO: Task 4881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1583ae77361a6889f992ce4411eec1e5 with result from task 4881 parsl.dataflow.dflow DEBUG: Task 4881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/a137844f-c259-4477-9294-351324f7d8ba_isrForCrosstalkSources_855436_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/a137844f-c259-4477-9294-351324f7d8ba_isrForCrosstalkSources_855436_12.stderr parsl.dataflow.dflow INFO: Task 4909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c89269f8bc6155effd8f0eae570a24d with result from task 4909 parsl.dataflow.dflow DEBUG: Task 4909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ba97768e-b8ee-4329-9459-63af57c463bc_isrForCrosstalkSources_855451_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/ba97768e-b8ee-4329-9459-63af57c463bc_isrForCrosstalkSources_855451_7.stderr parsl.dataflow.dflow INFO: Task 4872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09984d48381e5e6ee30a0237e07a6b66 with result from task 4872 parsl.dataflow.dflow DEBUG: Task 4872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/08a1d86a-01d5-44dc-8bfd-6533d5d8557e_isrForCrosstalkSources_855442_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/08a1d86a-01d5-44dc-8bfd-6533d5d8557e_isrForCrosstalkSources_855442_8.stderr parsl.dataflow.dflow INFO: Task 4859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c64f6ebbb815b7c17699c636989305cd with result from task 4859 parsl.dataflow.dflow DEBUG: Task 4859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8c5d357c-1e0c-40f2-a21e-d46d7486f434_isrForCrosstalkSources_855427_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855427/8c5d357c-1e0c-40f2-a21e-d46d7486f434_isrForCrosstalkSources_855427_18.stderr parsl.dataflow.dflow INFO: Task 4870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15dd34003529e89493eb8d0b1f43f6b5 with result from task 4870 parsl.dataflow.dflow DEBUG: Task 4870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4eacd829-7b75-4c43-9fbc-523c015f99fd_isrForCrosstalkSources_855392_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/4eacd829-7b75-4c43-9fbc-523c015f99fd_isrForCrosstalkSources_855392_38.stderr parsl.dataflow.dflow INFO: Task 4880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0bfeec156c9a3921fbb83c7db67e8c with result from task 4880 parsl.dataflow.dflow DEBUG: Task 4880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/90fa3e93-c6a7-4b7c-a2ca-b6a30869a094_isrForCrosstalkSources_855394_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/90fa3e93-c6a7-4b7c-a2ca-b6a30869a094_isrForCrosstalkSources_855394_13.stderr parsl.dataflow.dflow INFO: Task 4911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb08fa4ed680f8c296e2989135c374cd with result from task 4911 parsl.dataflow.dflow DEBUG: Task 4911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a883641e-b10e-4b58-bc1f-7eb3b84d2256_isrForCrosstalkSources_855434_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/a883641e-b10e-4b58-bc1f-7eb3b84d2256_isrForCrosstalkSources_855434_29.stderr parsl.dataflow.dflow INFO: Task 4871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a38494c057bce01125f283e9ae487cee with result from task 4871 parsl.dataflow.dflow DEBUG: Task 4871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7d7f939d-a3ec-490f-961c-940d5b16b3be_isrForCrosstalkSources_855388_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/7d7f939d-a3ec-490f-961c-940d5b16b3be_isrForCrosstalkSources_855388_5.stderr parsl.dataflow.dflow INFO: Task 4879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a04ec3eb796d5858ce3b5e5722a25848 with result from task 4879 parsl.dataflow.dflow DEBUG: Task 4879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/45a4dbd4-67ca-49f4-ac07-8b0acddcace5_isrForCrosstalkSources_855407_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855407/45a4dbd4-67ca-49f4-ac07-8b0acddcace5_isrForCrosstalkSources_855407_18.stderr parsl.dataflow.dflow INFO: Task 4921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7a4eefa50764e061a48fcb233af0d6a with result from task 4921 parsl.dataflow.dflow DEBUG: Task 4921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36eb6f22-8242-4dba-809a-65f12afa8f8f_isrForCrosstalkSources_855390_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/36eb6f22-8242-4dba-809a-65f12afa8f8f_isrForCrosstalkSources_855390_52.stderr parsl.dataflow.dflow INFO: Task 4694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89e2fad4f0913ac00216c8c7a28c3a85 with result from task 4694 parsl.dataflow.dflow DEBUG: Task 4694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e10254b9-20e0-4c9e-adac-6d9885639274_isrForCrosstalkSources_855451_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e10254b9-20e0-4c9e-adac-6d9885639274_isrForCrosstalkSources_855451_47.stderr parsl.dataflow.dflow INFO: Task 4714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5d602de3b39b3d3afd8b146b7f821a6 with result from task 4714 parsl.dataflow.dflow DEBUG: Task 4714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/76f4bc9c-0fa9-456c-8cb7-363ffd29cc94_isrForCrosstalkSources_855451_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/76f4bc9c-0fa9-456c-8cb7-363ffd29cc94_isrForCrosstalkSources_855451_59.stderr parsl.dataflow.dflow INFO: Task 4894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa3d83b8b18fd7981d9dda6ad47bffe2 with result from task 4894 parsl.dataflow.dflow DEBUG: Task 4894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/06502dcb-30dd-4032-ae28-49fe3a30fb4c_isrForCrosstalkSources_855423_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/06502dcb-30dd-4032-ae28-49fe3a30fb4c_isrForCrosstalkSources_855423_38.stderr parsl.dataflow.dflow INFO: Task 4874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eec1cf14f13dba3eac56961481ecf2e with result from task 4874 parsl.dataflow.dflow DEBUG: Task 4874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32ace4ea-88b8-45a5-b826-d73dbf1c4f2e_isrForCrosstalkSources_855380_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855380/32ace4ea-88b8-45a5-b826-d73dbf1c4f2e_isrForCrosstalkSources_855380_9.stderr parsl.dataflow.dflow INFO: Task 4971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30663c1915a00c2c3882bdd9552c6ecd with result from task 4971 parsl.dataflow.dflow DEBUG: Task 4971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7c2d7901-4495-488e-8957-e41394acba2a_isrForCrosstalkSources_855396_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/7c2d7901-4495-488e-8957-e41394acba2a_isrForCrosstalkSources_855396_8.stderr parsl.dataflow.dflow INFO: Task 4451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f528ce6982f538117aa89db9f13ec6cb with result from task 4451 parsl.dataflow.dflow DEBUG: Task 4451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3cecf6ef-e8e2-4bda-8059-25af86f61286_isrForCrosstalkSources_855432_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3cecf6ef-e8e2-4bda-8059-25af86f61286_isrForCrosstalkSources_855432_3.stderr parsl.dataflow.dflow INFO: Task 4865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c216cced7afd91d854349b943e210050 with result from task 4865 parsl.dataflow.dflow DEBUG: Task 4865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/bd4ad25c-2e6b-4d8c-a713-50023a2e8f6d_isrForCrosstalkSources_855416_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/bd4ad25c-2e6b-4d8c-a713-50023a2e8f6d_isrForCrosstalkSources_855416_44.stderr parsl.dataflow.dflow INFO: Task 4945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d4cd0f1e79e7fc5204840502f45536 with result from task 4945 parsl.dataflow.dflow DEBUG: Task 4945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/60dba67e-46a4-431c-8173-c295fa18651b_isrForCrosstalkSources_855423_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/60dba67e-46a4-431c-8173-c295fa18651b_isrForCrosstalkSources_855423_3.stderr parsl.dataflow.dflow INFO: Task 4915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33bfd1db72715b674f3bf89c3a3a429e with result from task 4915 parsl.dataflow.dflow DEBUG: Task 4915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/aa63724e-7131-4086-a41b-25f8a3daf8f9_isrForCrosstalkSources_855406_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/aa63724e-7131-4086-a41b-25f8a3daf8f9_isrForCrosstalkSources_855406_28.stderr parsl.dataflow.dflow INFO: Task 4884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc3de497a6679df1864166e6d29363f0 with result from task 4884 parsl.dataflow.dflow DEBUG: Task 4884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3220b63e-d273-4c39-b115-d38c5d679ac9_isrForCrosstalkSources_855443_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/3220b63e-d273-4c39-b115-d38c5d679ac9_isrForCrosstalkSources_855443_5.stderr parsl.dataflow.dflow INFO: Task 4940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55c77082a4fd96ec57571780bd1af486 with result from task 4940 parsl.dataflow.dflow DEBUG: Task 4940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ae1542fd-8ccf-46e0-858e-4ad0930ac6be_isrForCrosstalkSources_855455_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/ae1542fd-8ccf-46e0-858e-4ad0930ac6be_isrForCrosstalkSources_855455_18.stderr parsl.dataflow.dflow INFO: Task 4860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 297b7e0cf07654e40dadfd8443d40095 with result from task 4860 parsl.dataflow.dflow DEBUG: Task 4860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/35499a07-7474-4ef8-9a46-6c7c408e041c_isrForCrosstalkSources_855459_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/35499a07-7474-4ef8-9a46-6c7c408e041c_isrForCrosstalkSources_855459_1.stderr parsl.dataflow.dflow INFO: Task 4852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1d915493a378a52fc055073c9954387 with result from task 4852 parsl.dataflow.dflow DEBUG: Task 4852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/04cfe93f-124d-4c79-ae6c-eb8e817b97f7_isrForCrosstalkSources_855405_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/04cfe93f-124d-4c79-ae6c-eb8e817b97f7_isrForCrosstalkSources_855405_2.stderr parsl.dataflow.dflow INFO: Task 4885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4793e3b35bd6028ca4912814622bda5 with result from task 4885 parsl.dataflow.dflow DEBUG: Task 4885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/170f4d37-5768-4792-ba89-a32f06a34019_isrForCrosstalkSources_855424_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/170f4d37-5768-4792-ba89-a32f06a34019_isrForCrosstalkSources_855424_41.stderr parsl.dataflow.dflow INFO: Task 4772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8068983328a65df6415785f15aa4e0a5 with result from task 4772 parsl.dataflow.dflow DEBUG: Task 4772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/0ed030bb-90df-4ca2-9ca4-b9d3abe1147f_isrForCrosstalkSources_855398_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855398/0ed030bb-90df-4ca2-9ca4-b9d3abe1147f_isrForCrosstalkSources_855398_35.stderr parsl.dataflow.dflow INFO: Task 4892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 593cf816c74681e2f600ef37a3e46543 with result from task 4892 parsl.dataflow.dflow DEBUG: Task 4892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/76911753-91d6-452b-b80a-f3e488295360_isrForCrosstalkSources_855450_57.stdout parsl.dataflow.dflow INFO: Standard error for task 4892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/76911753-91d6-452b-b80a-f3e488295360_isrForCrosstalkSources_855450_57.stderr parsl.dataflow.dflow INFO: Task 4916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d35e1619307caa1ab3c8c1b7bba1f660 with result from task 4916 parsl.dataflow.dflow DEBUG: Task 4916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e899da78-9c5c-43a5-aab9-0ee0df0a5ff1_isrForCrosstalkSources_855451_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/e899da78-9c5c-43a5-aab9-0ee0df0a5ff1_isrForCrosstalkSources_855451_11.stderr parsl.dataflow.dflow INFO: Task 4834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3721f53112c290377f0f81813dd28d with result from task 4834 parsl.dataflow.dflow DEBUG: Task 4834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2ed6c27f-5d05-401d-9cc4-7163af8e7a3b_isrForCrosstalkSources_855457_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/2ed6c27f-5d05-401d-9cc4-7163af8e7a3b_isrForCrosstalkSources_855457_49.stderr parsl.dataflow.dflow INFO: Task 4895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbb88b5f06c3538760a9dcda6e5369c with result from task 4895 parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4895 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/4c55e689-b11d-4112-8072-d9bd37d7f43d_isrForCrosstalkSources_855394_31.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/4c55e689-b11d-4112-8072-d9bd37d7f43d_isrForCrosstalkSources_855394_31.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4862 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 9f4ff2bff43eda9e8743a4bf7b8bb037 with result from task 4862 parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4862 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d5f62376-37a9-4d50-b717-eafd0c541142_isrForCrosstalkSources_855449_41.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855449/d5f62376-37a9-4d50-b717-eafd0c541142_isrForCrosstalkSources_855449_41.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4873 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 4425e437562be82012e44af66f294068 with result from task 4873 parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4873 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/41a2a586-06bb-42d3-b195-8cf4ef9465bb_isrForCrosstalkSources_855412_50.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/41a2a586-06bb-42d3-b195-8cf4ef9465bb_isrForCrosstalkSources_855412_50.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 4864 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:44 proc_lsst.multi:146 [INFO] found job 16964144 in provider checkpoint parsl.dataflow.memoization DEBUG: Storing app cache entry 03a299856a69a566d1d79bd4b22fbf0f with result from task 4864 proc_lsst.multi INFO: found job 16964144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow DEBUG: Task 4864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7ed1bf79-e04d-4864-8365-127a32beb8bd_isrForCrosstalkSources_855414_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/7ed1bf79-e04d-4864-8365-127a32beb8bd_isrForCrosstalkSources_855414_45.stderr parsl.dataflow.dflow INFO: Task 4891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7ac28ae3f3abdc9c877a3000787fff1 with result from task 4891 parsl.dataflow.dflow DEBUG: Task 4891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18be49f4-6a86-476f-a1da-c256ae754150_isrForCrosstalkSources_855443_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/18be49f4-6a86-476f-a1da-c256ae754150_isrForCrosstalkSources_855443_52.stderr parsl.dataflow.dflow INFO: Task 4839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941741b941863b7604be6d9d2cd7a875 with result from task 4839 parsl.dataflow.dflow DEBUG: Task 4839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/6f0db164-931e-4bae-870f-41cf75421dd9_isrForCrosstalkSources_855434_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/6f0db164-931e-4bae-870f-41cf75421dd9_isrForCrosstalkSources_855434_43.stderr parsl.dataflow.dflow INFO: Task 4822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcf4b88593c4a9a0d54423d3cfa6343c with result from task 4822 parsl.dataflow.dflow DEBUG: Task 4822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/34e76d03-47e6-400f-bc34-b7123bd07a6a_isrForCrosstalkSources_855438_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/34e76d03-47e6-400f-bc34-b7123bd07a6a_isrForCrosstalkSources_855438_48.stderr parsl.dataflow.dflow INFO: Task 4836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40549f0228708889cf9e58c92cab4a30 with result from task 4836 parsl.dataflow.dflow DEBUG: Task 4836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/446589b3-594c-4990-8d88-e200c3d32ea7_isrForCrosstalkSources_855420_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/446589b3-594c-4990-8d88-e200c3d32ea7_isrForCrosstalkSources_855420_61.stderr parsl.dataflow.dflow INFO: Task 4842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58156b105cd28f8b3f3199270a6d0c5 with result from task 4842 parsl.dataflow.dflow DEBUG: Task 4842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8ca73e5-8eb3-4fb0-9755-c9a161dd7db6_isrForCrosstalkSources_855441_48.stdout parsl.dataflow.dflow INFO: Standard error for task 4842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855441/f8ca73e5-8eb3-4fb0-9755-c9a161dd7db6_isrForCrosstalkSources_855441_48.stderr parsl.dataflow.dflow INFO: Task 4844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb1705fbd8d70e9af156a387d35ed17 with result from task 4844 parsl.dataflow.dflow DEBUG: Task 4844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01437f8b-e0aa-417f-9086-62bc231def8a_isrForCrosstalkSources_855433_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/01437f8b-e0aa-417f-9086-62bc231def8a_isrForCrosstalkSources_855433_41.stderr parsl.dataflow.dflow INFO: Task 4794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66f8bd1a2a03e1bf11d3760b63a8a21b with result from task 4794 parsl.dataflow.dflow DEBUG: Task 4794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1f589b30-fe65-40c2-a14d-ee947edaf419_isrForCrosstalkSources_855437_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/1f589b30-fe65-40c2-a14d-ee947edaf419_isrForCrosstalkSources_855437_38.stderr parsl.dataflow.dflow INFO: Task 4567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c7a6e6cfb0dbd457e0d82dc337ca0e0 with result from task 4567 parsl.dataflow.dflow DEBUG: Task 4567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2faa0fa5-11f1-485d-8e7f-9815c022b1b9_isrForCrosstalkSources_855429_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/2faa0fa5-11f1-485d-8e7f-9815c022b1b9_isrForCrosstalkSources_855429_59.stderr parsl.dataflow.dflow INFO: Task 4697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a79a9220cbd1627cf70a6a8fe0a6126 with result from task 4697 parsl.dataflow.dflow DEBUG: Task 4697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/33d58ec9-dd3e-4270-ac86-82dab582e41e_isrForCrosstalkSources_855389_9.stdout parsl.dataflow.dflow INFO: Standard error for task 4697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/33d58ec9-dd3e-4270-ac86-82dab582e41e_isrForCrosstalkSources_855389_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:44 proc_lsst.multi:146 [INFO] found job 16964145 in provider checkpoint proc_lsst.multi INFO: found job 16964145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e2c2a1fb3f008204d3e76f3ea7e3096 with result from task 4568 parsl.dataflow.dflow DEBUG: Task 4568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b4237304-2fd2-4ec0-b2d3-52ed6b72fe86_isrForCrosstalkSources_855452_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/b4237304-2fd2-4ec0-b2d3-52ed6b72fe86_isrForCrosstalkSources_855452_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:44 proc_lsst.multi:146 [INFO] found job 16964146 in provider checkpoint proc_lsst.multi INFO: found job 16964146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dae2f8d21840e88816fc106d51ab6ac with result from task 4325 parsl.dataflow.dflow DEBUG: Task 4325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3c3ba501-f18c-4c91-b2c4-68173f520963_isrForCrosstalkSources_855432_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/3c3ba501-f18c-4c91-b2c4-68173f520963_isrForCrosstalkSources_855432_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:45 proc_lsst.multi:146 [INFO] found job 16964147 in provider checkpoint proc_lsst.multi INFO: found job 16964147 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:45 proc_lsst.multi:146 [INFO] found job 16964148 in provider checkpoint proc_lsst.multi INFO: found job 16964148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:45 proc_lsst.multi:146 [INFO] found job 16964149 in provider checkpoint proc_lsst.multi INFO: found job 16964149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:45 proc_lsst.multi:146 [INFO] found job 16964150 in provider checkpoint proc_lsst.multi INFO: found job 16964150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4798 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry ebf6dbf694abb5dfff69134a3dd64726 with result from task 4798 parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4798 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/be6a3ad9-fa38-4a51-9d50-3f172a7e2798_isrForCrosstalkSources_855382_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855382/be6a3ad9-fa38-4a51-9d50-3f172a7e2798_isrForCrosstalkSources_855382_11.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:46 proc_lsst.multi:146 [INFO] found job 16964151 in provider checkpoint proc_lsst.multi INFO: found job 16964151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:46 proc_lsst.multi:146 [INFO] found job 16964152 in provider checkpoint proc_lsst.multi INFO: found job 16964152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:46 proc_lsst.multi:146 [INFO] found job 16964153 in provider checkpoint proc_lsst.multi INFO: found job 16964153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:46 proc_lsst.multi:146 [INFO] found job 16964154 in provider checkpoint proc_lsst.multi INFO: found job 16964154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4788c282838384e9f49480c360e04a6 with result from task 4775 parsl.dataflow.dflow DEBUG: Task 4775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/1356a133-82b3-461a-9130-f408f60ac8cf_isrForCrosstalkSources_855416_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/1356a133-82b3-461a-9130-f408f60ac8cf_isrForCrosstalkSources_855416_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:47 proc_lsst.multi:146 [INFO] found job 16964155 in provider checkpoint proc_lsst.multi INFO: found job 16964155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:47 proc_lsst.multi:146 [INFO] found job 16964156 in provider checkpoint proc_lsst.multi INFO: found job 16964156 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:47 proc_lsst.multi:146 [INFO] found job 16964157 in provider checkpoint proc_lsst.multi INFO: found job 16964157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0898bcd693bf6d665dd1626edabbfd7d with result from task 4711 parsl.dataflow.dflow DEBUG: Task 4711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/424d5b6f-d797-4de7-b7c7-a95899f2889e_isrForCrosstalkSources_855387_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855387/424d5b6f-d797-4de7-b7c7-a95899f2889e_isrForCrosstalkSources_855387_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:47 proc_lsst.multi:146 [INFO] found job 16964158 in provider checkpoint proc_lsst.multi INFO: found job 16964158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8637a79b249bcafdc43e1ae51f10b0 with result from task 4565 parsl.dataflow.dflow DEBUG: Task 4565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/113adcb8-8758-44b6-8d50-7c1dcb98cee5_isrForCrosstalkSources_855434_26.stdout parsl.dataflow.dflow INFO: Standard error for task 4565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/113adcb8-8758-44b6-8d50-7c1dcb98cee5_isrForCrosstalkSources_855434_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:48 proc_lsst.multi:146 [INFO] found job 16964159 in provider checkpoint proc_lsst.multi INFO: found job 16964159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:48 proc_lsst.multi:146 [INFO] found job 16964160 in provider checkpoint proc_lsst.multi INFO: found job 16964160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:48 proc_lsst.multi:146 [INFO] found job 16964161 in provider checkpoint proc_lsst.multi INFO: found job 16964161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:48 proc_lsst.multi:146 [INFO] found job 16964162 in provider checkpoint proc_lsst.multi INFO: found job 16964162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:49 proc_lsst.multi:146 [INFO] found job 16964163 in provider checkpoint proc_lsst.multi INFO: found job 16964163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:49 proc_lsst.multi:146 [INFO] found job 16964164 in provider checkpoint proc_lsst.multi INFO: found job 16964164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:49 proc_lsst.multi:146 [INFO] found job 16964165 in provider checkpoint proc_lsst.multi INFO: found job 16964165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9dea4fe47a49d08386f46ca27a18294 with result from task 4571 parsl.dataflow.dflow DEBUG: Task 4571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d934a907-d057-4883-85da-5835b5d73073_isrForCrosstalkSources_855450_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/d934a907-d057-4883-85da-5835b5d73073_isrForCrosstalkSources_855450_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:49 proc_lsst.multi:146 [INFO] found job 16964166 in provider checkpoint proc_lsst.multi INFO: found job 16964166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:50 proc_lsst.multi:146 [INFO] found job 16964167 in provider checkpoint proc_lsst.multi INFO: found job 16964167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:50 proc_lsst.multi:146 [INFO] found job 16964168 in provider checkpoint proc_lsst.multi INFO: found job 16964168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:50 proc_lsst.multi:146 [INFO] found job 16964169 in provider checkpoint proc_lsst.multi INFO: found job 16964169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:50 proc_lsst.multi:146 [INFO] found job 16964170 in provider checkpoint proc_lsst.multi INFO: found job 16964170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:51 proc_lsst.multi:146 [INFO] found job 16964171 in provider checkpoint proc_lsst.multi INFO: found job 16964171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:51 proc_lsst.multi:146 [INFO] found job 16964172 in provider checkpoint proc_lsst.multi INFO: found job 16964172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:51 proc_lsst.multi:146 [INFO] found job 16964173 in provider checkpoint proc_lsst.multi INFO: found job 16964173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:53 proc_lsst.multi:146 [INFO] found job 16964174 in provider checkpoint proc_lsst.multi INFO: found job 16964174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 689cd89a2bb49e766e75c6f7a5c2439e with result from task 4952 parsl.dataflow.dflow DEBUG: Task 4952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5edeae24-5e64-4a2e-b3a8-7869b74c2cb1_isrForCrosstalkSources_855436_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/5edeae24-5e64-4a2e-b3a8-7869b74c2cb1_isrForCrosstalkSources_855436_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:53 proc_lsst.multi:146 [INFO] found job 16964175 in provider checkpoint proc_lsst.multi INFO: found job 16964175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4c3044173bb7cfb7e8d092b2bfca68b with result from task 4974 parsl.dataflow.dflow DEBUG: Task 4974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/546b7446-f54f-4715-8b5c-066332464609_isrForCrosstalkSources_855408_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/546b7446-f54f-4715-8b5c-066332464609_isrForCrosstalkSources_855408_27.stderr parsl.dataflow.dflow INFO: Task 4968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1616478c7a0aec9a287c029910bb1a2a with result from task 4968 parsl.dataflow.dflow DEBUG: Task 4968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b791de74-c527-445f-9413-668c30c7db85_isrForCrosstalkSources_855438_32.stdout parsl.dataflow.dflow INFO: Standard error for task 4968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/b791de74-c527-445f-9413-668c30c7db85_isrForCrosstalkSources_855438_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:54 proc_lsst.multi:146 [INFO] found job 16964176 in provider checkpoint proc_lsst.multi INFO: found job 16964176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f143adbabc2aa1afa171709194841b with result from task 4953 parsl.dataflow.dflow DEBUG: Task 4953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c0d4430-316c-4a1f-be9e-66857a6dad38_isrForCrosstalkSources_855458_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/3c0d4430-316c-4a1f-be9e-66857a6dad38_isrForCrosstalkSources_855458_19.stderr parsl.dataflow.dflow INFO: Task 4957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97d028f3c632a306a5aea55164266b26 with result from task 4957 parsl.dataflow.dflow DEBUG: Task 4957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/545affee-9a8e-46d4-9dfc-42bb115bc79a_isrForCrosstalkSources_855390_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/545affee-9a8e-46d4-9dfc-42bb115bc79a_isrForCrosstalkSources_855390_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:55 proc_lsst.multi:146 [INFO] found job 16964177 in provider checkpoint proc_lsst.multi INFO: found job 16964177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:55 proc_lsst.multi:146 [INFO] found job 16964178 in provider checkpoint proc_lsst.multi INFO: found job 16964178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:55 proc_lsst.multi:146 [INFO] found job 16964179 in provider checkpoint proc_lsst.multi INFO: found job 16964179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:56 proc_lsst.multi:146 [INFO] found job 16964180 in provider checkpoint proc_lsst.multi INFO: found job 16964180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:56 proc_lsst.multi:146 [INFO] found job 16964181 in provider checkpoint proc_lsst.multi INFO: found job 16964181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:56 proc_lsst.multi:146 [INFO] found job 16964182 in provider checkpoint proc_lsst.multi INFO: found job 16964182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:56 proc_lsst.multi:146 [INFO] found job 16964183 in provider checkpoint proc_lsst.multi INFO: found job 16964183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7cf53e9fc3c0d58ee5cc5cb998a4acd with result from task 4960 parsl.dataflow.dflow DEBUG: Task 4960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/6f827389-0af5-4b46-909c-b4a40ca3a424_isrForCrosstalkSources_855437_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/6f827389-0af5-4b46-909c-b4a40ca3a424_isrForCrosstalkSources_855437_46.stderr parsl.dataflow.dflow INFO: Task 4950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd5cf5996606dc91c42e4727271f6237 with result from task 4950 parsl.dataflow.dflow DEBUG: Task 4950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/aa06734c-098c-4f55-bd0b-0a732079035b_isrForCrosstalkSources_855413_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/aa06734c-098c-4f55-bd0b-0a732079035b_isrForCrosstalkSources_855413_51.stderr parsl.dataflow.dflow INFO: Task 4972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18450aa8b3cd7e541f389e54a27e1102 with result from task 4972 parsl.dataflow.dflow DEBUG: Task 4972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8c6c2b89-aee6-4676-a777-423683b84ae7_isrForCrosstalkSources_855430_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855430/8c6c2b89-aee6-4676-a777-423683b84ae7_isrForCrosstalkSources_855430_40.stderr parsl.dataflow.dflow INFO: Task 4965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e105a1cfde662bed66ddb57a41cdb140 with result from task 4965 parsl.dataflow.dflow DEBUG: Task 4965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/38820c9e-6adb-4217-bfb1-0dad5df2edd2_isrForCrosstalkSources_855426_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/38820c9e-6adb-4217-bfb1-0dad5df2edd2_isrForCrosstalkSources_855426_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:56 proc_lsst.multi:146 [INFO] found job 16964184 in provider checkpoint proc_lsst.multi INFO: found job 16964184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:57 proc_lsst.multi:146 [INFO] found job 16964185 in provider checkpoint proc_lsst.multi INFO: found job 16964185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:57 proc_lsst.multi:146 [INFO] found job 16964186 in provider checkpoint proc_lsst.multi INFO: found job 16964186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78eb0196e3b919c98815355cd1b95838 with result from task 4961 parsl.dataflow.dflow DEBUG: Task 4961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/42324106-00bf-4466-9bdc-dfb6ea20db32_isrForCrosstalkSources_855445_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/42324106-00bf-4466-9bdc-dfb6ea20db32_isrForCrosstalkSources_855445_21.stderr parsl.dataflow.dflow INFO: Task 4963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dbc76fe6be7c0df61fa3969ccb9d524 with result from task 4963 parsl.dataflow.dflow DEBUG: Task 4963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/52e7373c-2159-4471-9474-1863d9d0dae2_isrForCrosstalkSources_855436_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/52e7373c-2159-4471-9474-1863d9d0dae2_isrForCrosstalkSources_855436_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:57 proc_lsst.multi:146 [INFO] found job 16964187 in provider checkpoint proc_lsst.multi INFO: found job 16964187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45178c494abe8e4c9d0bd23fd5c654fc with result from task 4951 parsl.dataflow.dflow DEBUG: Task 4951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9431a826-5879-46f7-b3a9-862303e14652_isrForCrosstalkSources_855383_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855383/9431a826-5879-46f7-b3a9-862303e14652_isrForCrosstalkSources_855383_5.stderr parsl.dataflow.dflow INFO: Task 4967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 725baa715c08bd14c42cbe3963cf83ca with result from task 4967 parsl.dataflow.dflow DEBUG: Task 4967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b3682ac3-470a-45d4-949f-21aa6d84cfd0_isrForCrosstalkSources_855448_10.stdout parsl.dataflow.dflow INFO: Standard error for task 4967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b3682ac3-470a-45d4-949f-21aa6d84cfd0_isrForCrosstalkSources_855448_10.stderr parsl.dataflow.dflow INFO: Task 4962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b58f645f74bb5b65b16f4dc2e775a4 with result from task 4962 parsl.dataflow.dflow DEBUG: Task 4962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/1419659e-b5c4-49b3-9927-123923e8b0db_isrForCrosstalkSources_855440_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855440/1419659e-b5c4-49b3-9927-123923e8b0db_isrForCrosstalkSources_855440_12.stderr parsl.dataflow.dflow INFO: Task 4958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69296ba7f859f5f58018f855421da7b6 with result from task 4958 parsl.dataflow.dflow DEBUG: Task 4958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f155e6ed-9254-418c-900c-675c684f71b7_isrForCrosstalkSources_855425_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f155e6ed-9254-418c-900c-675c684f71b7_isrForCrosstalkSources_855425_11.stderr parsl.dataflow.dflow INFO: Task 4959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c1eaf98afa17a167505906429b9cea4 with result from task 4959 parsl.dataflow.dflow DEBUG: Task 4959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/240d2022-503d-4c0c-91d4-74ed70879e00_isrForCrosstalkSources_855451_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/240d2022-503d-4c0c-91d4-74ed70879e00_isrForCrosstalkSources_855451_16.stderr parsl.dataflow.dflow INFO: Task 4955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee600c8a8397e4fdb13552003fa17cc9 with result from task 4955 parsl.dataflow.dflow DEBUG: Task 4955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/076f1b5a-971b-40c0-98eb-6cf8dc73783f_isrForCrosstalkSources_855416_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855416/076f1b5a-971b-40c0-98eb-6cf8dc73783f_isrForCrosstalkSources_855416_38.stderr parsl.dataflow.dflow INFO: Task 4954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ba8c5c43b0a19c8c03a28cc2ff43c04 with result from task 4954 parsl.dataflow.dflow DEBUG: Task 4954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/fad810c0-d6fe-485d-b690-0bd42c243b19_isrForCrosstalkSources_855403_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855403/fad810c0-d6fe-485d-b690-0bd42c243b19_isrForCrosstalkSources_855403_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:57 proc_lsst.multi:146 [INFO] found job 16964188 in provider checkpoint proc_lsst.multi INFO: found job 16964188 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f98b31e0409ff42e92de80d555aba38 with result from task 4956 parsl.dataflow.dflow DEBUG: Task 4956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d3ffd278-a791-4ca8-97d4-dd3771b818d2_isrForCrosstalkSources_855381_52.stdout parsl.dataflow.dflow INFO: Standard error for task 4956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/d3ffd278-a791-4ca8-97d4-dd3771b818d2_isrForCrosstalkSources_855381_52.stderr parsl.dataflow.dflow INFO: Task 4975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06dd1b8cade6c017c4b317332f1f8ae4 with result from task 4975 parsl.dataflow.dflow DEBUG: Task 4975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fde5c120-f749-4bc0-855a-a309863ca09a_isrForCrosstalkSources_855438_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/fde5c120-f749-4bc0-855a-a309863ca09a_isrForCrosstalkSources_855438_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 4949 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 0970410b25688da3ce6709982680fae1 with result from task 4949 parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 4949 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 4949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/26575103-6d77-4cfc-9529-6ea147681642_isrForCrosstalkSources_855389_42.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 4949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/26575103-6d77-4cfc-9529-6ea147681642_isrForCrosstalkSources_855389_42.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:58 proc_lsst.multi:146 [INFO] found job 16964189 in provider checkpoint proc_lsst.multi INFO: found job 16964189 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:58 proc_lsst.multi:146 [INFO] found job 16964190 in provider checkpoint proc_lsst.multi INFO: found job 16964190 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edbdf924a0f06be71e4a5c389c9f5110 with result from task 4966 parsl.dataflow.dflow DEBUG: Task 4966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a1c51628-922f-4927-adad-3b0161f5353a_isrForCrosstalkSources_855435_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855435/a1c51628-922f-4927-adad-3b0161f5353a_isrForCrosstalkSources_855435_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:58 proc_lsst.multi:146 [INFO] found job 16964191 in provider checkpoint proc_lsst.multi INFO: found job 16964191 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f9778797f710333aa76eae41f29e87f with result from task 4970 parsl.dataflow.dflow DEBUG: Task 4970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/6b29227e-5e36-4b4d-89ef-eecfb4aa2431_isrForCrosstalkSources_855450_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/6b29227e-5e36-4b4d-89ef-eecfb4aa2431_isrForCrosstalkSources_855450_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:58 proc_lsst.multi:146 [INFO] found job 16964192 in provider checkpoint proc_lsst.multi INFO: found job 16964192 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:59 proc_lsst.multi:146 [INFO] found job 16964193 in provider checkpoint proc_lsst.multi INFO: found job 16964193 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:59 proc_lsst.multi:146 [INFO] found job 16964194 in provider checkpoint proc_lsst.multi INFO: found job 16964194 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:59 proc_lsst.multi:146 [INFO] found job 16964195 in provider checkpoint proc_lsst.multi INFO: found job 16964195 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:56:59 proc_lsst.multi:146 [INFO] found job 16964196 in provider checkpoint proc_lsst.multi INFO: found job 16964196 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:00 proc_lsst.multi:146 [INFO] found job 16964197 in provider checkpoint proc_lsst.multi INFO: found job 16964197 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4284 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 7cbad348ffb823684164694461b27979 with result from task 4284 parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4284 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/515fb82d-b9cd-4b08-8cf4-75429d5f945a_isrForCrosstalkSources_855390_49.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/515fb82d-b9cd-4b08-8cf4-75429d5f945a_isrForCrosstalkSources_855390_49.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:00 proc_lsst.multi:146 [INFO] found job 16964198 in provider checkpoint proc_lsst.multi INFO: found job 16964198 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:00 proc_lsst.multi:146 [INFO] found job 16964199 in provider checkpoint proc_lsst.multi INFO: found job 16964199 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:00 proc_lsst.multi:146 [INFO] found job 16964200 in provider checkpoint proc_lsst.multi INFO: found job 16964200 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:01 proc_lsst.multi:146 [INFO] found job 16964201 in provider checkpoint proc_lsst.multi INFO: found job 16964201 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:01 proc_lsst.multi:146 [INFO] found job 16964202 in provider checkpoint proc_lsst.multi INFO: found job 16964202 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c1ee446b6e27cab1508d3948e5328fc with result from task 5021 parsl.dataflow.dflow DEBUG: Task 5021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/4e3286bd-abdb-4910-8013-9aa09185b766_isrForCrosstalkSources_855424_26.stdout parsl.dataflow.dflow INFO: Standard error for task 5021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/4e3286bd-abdb-4910-8013-9aa09185b766_isrForCrosstalkSources_855424_26.stderr parsl.dataflow.dflow INFO: Task 5019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b1c156d619f4d9b861bdffefd0d72f with result from task 5019 parsl.dataflow.dflow DEBUG: Task 5019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/779d8c58-9f0b-4d48-8969-e94fda7d1595_isrForCrosstalkSources_855439_25.stdout parsl.dataflow.dflow INFO: Standard error for task 5019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/779d8c58-9f0b-4d48-8969-e94fda7d1595_isrForCrosstalkSources_855439_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:01 proc_lsst.multi:146 [INFO] found job 16964203 in provider checkpoint proc_lsst.multi INFO: found job 16964203 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry eee190888085a014e2496ec72d0baad1 with result from task 793 parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f7aca349-2310-43ea-81e8-c4476279f493_isrForCrosstalkSources_855414_34.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:01 proc_lsst.multi:146 [INFO] found job 16964204 in provider checkpoint proc_lsst.multi INFO: found job 16964204 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:01 proc_lsst.multi:146 [INFO] found job 16964205 in provider checkpoint proc_lsst.multi INFO: found job 16964205 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39f26669241af09695a25c42055b155a with result from task 5008 parsl.dataflow.dflow DEBUG: Task 5008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f42dd90c-5b14-4f95-9a63-87d4911482e5_isrForCrosstalkSources_855423_51.stdout parsl.dataflow.dflow INFO: Standard error for task 5008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/f42dd90c-5b14-4f95-9a63-87d4911482e5_isrForCrosstalkSources_855423_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:02 proc_lsst.multi:146 [INFO] found job 16964206 in provider checkpoint proc_lsst.multi INFO: found job 16964206 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:02 proc_lsst.multi:146 [INFO] found job 16964207 in provider checkpoint proc_lsst.multi INFO: found job 16964207 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:02 proc_lsst.multi:146 [INFO] found job 16964208 in provider checkpoint proc_lsst.multi INFO: found job 16964208 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba985d823064dcd2db48d9d0a8776ee2 with result from task 5013 parsl.dataflow.dflow DEBUG: Task 5013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e05ee-2ce8-43ed-a1d4-82e4819e604d_isrForCrosstalkSources_855448_46.stdout parsl.dataflow.dflow INFO: Standard error for task 5013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/a64e05ee-2ce8-43ed-a1d4-82e4819e604d_isrForCrosstalkSources_855448_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:02 proc_lsst.multi:146 [INFO] found job 16964209 in provider checkpoint proc_lsst.multi INFO: found job 16964209 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 680d5f19f29947e68a6372db64eb421b with result from task 5010 parsl.dataflow.dflow DEBUG: Task 5010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/969d9e5b-d948-4d31-806c-4178d3991628_isrForCrosstalkSources_855438_22.stdout parsl.dataflow.dflow INFO: Standard error for task 5010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855438/969d9e5b-d948-4d31-806c-4178d3991628_isrForCrosstalkSources_855438_22.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71ea44cf6efbebf6de2ba24206180614 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/82d19897-f32c-4096-a7c7-0283ec1c8f66_isrForCrosstalkSources_855411_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:03 proc_lsst.multi:146 [INFO] found job 16964210 in provider checkpoint proc_lsst.multi INFO: found job 16964210 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2d020d98c9f6fb6df30831818517ee0 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/89761f69-4632-4f47-a00a-a48ed47e99e1_isrForCrosstalkSources_855417_51.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 347d4182371e0f83516a93341e40f981 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/02bc62e7-78b8-4e1f-8e58-8afc1ad0a368_isrForCrosstalkSources_855419_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:03 proc_lsst.multi:146 [INFO] found job 16964211 in provider checkpoint proc_lsst.multi INFO: found job 16964211 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:03 proc_lsst.multi:146 [INFO] found job 16964212 in provider checkpoint proc_lsst.multi INFO: found job 16964212 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3f23a67a0d40f776cdd5221d20f985a with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855453/8282667b-9616-4e39-bd8f-226480a12267_isrForCrosstalkSources_855453_16.stderr parsl.dataflow.dflow INFO: Task 5016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e53dec194564a6de49e0f32250b7b79 with result from task 5016 parsl.dataflow.dflow DEBUG: Task 5016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8c15864c-e207-4e47-85cc-625e046e8d90_isrForCrosstalkSources_855426_11.stdout parsl.dataflow.dflow INFO: Standard error for task 5016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/8c15864c-e207-4e47-85cc-625e046e8d90_isrForCrosstalkSources_855426_11.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daf3d1daf138b382a49930693605b896 with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855385/3b556155-0144-4272-b6ee-ca54a09f8096_isrForCrosstalkSources_855385_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:03 proc_lsst.multi:146 [INFO] found job 16964213 in provider checkpoint proc_lsst.multi INFO: found job 16964213 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e601272d001a24d344ca56f8dbc31185 with result from task 5014 parsl.dataflow.dflow DEBUG: Task 5014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fe7a871c-c7f1-484f-8054-0e21d371bd28_isrForCrosstalkSources_855413_18.stdout parsl.dataflow.dflow INFO: Standard error for task 5014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855413/fe7a871c-c7f1-484f-8054-0e21d371bd28_isrForCrosstalkSources_855413_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:04 proc_lsst.multi:146 [INFO] found job 16964214 in provider checkpoint proc_lsst.multi INFO: found job 16964214 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:04 proc_lsst.multi:146 [INFO] found job 16964215 in provider checkpoint proc_lsst.multi INFO: found job 16964215 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d019931c12f1bb700ce52d80293e7ed with result from task 5015 parsl.dataflow.dflow DEBUG: Task 5015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/187a97b8-c230-4c9e-a35c-190954859aee_isrForCrosstalkSources_855452_10.stdout parsl.dataflow.dflow INFO: Standard error for task 5015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/187a97b8-c230-4c9e-a35c-190954859aee_isrForCrosstalkSources_855452_10.stderr parsl.dataflow.dflow INFO: Task 5020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3de93384695852f13a347aa2b3d027b with result from task 5020 parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 5020 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 5020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6ff1ce12-a25b-4a26-bc54-1bdbf70e2eaa_isrForCrosstalkSources_855458_7.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 5020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855458/6ff1ce12-a25b-4a26-bc54-1bdbf70e2eaa_isrForCrosstalkSources_855458_7.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:04 proc_lsst.multi:146 [INFO] found job 16964216 in provider checkpoint proc_lsst.multi INFO: found job 16964216 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964138 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:04 proc_lsst.multi:146 [INFO] found job 16964217 in provider checkpoint proc_lsst.multi INFO: found job 16964217 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de2a63c4f0047168d5cdf57f96f3c2ef with result from task 5017 parsl.dataflow.dflow DEBUG: Task 5017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/aba73dfa-6f31-4c02-a532-001e42f69fb0_isrForCrosstalkSources_855422_30.stdout parsl.dataflow.dflow INFO: Standard error for task 5017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/aba73dfa-6f31-4c02-a532-001e42f69fb0_isrForCrosstalkSources_855422_30.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b206c1e89c81ea81fde7bb005171730 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855406/ad0bddcb-5a26-4a88-936a-20c4540f3fbe_isrForCrosstalkSources_855406_35.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d6aad113247ed5ac8eb070948cdc11f with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/ed23478d-80e8-4a18-983c-bef2b82bc358_isrForCrosstalkSources_855426_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:05 proc_lsst.multi:146 [INFO] found job 16964218 in provider checkpoint proc_lsst.multi INFO: found job 16964218 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9952d59c8ebfd5b0565529b1da2bcc9c with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/dabc37dc-56fd-4148-82b7-fd9d976989db_isrForCrosstalkSources_855415_30.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415f643c16d33fb8b9172458dab79d6b with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/1369f0a6-221d-464f-9e5f-273cb60c61cb_isrForCrosstalkSources_855454_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:05 proc_lsst.multi:146 [INFO] found job 16964220 in provider checkpoint proc_lsst.multi INFO: found job 16964220 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ffbde344c0e30bed8e53a01bf859339 with result from task 5009 parsl.dataflow.dflow DEBUG: Task 5009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3f37cb57-6601-4f4d-b03d-0cde40f30557_isrForCrosstalkSources_855411_49.stdout parsl.dataflow.dflow INFO: Standard error for task 5009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/3f37cb57-6601-4f4d-b03d-0cde40f30557_isrForCrosstalkSources_855411_49.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05794fc12ed60142732abdaed4d25bec with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9dd301d6-a255-476b-9da1-475474e558a9_isrForCrosstalkSources_855381_29.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63334f9bd3766f10de65a7c9d6cd1930 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/c86aa432-5809-4fc9-9727-136c9be5160d_isrForCrosstalkSources_855434_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:05 proc_lsst.multi:146 [INFO] found job 16964221 in provider checkpoint proc_lsst.multi INFO: found job 16964221 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7a458c5c77278ec905ecaa6e2adf22 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855402/9776078b-ec20-413a-9c9b-b001bfe00a2a_isrForCrosstalkSources_855402_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:06 proc_lsst.multi:146 [INFO] found job 16964222 in provider checkpoint proc_lsst.multi INFO: found job 16964222 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c3e06d0ae95866fbe420644ed949c6e with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855436/ea1471eb-0ac5-4d0a-b8c5-dc7fea46ab95_isrForCrosstalkSources_855436_60.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec11fa2fffad88bc471b77b669a7a928 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855459/d1e07fda-dea9-41bb-b7de-a3bd60878655_isrForCrosstalkSources_855459_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:06 proc_lsst.multi:146 [INFO] found job 16964223 in provider checkpoint proc_lsst.multi INFO: found job 16964223 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:06 proc_lsst.multi:146 [INFO] found job 16964224 in provider checkpoint proc_lsst.multi INFO: found job 16964224 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:06 proc_lsst.multi:146 [INFO] found job 16964225 in provider checkpoint proc_lsst.multi INFO: found job 16964225 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:07 proc_lsst.multi:146 [INFO] found job 16964226 in provider checkpoint proc_lsst.multi INFO: found job 16964226 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:07 proc_lsst.multi:146 [INFO] found job 16964227 in provider checkpoint proc_lsst.multi INFO: found job 16964227 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:07 proc_lsst.multi:146 [INFO] found job 16964228 in provider checkpoint proc_lsst.multi INFO: found job 16964228 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:07 proc_lsst.multi:146 [INFO] found job 16964229 in provider checkpoint proc_lsst.multi INFO: found job 16964229 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:08 proc_lsst.multi:146 [INFO] found job 16964230 in provider checkpoint proc_lsst.multi INFO: found job 16964230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:08 proc_lsst.multi:146 [INFO] found job 16964231 in provider checkpoint proc_lsst.multi INFO: found job 16964231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:08 proc_lsst.multi:146 [INFO] found job 16964232 in provider checkpoint proc_lsst.multi INFO: found job 16964232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:08 proc_lsst.multi:146 [INFO] found job 16964233 in provider checkpoint proc_lsst.multi INFO: found job 16964233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 CG 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:09 proc_lsst.multi:146 [INFO] found job 16964234 in provider checkpoint proc_lsst.multi INFO: found job 16964234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:09 proc_lsst.multi:146 [INFO] found job 16964235 in provider checkpoint proc_lsst.multi INFO: found job 16964235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:09 proc_lsst.multi:146 [INFO] found job 16964236 in provider checkpoint proc_lsst.multi INFO: found job 16964236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:09 proc_lsst.multi:146 [INFO] found job 16964237 in provider checkpoint proc_lsst.multi INFO: found job 16964237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:10 proc_lsst.multi:146 [INFO] found job 16964238 in provider checkpoint proc_lsst.multi INFO: found job 16964238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:10 proc_lsst.multi:146 [INFO] found job 16964239 in provider checkpoint proc_lsst.multi INFO: found job 16964239 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:10 proc_lsst.multi:146 [INFO] found job 16964242 in provider checkpoint proc_lsst.multi INFO: found job 16964242 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:10 proc_lsst.multi:146 [INFO] found job 16964243 in provider checkpoint proc_lsst.multi INFO: found job 16964243 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:10 proc_lsst.multi:146 [INFO] found job 16964244 in provider checkpoint proc_lsst.multi INFO: found job 16964244 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:11 proc_lsst.multi:146 [INFO] found job 16964245 in provider checkpoint proc_lsst.multi INFO: found job 16964245 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:11 proc_lsst.multi:146 [INFO] found job 16964246 in provider checkpoint proc_lsst.multi INFO: found job 16964246 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:11 proc_lsst.multi:146 [INFO] found job 16964247 in provider checkpoint proc_lsst.multi INFO: found job 16964247 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:11 proc_lsst.multi:146 [INFO] found job 16964248 in provider checkpoint proc_lsst.multi INFO: found job 16964248 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:12 proc_lsst.multi:146 [INFO] found job 16964249 in provider checkpoint proc_lsst.multi INFO: found job 16964249 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:12 proc_lsst.multi:146 [INFO] found job 16964250 in provider checkpoint proc_lsst.multi INFO: found job 16964250 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:12 proc_lsst.multi:146 [INFO] found job 16964251 in provider checkpoint proc_lsst.multi INFO: found job 16964251 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:12 proc_lsst.multi:146 [INFO] found job 16964252 in provider checkpoint proc_lsst.multi INFO: found job 16964252 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:13 proc_lsst.multi:146 [INFO] found job 16964253 in provider checkpoint proc_lsst.multi INFO: found job 16964253 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:13 proc_lsst.multi:146 [INFO] found job 16964254 in provider checkpoint proc_lsst.multi INFO: found job 16964254 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:13 proc_lsst.multi:146 [INFO] found job 16964255 in provider checkpoint proc_lsst.multi INFO: found job 16964255 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:13 proc_lsst.multi:146 [INFO] found job 16964256 in provider checkpoint proc_lsst.multi INFO: found job 16964256 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:14 proc_lsst.multi:146 [INFO] found job 16964257 in provider checkpoint proc_lsst.multi INFO: found job 16964257 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:14 proc_lsst.multi:146 [INFO] found job 16964258 in provider checkpoint proc_lsst.multi INFO: found job 16964258 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:14 proc_lsst.multi:146 [INFO] found job 16964259 in provider checkpoint proc_lsst.multi INFO: found job 16964259 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:14 proc_lsst.multi:146 [INFO] found job 16964260 in provider checkpoint proc_lsst.multi INFO: found job 16964260 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:15 proc_lsst.multi:146 [INFO] found job 16964261 in provider checkpoint proc_lsst.multi INFO: found job 16964261 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:15 proc_lsst.multi:146 [INFO] found job 16964262 in provider checkpoint proc_lsst.multi INFO: found job 16964262 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:15 proc_lsst.multi:146 [INFO] found job 16964263 in provider checkpoint proc_lsst.multi INFO: found job 16964263 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:15 proc_lsst.multi:146 [INFO] found job 16964264 in provider checkpoint proc_lsst.multi INFO: found job 16964264 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:16 proc_lsst.multi:146 [INFO] found job 16964265 in provider checkpoint proc_lsst.multi INFO: found job 16964265 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de0d00c7a9705c72348f095c681060e with result from task 4570 parsl.dataflow.dflow DEBUG: Task 4570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f79f19b5-c771-417e-bd2a-154805cde88e_isrForCrosstalkSources_855401_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/f79f19b5-c771-417e-bd2a-154805cde88e_isrForCrosstalkSources_855401_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:16 proc_lsst.multi:146 [INFO] found job 16964266 in provider checkpoint proc_lsst.multi INFO: found job 16964266 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:16 proc_lsst.multi:146 [INFO] found job 16964267 in provider checkpoint proc_lsst.multi INFO: found job 16964267 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:16 proc_lsst.multi:146 [INFO] found job 16964268 in provider checkpoint proc_lsst.multi INFO: found job 16964268 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:17 proc_lsst.multi:146 [INFO] found job 16964269 in provider checkpoint proc_lsst.multi INFO: found job 16964269 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:17 proc_lsst.multi:146 [INFO] found job 16964270 in provider checkpoint proc_lsst.multi INFO: found job 16964270 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a446b74e0e54d04307daca266f14297b with result from task 4572 parsl.dataflow.dflow DEBUG: Task 4572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e6b63f05-f1fe-470e-ba9b-03163efe3c26_isrForCrosstalkSources_855421_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/e6b63f05-f1fe-470e-ba9b-03163efe3c26_isrForCrosstalkSources_855421_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:17 proc_lsst.multi:146 [INFO] found job 16964271 in provider checkpoint proc_lsst.multi INFO: found job 16964271 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:17 proc_lsst.multi:146 [INFO] found job 16964272 in provider checkpoint proc_lsst.multi INFO: found job 16964272 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 274eef1d38c65c3e2ef370f46c8b9c91 with result from task 4566 parsl.dataflow.dflow DEBUG: Task 4566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/aee60548-94ec-40d1-9da9-2af87ecc6def_isrForCrosstalkSources_855433_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855433/aee60548-94ec-40d1-9da9-2af87ecc6def_isrForCrosstalkSources_855433_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:18 proc_lsst.multi:146 [INFO] found job 16964273 in provider checkpoint proc_lsst.multi INFO: found job 16964273 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:18 proc_lsst.multi:146 [INFO] found job 16964274 in provider checkpoint proc_lsst.multi INFO: found job 16964274 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:18 proc_lsst.multi:146 [INFO] found job 16964275 in provider checkpoint proc_lsst.multi INFO: found job 16964275 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:18 proc_lsst.multi:146 [INFO] found job 16964276 in provider checkpoint proc_lsst.multi INFO: found job 16964276 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:19 proc_lsst.multi:146 [INFO] found job 16964277 in provider checkpoint proc_lsst.multi INFO: found job 16964277 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:19 proc_lsst.multi:146 [INFO] found job 16964278 in provider checkpoint proc_lsst.multi INFO: found job 16964278 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:19 proc_lsst.multi:146 [INFO] found job 16964284 in provider checkpoint proc_lsst.multi INFO: found job 16964284 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:19 proc_lsst.multi:146 [INFO] found job 16964286 in provider checkpoint proc_lsst.multi INFO: found job 16964286 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:19 proc_lsst.multi:146 [INFO] found job 16964287 in provider checkpoint proc_lsst.multi INFO: found job 16964287 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:20 proc_lsst.multi:146 [INFO] found job 16964288 in provider checkpoint proc_lsst.multi INFO: found job 16964288 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:20 proc_lsst.multi:146 [INFO] found job 16964289 in provider checkpoint proc_lsst.multi INFO: found job 16964289 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:20 proc_lsst.multi:146 [INFO] found job 16964290 in provider checkpoint proc_lsst.multi INFO: found job 16964290 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:21 proc_lsst.multi:146 [INFO] found job 16964291 in provider checkpoint proc_lsst.multi INFO: found job 16964291 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:21 proc_lsst.multi:146 [INFO] found job 16964292 in provider checkpoint proc_lsst.multi INFO: found job 16964292 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:21 proc_lsst.multi:146 [INFO] found job 16964293 in provider checkpoint proc_lsst.multi INFO: found job 16964293 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:21 proc_lsst.multi:146 [INFO] found job 16964294 in provider checkpoint proc_lsst.multi INFO: found job 16964294 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 003415eaf564319dc97fd0926b3c345e with result from task 4599 parsl.dataflow.dflow DEBUG: Task 4599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a8dc423a-458f-4f46-86c5-17842194fb1b_isrForCrosstalkSources_855439_38.stdout parsl.dataflow.dflow INFO: Standard error for task 4599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/a8dc423a-458f-4f46-86c5-17842194fb1b_isrForCrosstalkSources_855439_38.stderr parsl.dataflow.dflow INFO: Task 4598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d81909b01f2f140b8aaa00d9b1ffdee with result from task 4598 parsl.dataflow.dflow DEBUG: Task 4598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b6f0be86-f6fa-4db2-a681-0598e35e87ab_isrForCrosstalkSources_855448_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/b6f0be86-f6fa-4db2-a681-0598e35e87ab_isrForCrosstalkSources_855448_61.stderr parsl.dataflow.dflow INFO: Task 4601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57a67d85ce41d6a34c89b112c8e4df83 with result from task 4601 parsl.dataflow.dflow DEBUG: Task 4601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f33898bf-c857-43b2-bfbc-c79e1383d7b2_isrForCrosstalkSources_855454_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855454/f33898bf-c857-43b2-bfbc-c79e1383d7b2_isrForCrosstalkSources_855454_41.stderr parsl.dataflow.dflow INFO: Task 4610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c0dda359fefd35ff0ea03c3dbcd4546 with result from task 4610 parsl.dataflow.dflow DEBUG: Task 4610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/935e514f-846b-4bdb-aa8d-9246426916aa_isrForCrosstalkSources_855432_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/935e514f-846b-4bdb-aa8d-9246426916aa_isrForCrosstalkSources_855432_24.stderr parsl.dataflow.dflow INFO: Task 4612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0792eb3f721c9f30c11354da17b39369 with result from task 4612 parsl.dataflow.dflow DEBUG: Task 4612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8cc66167-8d38-4f72-8410-f14e5f0479fd_isrForCrosstalkSources_855425_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/8cc66167-8d38-4f72-8410-f14e5f0479fd_isrForCrosstalkSources_855425_33.stderr parsl.dataflow.dflow INFO: Task 4611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ce6db4f8dd2d0c06d0b3b34af3cc0ed with result from task 4611 parsl.dataflow.dflow DEBUG: Task 4611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0ffa60ed-0105-414e-baf0-9859a3ceeb3a_isrForCrosstalkSources_855443_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/0ffa60ed-0105-414e-baf0-9859a3ceeb3a_isrForCrosstalkSources_855443_45.stderr parsl.dataflow.dflow INFO: Task 4585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6492de2308efd3eba7df1ac06a793666 with result from task 4585 parsl.dataflow.dflow DEBUG: Task 4585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9e50d0de-0159-468a-8af2-7f6e5dda40a1_isrForCrosstalkSources_855381_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/9e50d0de-0159-468a-8af2-7f6e5dda40a1_isrForCrosstalkSources_855381_53.stderr parsl.dataflow.dflow INFO: Task 4574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 832278823b198a5bbeede070e514fda8 with result from task 4574 parsl.dataflow.dflow DEBUG: Task 4574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ba4b8835-06f4-41f7-b8ae-fb494f25e4eb_isrForCrosstalkSources_855457_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/ba4b8835-06f4-41f7-b8ae-fb494f25e4eb_isrForCrosstalkSources_855457_17.stderr parsl.dataflow.dflow INFO: Task 4575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf296bf69da38a32b6cc95e8b699693 with result from task 4575 parsl.dataflow.dflow DEBUG: Task 4575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d5e8829c-6597-4d19-9f54-ab2f147b8bd7_isrForCrosstalkSources_855442_41.stdout parsl.dataflow.dflow INFO: Standard error for task 4575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855442/d5e8829c-6597-4d19-9f54-ab2f147b8bd7_isrForCrosstalkSources_855442_41.stderr parsl.dataflow.dflow INFO: Task 4577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c40bc3cbbd3e5ac0c4ba35cc5e223f48 with result from task 4577 parsl.dataflow.dflow DEBUG: Task 4577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9cba7650-d7a9-47a1-a2f1-271638f837b0_isrForCrosstalkSources_855457_5.stdout parsl.dataflow.dflow INFO: Standard error for task 4577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/9cba7650-d7a9-47a1-a2f1-271638f837b0_isrForCrosstalkSources_855457_5.stderr parsl.dataflow.dflow INFO: Task 4594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c222278222c936df8715a116a1aaf175 with result from task 4594 parsl.dataflow.dflow DEBUG: Task 4594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/09388ca4-1081-4c3e-97fb-dec1a284135a_isrForCrosstalkSources_855446_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855446/09388ca4-1081-4c3e-97fb-dec1a284135a_isrForCrosstalkSources_855446_55.stderr parsl.dataflow.dflow INFO: Task 4593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a08da452a787ccb409fec99821e9357 with result from task 4593 parsl.dataflow.dflow DEBUG: Task 4593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f625ccb0-32cd-4074-9420-9459da26716d_isrForCrosstalkSources_855381_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/f625ccb0-32cd-4074-9420-9459da26716d_isrForCrosstalkSources_855381_3.stderr parsl.dataflow.dflow INFO: Task 4597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca669477653a15802131742d1387e1f with result from task 4597 parsl.dataflow.dflow DEBUG: Task 4597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d3d29bad-8e46-429d-ba5a-89bbde93e043_isrForCrosstalkSources_855448_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/d3d29bad-8e46-429d-ba5a-89bbde93e043_isrForCrosstalkSources_855448_59.stderr parsl.dataflow.dflow INFO: Task 4600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3f664bbfe7bbddfaabf8ac537c8063a with result from task 4600 parsl.dataflow.dflow DEBUG: Task 4600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e0514520-baf0-4689-b876-dc5ee49540fe_isrForCrosstalkSources_855409_36.stdout parsl.dataflow.dflow INFO: Standard error for task 4600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/e0514520-baf0-4689-b876-dc5ee49540fe_isrForCrosstalkSources_855409_36.stderr parsl.dataflow.dflow INFO: Task 4578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecdf9acae82224ca82c845898981128d with result from task 4578 parsl.dataflow.dflow DEBUG: Task 4578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ab3ac9eb-2fab-4be5-b2e7-22cef37d159b_isrForCrosstalkSources_855412_16.stdout parsl.dataflow.dflow INFO: Standard error for task 4578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ab3ac9eb-2fab-4be5-b2e7-22cef37d159b_isrForCrosstalkSources_855412_16.stderr parsl.dataflow.dflow INFO: Task 4579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b8e6e4aa44f3fe1561232a6c2f1d4eb with result from task 4579 parsl.dataflow.dflow DEBUG: Task 4579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ee3fba0f-cf0d-4b7c-b53c-f188c38598d3_isrForCrosstalkSources_855401_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ee3fba0f-cf0d-4b7c-b53c-f188c38598d3_isrForCrosstalkSources_855401_21.stderr parsl.dataflow.dflow INFO: Task 4606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f41413fb052067139d10397234cc6d with result from task 4606 parsl.dataflow.dflow DEBUG: Task 4606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ab782550-12b4-4181-aed6-d419beac6967_isrForCrosstalkSources_855401_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/ab782550-12b4-4181-aed6-d419beac6967_isrForCrosstalkSources_855401_44.stderr parsl.dataflow.dflow INFO: Task 4607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ee0a4d1967794757f08d3fb8eac71b5 with result from task 4607 parsl.dataflow.dflow DEBUG: Task 4607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/90189218-1e20-4b45-8bd6-14a30a059b57_isrForCrosstalkSources_855452_17.stdout parsl.dataflow.dflow INFO: Standard error for task 4607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/90189218-1e20-4b45-8bd6-14a30a059b57_isrForCrosstalkSources_855452_17.stderr parsl.dataflow.dflow INFO: Task 4605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff94133735ad140e7258ca7e803e9823 with result from task 4605 parsl.dataflow.dflow DEBUG: Task 4605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/64f1ef32-217a-4789-8dce-f0d99cf5b930_isrForCrosstalkSources_855443_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/64f1ef32-217a-4789-8dce-f0d99cf5b930_isrForCrosstalkSources_855443_2.stderr parsl.dataflow.dflow INFO: Task 4591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91a6288ca890e755f3c6a5921428281 with result from task 4591 parsl.dataflow.dflow DEBUG: Task 4591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3c2f8fcb-f688-4e88-8490-3bdf2f0000e2_isrForCrosstalkSources_855450_49.stdout parsl.dataflow.dflow INFO: Standard error for task 4591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/3c2f8fcb-f688-4e88-8490-3bdf2f0000e2_isrForCrosstalkSources_855450_49.stderr parsl.dataflow.dflow INFO: Task 4590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c153e1e26ee2cea935ed4900d4e22d5 with result from task 4590 parsl.dataflow.dflow DEBUG: Task 4590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/37ba2fc4-8e69-4286-a233-af04148960ea_isrForCrosstalkSources_855439_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855439/37ba2fc4-8e69-4286-a233-af04148960ea_isrForCrosstalkSources_855439_35.stderr parsl.dataflow.dflow INFO: Task 4589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25ed212c8faac55504a6d70bac64904a with result from task 4589 parsl.dataflow.dflow DEBUG: Task 4589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d29510e4-00e2-4314-a90f-f5290dee5c53_isrForCrosstalkSources_855392_12.stdout parsl.dataflow.dflow INFO: Standard error for task 4589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855392/d29510e4-00e2-4314-a90f-f5290dee5c53_isrForCrosstalkSources_855392_12.stderr parsl.dataflow.dflow INFO: Task 4595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee28af9618978f1024fb4cfe27cd8bee with result from task 4595 parsl.dataflow.dflow DEBUG: Task 4595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/a11815a7-471e-4f3b-9788-099023807b4f_isrForCrosstalkSources_855423_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855423/a11815a7-471e-4f3b-9788-099023807b4f_isrForCrosstalkSources_855423_55.stderr parsl.dataflow.dflow INFO: Task 4588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47089ea0fc2661317aed181c58c647eb with result from task 4588 parsl.dataflow.dflow DEBUG: Task 4588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f6573a17-bce2-4877-b239-33d9c4e8312d_isrForCrosstalkSources_855397_18.stdout parsl.dataflow.dflow INFO: Standard error for task 4588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855397/f6573a17-bce2-4877-b239-33d9c4e8312d_isrForCrosstalkSources_855397_18.stderr parsl.dataflow.dflow INFO: Task 4586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32683eefc8e950f6895275bbcf412148 with result from task 4586 parsl.dataflow.dflow DEBUG: Task 4586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6d8a47c-d860-4372-b2a0-a247d25e594f_isrForCrosstalkSources_855457_47.stdout parsl.dataflow.dflow INFO: Standard error for task 4586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855457/e6d8a47c-d860-4372-b2a0-a247d25e594f_isrForCrosstalkSources_855457_47.stderr parsl.dataflow.dflow INFO: Task 4584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb8f12e16a41e8240c5d4b9a5a34e6a9 with result from task 4584 parsl.dataflow.dflow DEBUG: Task 4584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/235cad24-e8d5-4941-9b4c-aeef1a182b00_isrForCrosstalkSources_855448_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/235cad24-e8d5-4941-9b4c-aeef1a182b00_isrForCrosstalkSources_855448_13.stderr parsl.dataflow.dflow INFO: Task 4582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dca02eaea0ed9c033dba61990f2e4358 with result from task 4582 parsl.dataflow.dflow DEBUG: Task 4582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/8be35f57-e603-408f-b79c-2bf2128c7064_isrForCrosstalkSources_855396_46.stdout parsl.dataflow.dflow INFO: Standard error for task 4582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/8be35f57-e603-408f-b79c-2bf2128c7064_isrForCrosstalkSources_855396_46.stderr parsl.dataflow.dflow INFO: Task 4587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 018d393f37e2f3b8fcbd5bf5a96874ee with result from task 4587 parsl.dataflow.dflow DEBUG: Task 4587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7df9ab20-8abb-4bbc-88ca-e5e9bf69971c_isrForCrosstalkSources_855405_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/7df9ab20-8abb-4bbc-88ca-e5e9bf69971c_isrForCrosstalkSources_855405_35.stderr parsl.dataflow.dflow INFO: Task 4602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c4146eb9f4c48e3964e914f5115e912 with result from task 4602 parsl.dataflow.dflow DEBUG: Task 4602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/426439e4-f3be-4a78-a1fc-b7fe149a21b2_isrForCrosstalkSources_855429_29.stdout parsl.dataflow.dflow INFO: Standard error for task 4602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855429/426439e4-f3be-4a78-a1fc-b7fe149a21b2_isrForCrosstalkSources_855429_29.stderr parsl.dataflow.dflow INFO: Task 4604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0ac105b21e186f6faa45d8f4c95e1f with result from task 4604 parsl.dataflow.dflow DEBUG: Task 4604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/84d76693-f913-486c-aea5-94f3e97fb20d_isrForCrosstalkSources_855456_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855456/84d76693-f913-486c-aea5-94f3e97fb20d_isrForCrosstalkSources_855456_2.stderr parsl.dataflow.dflow INFO: Task 4609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ce3abb6595f1b833cf63a22722ee7e0 with result from task 4609 parsl.dataflow.dflow DEBUG: Task 4609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/07e56acd-6204-49a4-b65f-7fad71d4a883_isrForCrosstalkSources_855391_33.stdout parsl.dataflow.dflow INFO: Standard error for task 4609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855391/07e56acd-6204-49a4-b65f-7fad71d4a883_isrForCrosstalkSources_855391_33.stderr parsl.dataflow.dflow INFO: Task 4580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac05b34b4e7ca23746a61363ea97dddd with result from task 4580 parsl.dataflow.dflow DEBUG: Task 4580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a9ca11e5-a1e2-4026-9de9-def1bd62739d_isrForCrosstalkSources_855426_22.stdout parsl.dataflow.dflow INFO: Standard error for task 4580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855426/a9ca11e5-a1e2-4026-9de9-def1bd62739d_isrForCrosstalkSources_855426_22.stderr parsl.dataflow.dflow INFO: Task 4576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0ef34f4e6ff0e27c492c770cdf01bf with result from task 4576 parsl.dataflow.dflow DEBUG: Task 4576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dde19a2a-80c6-49db-869f-a64b7c7b25c8_isrForCrosstalkSources_855405_1.stdout parsl.dataflow.dflow INFO: Standard error for task 4576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855405/dde19a2a-80c6-49db-869f-a64b7c7b25c8_isrForCrosstalkSources_855405_1.stderr parsl.dataflow.dflow INFO: Task 4596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fbb6b06dd9efd5a4883430dd3a5419d with result from task 4596 parsl.dataflow.dflow DEBUG: Task 4596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/815191dc-9319-4679-abbf-21e04655c16e_isrForCrosstalkSources_855411_21.stdout parsl.dataflow.dflow INFO: Standard error for task 4596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855411/815191dc-9319-4679-abbf-21e04655c16e_isrForCrosstalkSources_855411_21.stderr parsl.dataflow.dflow INFO: Task 4581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6833b4e3314ecc83b2b5ced631aa974 with result from task 4581 parsl.dataflow.dflow DEBUG: Task 4581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2dcdf1ef-1126-40a5-961a-15139ce9efc7_isrForCrosstalkSources_855388_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855388/2dcdf1ef-1126-40a5-961a-15139ce9efc7_isrForCrosstalkSources_855388_14.stderr parsl.dataflow.dflow INFO: Task 4583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c4102dd1f660397321775cf3a1889a1 with result from task 4583 parsl.dataflow.dflow DEBUG: Task 4583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/66026946-f5ce-4685-9c1f-1a54f603c0cf_isrForCrosstalkSources_855450_24.stdout parsl.dataflow.dflow INFO: Standard error for task 4583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/66026946-f5ce-4685-9c1f-1a54f603c0cf_isrForCrosstalkSources_855450_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:22 proc_lsst.multi:146 [INFO] found job 16964295 in provider checkpoint proc_lsst.multi INFO: found job 16964295 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1e22fe832e491aaecb395d809ea617 with result from task 4573 parsl.dataflow.dflow DEBUG: Task 4573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e6a6efd-8585-4cfc-bb57-bcaf5cda0ced_isrForCrosstalkSources_855408_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855408/0e6a6efd-8585-4cfc-bb57-bcaf5cda0ced_isrForCrosstalkSources_855408_55.stderr parsl.dataflow.dflow INFO: Task 4603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6c2a35c3576786dfb048ac2c3d408b8 with result from task 4603 parsl.dataflow.dflow DEBUG: Task 4603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2ea50818-4b31-4e8e-88dd-c40b836a810b_isrForCrosstalkSources_855410_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855410/2ea50818-4b31-4e8e-88dd-c40b836a810b_isrForCrosstalkSources_855410_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:22 proc_lsst.multi:146 [INFO] found job 16964296 in provider checkpoint proc_lsst.multi INFO: found job 16964296 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:22 proc_lsst.multi:146 [INFO] found job 16964297 in provider checkpoint proc_lsst.multi INFO: found job 16964297 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:22 proc_lsst.multi:146 [INFO] found job 16964298 in provider checkpoint proc_lsst.multi INFO: found job 16964298 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:23 proc_lsst.multi:146 [INFO] found job 16964299 in provider checkpoint proc_lsst.multi INFO: found job 16964299 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:23 proc_lsst.multi:146 [INFO] found job 16964300 in provider checkpoint proc_lsst.multi INFO: found job 16964300 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:23 proc_lsst.multi:146 [INFO] found job 16964301 in provider checkpoint proc_lsst.multi INFO: found job 16964301 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:23 proc_lsst.multi:146 [INFO] found job 16964304 in provider checkpoint proc_lsst.multi INFO: found job 16964304 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:24 proc_lsst.multi:146 [INFO] found job 16964305 in provider checkpoint proc_lsst.multi INFO: found job 16964305 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7cc1cf20e011fbb26f1a243f7347902 with result from task 4592 parsl.dataflow.dflow DEBUG: Task 4592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f3c75a99-3c8f-4ada-b9d2-a59af95e1e2f_isrForCrosstalkSources_855414_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855414/f3c75a99-3c8f-4ada-b9d2-a59af95e1e2f_isrForCrosstalkSources_855414_60.stderr parsl.dataflow.dflow INFO: Task 4608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d40a5637e45f8dbc98b5b1e3a5dd47c with result from task 4608 parsl.dataflow.dflow DEBUG: Task 4608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f336ec43-c5be-4f11-bc41-6082d7717cde_isrForCrosstalkSources_855384_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855384/f336ec43-c5be-4f11-bc41-6082d7717cde_isrForCrosstalkSources_855384_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:24 proc_lsst.multi:146 [INFO] found job 16964306 in provider checkpoint proc_lsst.multi INFO: found job 16964306 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:24 proc_lsst.multi:146 [INFO] found job 16964307 in provider checkpoint proc_lsst.multi INFO: found job 16964307 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:24 proc_lsst.multi:146 [INFO] found job 16964308 in provider checkpoint proc_lsst.multi INFO: found job 16964308 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:25 proc_lsst.multi:146 [INFO] found job 16964309 in provider checkpoint proc_lsst.multi INFO: found job 16964309 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:27 proc_lsst.multi:146 [INFO] found job 16964310 in provider checkpoint proc_lsst.multi INFO: found job 16964310 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:27 proc_lsst.multi:146 [INFO] found job 16964311 in provider checkpoint proc_lsst.multi INFO: found job 16964311 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:29 proc_lsst.multi:146 [INFO] found job 16964312 in provider checkpoint proc_lsst.multi INFO: found job 16964312 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:30 proc_lsst.multi:146 [INFO] found job 16964313 in provider checkpoint proc_lsst.multi INFO: found job 16964313 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:30 proc_lsst.multi:146 [INFO] found job 16964314 in provider checkpoint proc_lsst.multi INFO: found job 16964314 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:30 proc_lsst.multi:146 [INFO] found job 16964315 in provider checkpoint proc_lsst.multi INFO: found job 16964315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:30 proc_lsst.multi:146 [INFO] found job 16964316 in provider checkpoint proc_lsst.multi INFO: found job 16964316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:31 proc_lsst.multi:146 [INFO] found job 16964317 in provider checkpoint proc_lsst.multi INFO: found job 16964317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:31 proc_lsst.multi:146 [INFO] found job 16964318 in provider checkpoint proc_lsst.multi INFO: found job 16964318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:31 proc_lsst.multi:146 [INFO] found job 16964319 in provider checkpoint proc_lsst.multi INFO: found job 16964319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:31 proc_lsst.multi:146 [INFO] found job 16964320 in provider checkpoint proc_lsst.multi INFO: found job 16964320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:32 proc_lsst.multi:146 [INFO] found job 16964321 in provider checkpoint proc_lsst.multi INFO: found job 16964321 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:32 proc_lsst.multi:146 [INFO] found job 16964322 in provider checkpoint proc_lsst.multi INFO: found job 16964322 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:32 proc_lsst.multi:146 [INFO] found job 16964323 in provider checkpoint proc_lsst.multi INFO: found job 16964323 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:32 proc_lsst.multi:146 [INFO] found job 16964324 in provider checkpoint proc_lsst.multi INFO: found job 16964324 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:33 proc_lsst.multi:146 [INFO] found job 16964325 in provider checkpoint proc_lsst.multi INFO: found job 16964325 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:33 proc_lsst.multi:146 [INFO] found job 16964326 in provider checkpoint proc_lsst.multi INFO: found job 16964326 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:33 proc_lsst.multi:146 [INFO] found job 16964327 in provider checkpoint proc_lsst.multi INFO: found job 16964327 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:33 proc_lsst.multi:146 [INFO] found job 16964328 in provider checkpoint proc_lsst.multi INFO: found job 16964328 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:34 proc_lsst.multi:146 [INFO] found job 16964329 in provider checkpoint proc_lsst.multi INFO: found job 16964329 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:34 proc_lsst.multi:146 [INFO] found job 16964330 in provider checkpoint proc_lsst.multi INFO: found job 16964330 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:34 proc_lsst.multi:146 [INFO] found job 16964331 in provider checkpoint proc_lsst.multi INFO: found job 16964331 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:34 proc_lsst.multi:146 [INFO] found job 16964332 in provider checkpoint proc_lsst.multi INFO: found job 16964332 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:35 proc_lsst.multi:146 [INFO] found job 16964333 in provider checkpoint proc_lsst.multi INFO: found job 16964333 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:35 proc_lsst.multi:146 [INFO] found job 16964334 in provider checkpoint proc_lsst.multi INFO: found job 16964334 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:35 proc_lsst.multi:146 [INFO] found job 16964335 in provider checkpoint proc_lsst.multi INFO: found job 16964335 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:35 proc_lsst.multi:146 [INFO] found job 16964336 in provider checkpoint proc_lsst.multi INFO: found job 16964336 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:36 proc_lsst.multi:146 [INFO] found job 16964337 in provider checkpoint proc_lsst.multi INFO: found job 16964337 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:36 proc_lsst.multi:146 [INFO] found job 16964338 in provider checkpoint proc_lsst.multi INFO: found job 16964338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:36 proc_lsst.multi:146 [INFO] found job 16964339 in provider checkpoint proc_lsst.multi INFO: found job 16964339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:36 proc_lsst.multi:146 [INFO] found job 16964340 in provider checkpoint proc_lsst.multi INFO: found job 16964340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:37 proc_lsst.multi:146 [INFO] found job 16964341 in provider checkpoint proc_lsst.multi INFO: found job 16964341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:37 proc_lsst.multi:146 [INFO] found job 16964342 in provider checkpoint proc_lsst.multi INFO: found job 16964342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:37 proc_lsst.multi:146 [INFO] found job 16964343 in provider checkpoint proc_lsst.multi INFO: found job 16964343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:37 proc_lsst.multi:146 [INFO] found job 16964344 in provider checkpoint proc_lsst.multi INFO: found job 16964344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:38 proc_lsst.multi:146 [INFO] found job 16964345 in provider checkpoint proc_lsst.multi INFO: found job 16964345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:38 proc_lsst.multi:146 [INFO] found job 16964346 in provider checkpoint proc_lsst.multi INFO: found job 16964346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:38 proc_lsst.multi:146 [INFO] found job 16964347 in provider checkpoint proc_lsst.multi INFO: found job 16964347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:39 proc_lsst.multi:146 [INFO] found job 16964348 in provider checkpoint proc_lsst.multi INFO: found job 16964348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 312, active_tasks = 36 parsl.dataflow.strategy DEBUG: Executor multi has 36 active tasks, 131/181 running/pending blocks, and 689 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22375328864144 2024-03-09 14:57:39 proc_lsst.multi:146 [INFO] found job 77954 in provider local proc_lsst.multi INFO: found job 77954 in provider local 2024-03-09 14:57:39 proc_lsst.multi:146 [INFO] found job 16964026 in provider astro proc_lsst.multi INFO: found job 16964026 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:39 proc_lsst.multi:146 [INFO] found job 16964027 in provider astro proc_lsst.multi INFO: found job 16964027 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:39 proc_lsst.multi:146 [INFO] found job 16964028 in provider astro proc_lsst.multi INFO: found job 16964028 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:40 proc_lsst.multi:146 [INFO] found job 16964029 in provider astro proc_lsst.multi INFO: found job 16964029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:40 proc_lsst.multi:146 [INFO] found job 16964030 in provider astro proc_lsst.multi INFO: found job 16964030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:40 proc_lsst.multi:146 [INFO] found job 16964031 in provider astro proc_lsst.multi INFO: found job 16964031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:40 proc_lsst.multi:146 [INFO] found job 16964032 in provider astro proc_lsst.multi INFO: found job 16964032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:41 proc_lsst.multi:146 [INFO] found job 16964033 in provider astro proc_lsst.multi INFO: found job 16964033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:41 proc_lsst.multi:146 [INFO] found job 16964034 in provider astro proc_lsst.multi INFO: found job 16964034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:41 proc_lsst.multi:146 [INFO] found job 16964035 in provider astro proc_lsst.multi INFO: found job 16964035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:41 proc_lsst.multi:146 [INFO] found job 16964036 in provider astro proc_lsst.multi INFO: found job 16964036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:41 proc_lsst.multi:146 [INFO] found job 16964037 in provider astro proc_lsst.multi INFO: found job 16964037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:42 proc_lsst.multi:146 [INFO] found job 16964038 in provider astro proc_lsst.multi INFO: found job 16964038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:42 proc_lsst.multi:146 [INFO] found job 16964039 in provider astro proc_lsst.multi INFO: found job 16964039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:42 proc_lsst.multi:146 [INFO] found job 16964040 in provider astro proc_lsst.multi INFO: found job 16964040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:42 proc_lsst.multi:146 [INFO] found job 16964041 in provider astro proc_lsst.multi INFO: found job 16964041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:43 proc_lsst.multi:146 [INFO] found job 16964042 in provider astro proc_lsst.multi INFO: found job 16964042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:43 proc_lsst.multi:146 [INFO] found job 16964043 in provider astro proc_lsst.multi INFO: found job 16964043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:43 proc_lsst.multi:146 [INFO] found job 16964044 in provider astro proc_lsst.multi INFO: found job 16964044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:43 proc_lsst.multi:146 [INFO] found job 16964045 in provider astro proc_lsst.multi INFO: found job 16964045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:43 proc_lsst.multi:146 [INFO] found job 16964046 in provider astro proc_lsst.multi INFO: found job 16964046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:44 proc_lsst.multi:146 [INFO] found job 16964047 in provider astro proc_lsst.multi INFO: found job 16964047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:44 proc_lsst.multi:146 [INFO] found job 16964048 in provider astro proc_lsst.multi INFO: found job 16964048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:44 proc_lsst.multi:146 [INFO] found job 16964049 in provider astro proc_lsst.multi INFO: found job 16964049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:44 proc_lsst.multi:146 [INFO] found job 16964050 in provider astro proc_lsst.multi INFO: found job 16964050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:44 proc_lsst.multi:146 [INFO] found job 16964051 in provider astro proc_lsst.multi INFO: found job 16964051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:45 proc_lsst.multi:146 [INFO] found job 16964052 in provider astro proc_lsst.multi INFO: found job 16964052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:45 proc_lsst.multi:146 [INFO] found job 16964053 in provider astro proc_lsst.multi INFO: found job 16964053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:45 proc_lsst.multi:146 [INFO] found job 16964054 in provider astro proc_lsst.multi INFO: found job 16964054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:45 proc_lsst.multi:146 [INFO] found job 16964055 in provider astro proc_lsst.multi INFO: found job 16964055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:46 proc_lsst.multi:146 [INFO] found job 16964056 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964056 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:46 proc_lsst.multi:146 [INFO] found job 16964057 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964057 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:46 proc_lsst.multi:146 [INFO] found job 16964058 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964058 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:46 proc_lsst.multi:146 [INFO] found job 16964059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:46 proc_lsst.multi:146 [INFO] found job 16964060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:47 proc_lsst.multi:146 [INFO] found job 16964061 in provider escience_a40 proc_lsst.multi INFO: found job 16964061 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:47 proc_lsst.multi:146 [INFO] found job 16964062 in provider escience_a40 proc_lsst.multi INFO: found job 16964062 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:47 proc_lsst.multi:146 [INFO] found job 16964063 in provider escience_a40 proc_lsst.multi INFO: found job 16964063 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:47 proc_lsst.multi:146 [INFO] found job 16964064 in provider escience_a40 proc_lsst.multi INFO: found job 16964064 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:48 proc_lsst.multi:146 [INFO] found job 16964065 in provider escience_a40 proc_lsst.multi INFO: found job 16964065 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:48 proc_lsst.multi:146 [INFO] found job 16964066 in provider escience_a40 proc_lsst.multi INFO: found job 16964066 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:48 proc_lsst.multi:146 [INFO] found job 16964067 in provider escience_a40 proc_lsst.multi INFO: found job 16964067 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:48 proc_lsst.multi:146 [INFO] found job 16964068 in provider escience_a40 proc_lsst.multi INFO: found job 16964068 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:48 proc_lsst.multi:146 [INFO] found job 16964069 in provider escience_a40 proc_lsst.multi INFO: found job 16964069 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:49 proc_lsst.multi:146 [INFO] found job 16964070 in provider escience_a40 proc_lsst.multi INFO: found job 16964070 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:49 proc_lsst.multi:146 [INFO] found job 16964071 in provider escience_a40 proc_lsst.multi INFO: found job 16964071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:49 proc_lsst.multi:146 [INFO] found job 16964072 in provider escience_a40 proc_lsst.multi INFO: found job 16964072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:49 proc_lsst.multi:146 [INFO] found job 16964073 in provider escience_a40 proc_lsst.multi INFO: found job 16964073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:49 proc_lsst.multi:146 [INFO] found job 16964074 in provider escience_a40 proc_lsst.multi INFO: found job 16964074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:50 proc_lsst.multi:146 [INFO] found job 16964075 in provider escience_a40 proc_lsst.multi INFO: found job 16964075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:50 proc_lsst.multi:146 [INFO] found job 16964076 in provider escience_a40 proc_lsst.multi INFO: found job 16964076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:50 proc_lsst.multi:146 [INFO] found job 16964077 in provider escience_a40 proc_lsst.multi INFO: found job 16964077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:50 proc_lsst.multi:146 [INFO] found job 16964078 in provider escience_a40 proc_lsst.multi INFO: found job 16964078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:51 proc_lsst.multi:146 [INFO] found job 16964079 in provider escience_a40 proc_lsst.multi INFO: found job 16964079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:51 proc_lsst.multi:146 [INFO] found job 16964080 in provider escience_a40 proc_lsst.multi INFO: found job 16964080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:51 proc_lsst.multi:146 [INFO] found job 16964081 in provider checkpoint proc_lsst.multi INFO: found job 16964081 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:51 proc_lsst.multi:146 [INFO] found job 16964082 in provider checkpoint proc_lsst.multi INFO: found job 16964082 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:51 proc_lsst.multi:146 [INFO] found job 16964083 in provider checkpoint proc_lsst.multi INFO: found job 16964083 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:52 proc_lsst.multi:146 [INFO] found job 16964084 in provider checkpoint proc_lsst.multi INFO: found job 16964084 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:52 proc_lsst.multi:146 [INFO] found job 16964085 in provider checkpoint proc_lsst.multi INFO: found job 16964085 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:52 proc_lsst.multi:146 [INFO] found job 16964086 in provider checkpoint proc_lsst.multi INFO: found job 16964086 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:52 proc_lsst.multi:146 [INFO] found job 16964087 in provider checkpoint proc_lsst.multi INFO: found job 16964087 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:53 proc_lsst.multi:146 [INFO] found job 16964088 in provider checkpoint proc_lsst.multi INFO: found job 16964088 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:53 proc_lsst.multi:146 [INFO] found job 16964089 in provider checkpoint proc_lsst.multi INFO: found job 16964089 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:53 proc_lsst.multi:146 [INFO] found job 16964090 in provider checkpoint proc_lsst.multi INFO: found job 16964090 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:53 proc_lsst.multi:146 [INFO] found job 16964091 in provider checkpoint proc_lsst.multi INFO: found job 16964091 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:54 proc_lsst.multi:146 [INFO] found job 16964092 in provider checkpoint proc_lsst.multi INFO: found job 16964092 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:54 proc_lsst.multi:146 [INFO] found job 16964093 in provider checkpoint proc_lsst.multi INFO: found job 16964093 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:54 proc_lsst.multi:146 [INFO] found job 16964094 in provider checkpoint proc_lsst.multi INFO: found job 16964094 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:54 proc_lsst.multi:146 [INFO] found job 16964095 in provider checkpoint proc_lsst.multi INFO: found job 16964095 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:55 proc_lsst.multi:146 [INFO] found job 16964096 in provider checkpoint proc_lsst.multi INFO: found job 16964096 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:55 proc_lsst.multi:146 [INFO] found job 16964097 in provider checkpoint proc_lsst.multi INFO: found job 16964097 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:55 proc_lsst.multi:146 [INFO] found job 16964099 in provider checkpoint proc_lsst.multi INFO: found job 16964099 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:55 proc_lsst.multi:146 [INFO] found job 16964100 in provider checkpoint proc_lsst.multi INFO: found job 16964100 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:56 proc_lsst.multi:146 [INFO] found job 16964101 in provider checkpoint proc_lsst.multi INFO: found job 16964101 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:56 proc_lsst.multi:146 [INFO] found job 16964102 in provider checkpoint proc_lsst.multi INFO: found job 16964102 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:56 proc_lsst.multi:146 [INFO] found job 16964103 in provider checkpoint proc_lsst.multi INFO: found job 16964103 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:56 proc_lsst.multi:146 [INFO] found job 16964104 in provider checkpoint proc_lsst.multi INFO: found job 16964104 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:57 proc_lsst.multi:146 [INFO] found job 16964105 in provider checkpoint proc_lsst.multi INFO: found job 16964105 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:57 proc_lsst.multi:146 [INFO] found job 16964106 in provider checkpoint proc_lsst.multi INFO: found job 16964106 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:57 proc_lsst.multi:146 [INFO] found job 16964107 in provider checkpoint proc_lsst.multi INFO: found job 16964107 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:57 proc_lsst.multi:146 [INFO] found job 16964108 in provider checkpoint proc_lsst.multi INFO: found job 16964108 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:58 proc_lsst.multi:146 [INFO] found job 16964109 in provider checkpoint proc_lsst.multi INFO: found job 16964109 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:58 proc_lsst.multi:146 [INFO] found job 16964110 in provider checkpoint proc_lsst.multi INFO: found job 16964110 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:58 proc_lsst.multi:146 [INFO] found job 16964111 in provider checkpoint proc_lsst.multi INFO: found job 16964111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:58 proc_lsst.multi:146 [INFO] found job 16964112 in provider checkpoint proc_lsst.multi INFO: found job 16964112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:59 proc_lsst.multi:146 [INFO] found job 16964113 in provider checkpoint proc_lsst.multi INFO: found job 16964113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:59 proc_lsst.multi:146 [INFO] found job 16964114 in provider checkpoint proc_lsst.multi INFO: found job 16964114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:59 proc_lsst.multi:146 [INFO] found job 16964115 in provider checkpoint proc_lsst.multi INFO: found job 16964115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:57:59 proc_lsst.multi:146 [INFO] found job 16964116 in provider checkpoint proc_lsst.multi INFO: found job 16964116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:00 proc_lsst.multi:146 [INFO] found job 16964117 in provider checkpoint proc_lsst.multi INFO: found job 16964117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964163 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:00 proc_lsst.multi:146 [INFO] found job 16964118 in provider checkpoint proc_lsst.multi INFO: found job 16964118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:02 proc_lsst.multi:146 [INFO] found job 16964119 in provider checkpoint proc_lsst.multi INFO: found job 16964119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:02 proc_lsst.multi:146 [INFO] found job 16964120 in provider checkpoint proc_lsst.multi INFO: found job 16964120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964163 PD 16964126 PD 16964125 PD 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:03 proc_lsst.multi:146 [INFO] found job 16964121 in provider checkpoint proc_lsst.multi INFO: found job 16964121 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:05 proc_lsst.multi:146 [INFO] found job 16964122 in provider checkpoint proc_lsst.multi INFO: found job 16964122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:05 proc_lsst.multi:146 [INFO] found job 16964123 in provider checkpoint proc_lsst.multi INFO: found job 16964123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:05 proc_lsst.multi:146 [INFO] found job 16964124 in provider checkpoint proc_lsst.multi INFO: found job 16964124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4976 try 0 failed parsl.dataflow.dflow INFO: Task 4976 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4976 has memoization hash 068858275610c0db25c7484293638112 parsl.dataflow.memoization INFO: Task 4976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4976 try 1 launched on executor multi with executor id 5039 parsl.dataflow.dflow INFO: Standard output for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4978 try 0 failed parsl.dataflow.dflow INFO: Task 4978 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4978 has memoization hash 55ee5a74f16cb86e0656e48296dddfd9 parsl.dataflow.memoization INFO: Task 4978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4978 try 1 launched on executor multi with executor id 5040 parsl.dataflow.dflow INFO: Standard output for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4980 try 0 failed parsl.dataflow.dflow INFO: Task 4980 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4980 has memoization hash 34a8fcf1ab4ca09eb886036155926839 parsl.dataflow.memoization INFO: Task 4980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4980 try 1 launched on executor multi with executor id 5041 parsl.dataflow.dflow INFO: Standard output for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4981 try 0 failed parsl.dataflow.dflow INFO: Task 4981 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4981 has memoization hash a95860496d8dce78b509f989bbc14672 parsl.dataflow.memoization INFO: Task 4981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4981 try 1 launched on executor multi with executor id 5042 parsl.dataflow.dflow INFO: Standard output for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4985 try 0 failed parsl.dataflow.dflow INFO: Task 4985 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4985 has memoization hash 212508430e8f427def6b995e5cddf899 parsl.dataflow.memoization INFO: Task 4985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4985 try 1 launched on executor multi with executor id 5043 parsl.dataflow.dflow INFO: Standard output for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4988 try 0 failed parsl.dataflow.dflow INFO: Task 4988 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4988 has memoization hash b6f3b62139cdc193fbb39ce774284375 parsl.dataflow.memoization INFO: Task 4988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4988 try 1 launched on executor multi with executor id 5044 parsl.dataflow.dflow INFO: Standard output for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4995 try 0 failed parsl.dataflow.dflow INFO: Task 4995 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4995 has memoization hash ed67e44dc1337a7378c11504c33e6f60 parsl.dataflow.memoization INFO: Task 4995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4995 try 1 launched on executor multi with executor id 5045 parsl.dataflow.dflow INFO: Standard output for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4998 try 0 failed parsl.dataflow.dflow INFO: Task 4998 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4998 has memoization hash c42a7c2e2934760b901ba9520f6b6a04 parsl.dataflow.memoization INFO: Task 4998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4998 try 1 launched on executor multi with executor id 5046 parsl.dataflow.dflow INFO: Standard output for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4977 try 0 failed parsl.dataflow.dflow INFO: Task 4977 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4977 has memoization hash 1206d1cdb7dc7c9881b8129305dd1b61 parsl.dataflow.memoization INFO: Task 4977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4977 try 1 launched on executor multi with executor id 5047 parsl.dataflow.dflow INFO: Standard output for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4984 try 0 failed parsl.dataflow.dflow INFO: Task 4984 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4984 has memoization hash 724de7214076ca09833b5721e5ac99e2 parsl.dataflow.memoization INFO: Task 4984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4984 try 1 launched on executor multi with executor id 5048 parsl.dataflow.dflow INFO: Standard output for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4987 try 0 failed parsl.dataflow.dflow INFO: Task 4987 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4987 has memoization hash fdd034bfb3e9f4be7eb7587c5f69e5ad parsl.dataflow.memoization INFO: Task 4987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4987 try 1 launched on executor multi with executor id 5049 parsl.dataflow.dflow INFO: Standard output for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4996 try 0 failed parsl.dataflow.dflow INFO: Task 4996 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4996 has memoization hash d0ca23344bafe5fdb23732b17ec718de parsl.dataflow.memoization INFO: Task 4996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4996 try 1 launched on executor multi with executor id 5050 parsl.dataflow.dflow INFO: Standard output for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4997 try 0 failed parsl.dataflow.dflow INFO: Task 4997 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4997 has memoization hash 5106fa85b08887127e2b23337a274286 parsl.dataflow.memoization INFO: Task 4997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a57ef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4997 try 1 launched on executor multi with executor id 5051 parsl.dataflow.dflow INFO: Standard output for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5003 try 0 failed parsl.dataflow.dflow INFO: Task 5003 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5003 has memoization hash 6fb44d566e8defe88d9c42e2e4054c2c parsl.dataflow.memoization INFO: Task 5003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5003 try 1 launched on executor multi with executor id 5052 parsl.dataflow.dflow INFO: Standard output for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stdout parsl.dataflow.dflow INFO: Standard error for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5005 try 0 failed parsl.dataflow.dflow INFO: Task 5005 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5005 has memoization hash 38cbf8cd714e220118127c5f01fd21f5 parsl.dataflow.memoization INFO: Task 5005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5005 try 1 launched on executor multi with executor id 5053 parsl.dataflow.dflow INFO: Standard output for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5007 try 0 failed parsl.dataflow.dflow INFO: Task 5007 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5007 has memoization hash a22b1d9743bbbe69dad0519416559bcf parsl.dataflow.memoization INFO: Task 5007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5007 try 1 launched on executor multi with executor id 5054 parsl.dataflow.dflow INFO: Standard output for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4979 try 0 failed parsl.dataflow.dflow INFO: Task 4979 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4979 has memoization hash 99206de481ec81770d21f62194e39a86 parsl.dataflow.memoization INFO: Task 4979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4979 try 1 launched on executor multi with executor id 5055 parsl.dataflow.dflow INFO: Standard output for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4986 try 0 failed parsl.dataflow.dflow INFO: Task 4986 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4986 has memoization hash ec219040399c7ddb65ebfd74da300d83 parsl.dataflow.memoization INFO: Task 4986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4986 try 1 launched on executor multi with executor id 5056 parsl.dataflow.dflow INFO: Standard output for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4991 try 0 failed parsl.dataflow.dflow INFO: Task 4991 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4991 has memoization hash 7c2052f7bc83cfa610552ed96d0085f6 parsl.dataflow.memoization INFO: Task 4991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4991 try 1 launched on executor multi with executor id 5057 parsl.dataflow.dflow INFO: Standard output for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4992 try 0 failed parsl.dataflow.dflow INFO: Task 4992 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4992 has memoization hash 5dbece7a960a0ba2561c883360cc8987 parsl.dataflow.memoization INFO: Task 4992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4992 try 1 launched on executor multi with executor id 5058 parsl.dataflow.dflow INFO: Standard output for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4993 try 0 failed parsl.dataflow.dflow INFO: Task 4993 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4993 has memoization hash 3789066774b1c2013e72659d669f384c parsl.dataflow.memoization INFO: Task 4993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4993 try 1 launched on executor multi with executor id 5059 parsl.dataflow.dflow INFO: Standard output for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4999 try 0 failed parsl.dataflow.dflow INFO: Task 4999 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4999 has memoization hash b7efe11eeee3acb69723f5dc781f4556 parsl.dataflow.memoization INFO: Task 4999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4999 try 1 launched on executor multi with executor id 5060 parsl.dataflow.dflow INFO: Standard output for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5000 try 0 failed parsl.dataflow.dflow INFO: Task 5000 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stdout parsl.dataflow.dflow INFO: Standard error for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5000 has memoization hash 267c735a3cfaf9cd7b912c5c3c6eda5b parsl.dataflow.memoization INFO: Task 5000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5000 try 1 launched on executor multi with executor id 5061 parsl.dataflow.dflow INFO: Standard output for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stdout parsl.dataflow.dflow INFO: Standard error for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5006 try 0 failed parsl.dataflow.dflow INFO: Task 5006 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stdout parsl.dataflow.dflow INFO: Standard error for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5006 has memoization hash 5f2cf339a0546f49d7d1a7411056aa07 parsl.dataflow.memoization INFO: Task 5006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5006 try 1 launched on executor multi with executor id 5062 parsl.dataflow.dflow INFO: Standard output for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stdout parsl.dataflow.dflow INFO: Standard error for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4982 try 0 failed parsl.dataflow.dflow INFO: Task 4982 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4982 has memoization hash 7222e2a9c8d07a56cba94e02d6279b75 parsl.dataflow.memoization INFO: Task 4982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4982 try 1 launched on executor multi with executor id 5063 parsl.dataflow.dflow INFO: Standard output for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4983 try 0 failed parsl.dataflow.dflow INFO: Task 4983 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4983 has memoization hash 37a5025841d0f98d14f916b221cc6e00 parsl.dataflow.memoization INFO: Task 4983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4983 try 1 launched on executor multi with executor id 5064 parsl.dataflow.dflow INFO: Standard output for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4989 try 0 failed parsl.dataflow.dflow INFO: Task 4989 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4989 has memoization hash 4ffba1b4102873cd5de6f809ab7098b0 parsl.dataflow.memoization INFO: Task 4989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4989 try 1 launched on executor multi with executor id 5065 parsl.dataflow.dflow INFO: Standard output for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4990 try 0 failed parsl.dataflow.dflow INFO: Task 4990 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4990 has memoization hash 1d1d185ef6c991a59c7043eb4afb2631 parsl.dataflow.memoization INFO: Task 4990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4990 try 1 launched on executor multi with executor id 5066 parsl.dataflow.dflow INFO: Standard output for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4994 try 0 failed parsl.dataflow.dflow INFO: Task 4994 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4994 has memoization hash e0e2cd6a35b99700dbc768fd792e27fc parsl.dataflow.memoization INFO: Task 4994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4994 try 1 launched on executor multi with executor id 5067 parsl.dataflow.dflow INFO: Standard output for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5001 try 0 failed parsl.dataflow.dflow INFO: Task 5001 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stdout parsl.dataflow.dflow INFO: Standard error for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5001 has memoization hash c9be6f4ded3e4d1f9edd0653798c2302 parsl.dataflow.memoization INFO: Task 5001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5001 try 1 launched on executor multi with executor id 5068 parsl.dataflow.dflow INFO: Standard output for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stdout parsl.dataflow.dflow INFO: Standard error for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5002 try 0 failed parsl.dataflow.dflow INFO: Task 5002 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stdout parsl.dataflow.dflow INFO: Standard error for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5002 has memoization hash c02d454f07ede8b12d6877d62ce14217 parsl.dataflow.memoization INFO: Task 5002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5002 try 1 launched on executor multi with executor id 5069 parsl.dataflow.dflow INFO: Standard output for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stdout parsl.dataflow.dflow INFO: Standard error for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5004 try 0 failed parsl.dataflow.dflow INFO: Task 5004 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5004 has memoization hash 0781bc40ef3a12e7e94731748989b821 parsl.dataflow.memoization INFO: Task 5004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5004 try 1 launched on executor multi with executor id 5070 parsl.dataflow.dflow INFO: Standard output for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:05 proc_lsst.multi:146 [INFO] found job 16964125 in provider checkpoint proc_lsst.multi INFO: found job 16964125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:06 proc_lsst.multi:146 [INFO] found job 16964126 in provider checkpoint proc_lsst.multi INFO: found job 16964126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:06 proc_lsst.multi:146 [INFO] found job 16964127 in provider checkpoint proc_lsst.multi INFO: found job 16964127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:06 proc_lsst.multi:146 [INFO] found job 16964128 in provider checkpoint proc_lsst.multi INFO: found job 16964128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:06 proc_lsst.multi:146 [INFO] found job 16964129 in provider checkpoint proc_lsst.multi INFO: found job 16964129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:07 proc_lsst.multi:146 [INFO] found job 16964130 in provider checkpoint proc_lsst.multi INFO: found job 16964130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:07 proc_lsst.multi:146 [INFO] found job 16964131 in provider checkpoint proc_lsst.multi INFO: found job 16964131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:07 proc_lsst.multi:146 [INFO] found job 16964132 in provider checkpoint proc_lsst.multi INFO: found job 16964132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:07 proc_lsst.multi:146 [INFO] found job 16964133 in provider checkpoint proc_lsst.multi INFO: found job 16964133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:07 proc_lsst.multi:146 [INFO] found job 16964134 in provider checkpoint proc_lsst.multi INFO: found job 16964134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:08 proc_lsst.multi:146 [INFO] found job 16964135 in provider checkpoint proc_lsst.multi INFO: found job 16964135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:08 proc_lsst.multi:146 [INFO] found job 16964136 in provider checkpoint proc_lsst.multi INFO: found job 16964136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:08 proc_lsst.multi:146 [INFO] found job 16964137 in provider checkpoint proc_lsst.multi INFO: found job 16964137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:08 proc_lsst.multi:146 [INFO] found job 16964138 in provider checkpoint proc_lsst.multi INFO: found job 16964138 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:09 proc_lsst.multi:146 [INFO] found job 16964139 in provider checkpoint proc_lsst.multi INFO: found job 16964139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:09 proc_lsst.multi:146 [INFO] found job 16964140 in provider checkpoint proc_lsst.multi INFO: found job 16964140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:09 proc_lsst.multi:146 [INFO] found job 16964141 in provider checkpoint proc_lsst.multi INFO: found job 16964141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:09 proc_lsst.multi:146 [INFO] found job 16964142 in provider checkpoint proc_lsst.multi INFO: found job 16964142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:10 proc_lsst.multi:146 [INFO] found job 16964143 in provider checkpoint proc_lsst.multi INFO: found job 16964143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:10 proc_lsst.multi:146 [INFO] found job 16964144 in provider checkpoint proc_lsst.multi INFO: found job 16964144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:10 proc_lsst.multi:146 [INFO] found job 16964145 in provider checkpoint proc_lsst.multi INFO: found job 16964145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:10 proc_lsst.multi:146 [INFO] found job 16964146 in provider checkpoint proc_lsst.multi INFO: found job 16964146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:11 proc_lsst.multi:146 [INFO] found job 16964147 in provider checkpoint proc_lsst.multi INFO: found job 16964147 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:11 proc_lsst.multi:146 [INFO] found job 16964148 in provider checkpoint proc_lsst.multi INFO: found job 16964148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:11 proc_lsst.multi:146 [INFO] found job 16964149 in provider checkpoint proc_lsst.multi INFO: found job 16964149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:11 proc_lsst.multi:146 [INFO] found job 16964150 in provider checkpoint proc_lsst.multi INFO: found job 16964150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:12 proc_lsst.multi:146 [INFO] found job 16964151 in provider checkpoint proc_lsst.multi INFO: found job 16964151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:12 proc_lsst.multi:146 [INFO] found job 16964152 in provider checkpoint proc_lsst.multi INFO: found job 16964152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:12 proc_lsst.multi:146 [INFO] found job 16964153 in provider checkpoint proc_lsst.multi INFO: found job 16964153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:12 proc_lsst.multi:146 [INFO] found job 16964154 in provider checkpoint proc_lsst.multi INFO: found job 16964154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:13 proc_lsst.multi:146 [INFO] found job 16964155 in provider checkpoint proc_lsst.multi INFO: found job 16964155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:13 proc_lsst.multi:146 [INFO] found job 16964156 in provider checkpoint proc_lsst.multi INFO: found job 16964156 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ffba1b4102873cd5de6f809ab7098b0 with result from task 4989 parsl.dataflow.dflow DEBUG: Task 4989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855400/53c6382c-f8cc-46ef-9157-9a7f6e3ec7b9_isrForCrosstalkSources_855400_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:13 proc_lsst.multi:146 [INFO] found job 16964157 in provider checkpoint proc_lsst.multi INFO: found job 16964157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d1d185ef6c991a59c7043eb4afb2631 with result from task 4990 parsl.dataflow.dflow DEBUG: Task 4990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stdout parsl.dataflow.dflow INFO: Standard error for task 4990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855434/bd313630-bf41-4646-9fbe-827428e44ce6_isrForCrosstalkSources_855434_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:14 proc_lsst.multi:146 [INFO] found job 16964158 in provider checkpoint proc_lsst.multi INFO: found job 16964158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37a5025841d0f98d14f916b221cc6e00 with result from task 4983 parsl.dataflow.dflow DEBUG: Task 4983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855450/dc2ea022-a198-403e-989d-2c6a0637fcf2_isrForCrosstalkSources_855450_39.stderr parsl.dataflow.dflow INFO: Task 5001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9be6f4ded3e4d1f9edd0653798c2302 with result from task 5001 parsl.dataflow.dflow DEBUG: Task 5001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stdout parsl.dataflow.dflow INFO: Standard error for task 5001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/5a4055a9-6cc6-42a3-bbf8-92f48a82ea67_isrForCrosstalkSources_855452_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:14 proc_lsst.multi:146 [INFO] found job 16964159 in provider checkpoint proc_lsst.multi INFO: found job 16964159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:14 proc_lsst.multi:146 [INFO] found job 16964160 in provider checkpoint proc_lsst.multi INFO: found job 16964160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:14 proc_lsst.multi:146 [INFO] found job 16964161 in provider checkpoint proc_lsst.multi INFO: found job 16964161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02d454f07ede8b12d6877d62ce14217 with result from task 5002 parsl.dataflow.dflow DEBUG: Task 5002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stdout parsl.dataflow.dflow INFO: Standard error for task 5002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/34208c76-d5ed-4c7a-a717-0d695f1d9bc5_isrForCrosstalkSources_855425_51.stderr parsl.dataflow.dflow INFO: Task 4994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e2cd6a35b99700dbc768fd792e27fc with result from task 4994 parsl.dataflow.dflow DEBUG: Task 4994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stdout parsl.dataflow.dflow INFO: Standard error for task 4994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855443/1956d703-7181-4cbc-8c2f-db1d91289987_isrForCrosstalkSources_855443_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:14 proc_lsst.multi:146 [INFO] found job 16964162 in provider checkpoint proc_lsst.multi INFO: found job 16964162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 5004 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 0781bc40ef3a12e7e94731748989b821 with result from task 5004 parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 5004 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 5004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f375ac15-28bf-49e9-b259-32cfeffdb022_isrForCrosstalkSources_855396_43.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:15 proc_lsst.multi:146 [INFO] found job 16964163 in provider checkpoint proc_lsst.multi INFO: found job 16964163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:15 proc_lsst.multi:146 [INFO] found job 16964164 in provider checkpoint proc_lsst.multi INFO: found job 16964164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:15 proc_lsst.multi:146 [INFO] found job 16964165 in provider checkpoint proc_lsst.multi INFO: found job 16964165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:15 proc_lsst.multi:146 [INFO] found job 16964166 in provider checkpoint proc_lsst.multi INFO: found job 16964166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:16 proc_lsst.multi:146 [INFO] found job 16964167 in provider checkpoint proc_lsst.multi INFO: found job 16964167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:16 proc_lsst.multi:146 [INFO] found job 16964168 in provider checkpoint proc_lsst.multi INFO: found job 16964168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99206de481ec81770d21f62194e39a86 with result from task 4979 parsl.dataflow.dflow DEBUG: Task 4979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stdout parsl.dataflow.dflow INFO: Standard error for task 4979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855422/1a7e287e-f697-4992-beb0-acd6d33f3260_isrForCrosstalkSources_855422_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:16 proc_lsst.multi:146 [INFO] found job 16964169 in provider checkpoint proc_lsst.multi INFO: found job 16964169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:16 proc_lsst.multi:146 [INFO] found job 16964170 in provider checkpoint proc_lsst.multi INFO: found job 16964170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:17 proc_lsst.multi:146 [INFO] found job 16964171 in provider checkpoint proc_lsst.multi INFO: found job 16964171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdd034bfb3e9f4be7eb7587c5f69e5ad with result from task 4987 parsl.dataflow.dflow DEBUG: Task 4987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stdout parsl.dataflow.dflow INFO: Standard error for task 4987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855415/a5de23ea-5bf2-453c-81ee-8216ecc1d526_isrForCrosstalkSources_855415_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 4976 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 068858275610c0db25c7484293638112 with result from task 4976 parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 4976 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 4976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/3bc7f9b7-b44c-48a1-aa6f-d33258beb9c7_isrForCrosstalkSources_855389_7.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:17 proc_lsst.multi:146 [INFO] found job 16964172 in provider checkpoint proc_lsst.multi INFO: found job 16964172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:17 proc_lsst.multi:146 [INFO] found job 16964173 in provider checkpoint proc_lsst.multi INFO: found job 16964173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 724de7214076ca09833b5721e5ac99e2 with result from task 4984 parsl.dataflow.dflow DEBUG: Task 4984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stdout parsl.dataflow.dflow INFO: Standard error for task 4984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855445/54a2b216-5ca5-4f02-806f-e077c13406fd_isrForCrosstalkSources_855445_35.stderr parsl.dataflow.dflow INFO: Task 5007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a22b1d9743bbbe69dad0519416559bcf with result from task 5007 parsl.dataflow.dflow DEBUG: Task 5007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855432/cef4a688-30b0-4c5f-8080-097aae999c85_isrForCrosstalkSources_855432_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:17 proc_lsst.multi:146 [INFO] found job 16964174 in provider checkpoint proc_lsst.multi INFO: found job 16964174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:18 proc_lsst.multi:146 [INFO] found job 16964175 in provider checkpoint proc_lsst.multi INFO: found job 16964175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5106fa85b08887127e2b23337a274286 with result from task 4997 parsl.dataflow.dflow DEBUG: Task 4997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stdout parsl.dataflow.dflow INFO: Standard error for task 4997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855455/5f84ad27-88a1-4d26-bf3f-b0f992e29634_isrForCrosstalkSources_855455_43.stderr parsl.dataflow.dflow INFO: Task 4996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0ca23344bafe5fdb23732b17ec718de with result from task 4996 parsl.dataflow.dflow DEBUG: Task 4996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855381/18b3daa0-196c-4abd-9442-e4c54dac2989_isrForCrosstalkSources_855381_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:18 proc_lsst.multi:146 [INFO] found job 16964176 in provider checkpoint proc_lsst.multi INFO: found job 16964176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:18 proc_lsst.multi:146 [INFO] found job 16964177 in provider checkpoint proc_lsst.multi INFO: found job 16964177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:18 proc_lsst.multi:146 [INFO] found job 16964178 in provider checkpoint proc_lsst.multi INFO: found job 16964178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:19 proc_lsst.multi:146 [INFO] found job 16964179 in provider checkpoint proc_lsst.multi INFO: found job 16964179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:19 proc_lsst.multi:146 [INFO] found job 16964180 in provider checkpoint proc_lsst.multi INFO: found job 16964180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38cbf8cd714e220118127c5f01fd21f5 with result from task 5005 parsl.dataflow.dflow DEBUG: Task 5005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/d95e83ff-f23d-4582-952e-2ffb16125363_isrForCrosstalkSources_855389_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 5003 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 6fb44d566e8defe88d9c42e2e4054c2c with result from task 5003 parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 5003 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 5003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855412/ea30b38c-759e-42a4-a77b-ced8f402ea49_isrForCrosstalkSources_855412_61.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:19 proc_lsst.multi:146 [INFO] found job 16964181 in provider checkpoint proc_lsst.multi INFO: found job 16964181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:19 proc_lsst.multi:146 [INFO] found job 16964182 in provider checkpoint proc_lsst.multi INFO: found job 16964182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1206d1cdb7dc7c9881b8129305dd1b61 with result from task 4977 parsl.dataflow.dflow DEBUG: Task 4977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855417/c883b2da-b8dd-4431-af61-4d679bd8dd26_isrForCrosstalkSources_855417_45.stderr parsl.dataflow.dflow INFO: Task 4978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55ee5a74f16cb86e0656e48296dddfd9 with result from task 4978 parsl.dataflow.dflow DEBUG: Task 4978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stdout parsl.dataflow.dflow INFO: Standard error for task 4978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855396/f62f2c3c-74da-4453-8c07-ad82f6d9dc0d_isrForCrosstalkSources_855396_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:20 proc_lsst.multi:146 [INFO] found job 16964183 in provider checkpoint proc_lsst.multi INFO: found job 16964183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:20 proc_lsst.multi:146 [INFO] found job 16964184 in provider checkpoint proc_lsst.multi INFO: found job 16964184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c42a7c2e2934760b901ba9520f6b6a04 with result from task 4998 parsl.dataflow.dflow DEBUG: Task 4998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/99c54f80-712d-4080-a77d-2bec69bdfb57_isrForCrosstalkSources_855389_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:20 proc_lsst.multi:146 [INFO] found job 16964185 in provider checkpoint proc_lsst.multi INFO: found job 16964185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:20 proc_lsst.multi:146 [INFO] found job 16964186 in provider checkpoint proc_lsst.multi INFO: found job 16964186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a95860496d8dce78b509f989bbc14672 with result from task 4981 parsl.dataflow.dflow DEBUG: Task 4981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855448/34f228eb-68db-4458-b3e7-78dbec3c5c41_isrForCrosstalkSources_855448_39.stderr parsl.dataflow.dflow INFO: Task 4995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed67e44dc1337a7378c11504c33e6f60 with result from task 4995 parsl.dataflow.dflow DEBUG: Task 4995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stdout parsl.dataflow.dflow INFO: Standard error for task 4995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855420/737cabb0-c059-49b3-b917-68e091c031ba_isrForCrosstalkSources_855420_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:21 proc_lsst.multi:146 [INFO] found job 16964187 in provider checkpoint proc_lsst.multi INFO: found job 16964187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34a8fcf1ab4ca09eb886036155926839 with result from task 4980 parsl.dataflow.dflow DEBUG: Task 4980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855386/8079e127-c133-4b34-a2aa-e475e87da549_isrForCrosstalkSources_855386_3.stderr parsl.dataflow.dflow INFO: Task 4988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6f3b62139cdc193fbb39ce774284375 with result from task 4988 parsl.dataflow.dflow DEBUG: Task 4988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stdout parsl.dataflow.dflow INFO: Standard error for task 4988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855421/c5832ec7-1c99-4f55-88be-d75c09420467_isrForCrosstalkSources_855421_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:21 proc_lsst.multi:146 [INFO] found job 16964188 in provider checkpoint proc_lsst.multi INFO: found job 16964188 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212508430e8f427def6b995e5cddf899 with result from task 4985 parsl.dataflow.dflow DEBUG: Task 4985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stdout parsl.dataflow.dflow INFO: Standard error for task 4985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855452/a139d0fd-a300-4a64-94de-3122a92babdd_isrForCrosstalkSources_855452_50.stderr parsl.dataflow.dflow INFO: Task 5000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267c735a3cfaf9cd7b912c5c3c6eda5b with result from task 5000 parsl.dataflow.dflow DEBUG: Task 5000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stdout parsl.dataflow.dflow INFO: Standard error for task 5000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/5623898c-6049-49e4-983a-7ef40025b054_isrForCrosstalkSources_855419_19.stderr parsl.dataflow.dflow INFO: Task 4986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec219040399c7ddb65ebfd74da300d83 with result from task 4986 parsl.dataflow.dflow DEBUG: Task 4986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855419/140c07ea-cbfb-487c-b81d-6bd2f28a3b45_isrForCrosstalkSources_855419_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:21 proc_lsst.multi:146 [INFO] found job 16964189 in provider checkpoint proc_lsst.multi INFO: found job 16964189 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:21 proc_lsst.multi:146 [INFO] found job 16964190 in provider checkpoint proc_lsst.multi INFO: found job 16964190 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:22 proc_lsst.multi:146 [INFO] found job 16964191 in provider checkpoint proc_lsst.multi INFO: found job 16964191 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f2cf339a0546f49d7d1a7411056aa07 with result from task 5006 parsl.dataflow.dflow DEBUG: Task 5006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stdout parsl.dataflow.dflow INFO: Standard error for task 5006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855409/ff2f8524-b054-4df6-8fc7-53500f4e0fba_isrForCrosstalkSources_855409_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:22 proc_lsst.multi:146 [INFO] found job 16964192 in provider checkpoint proc_lsst.multi INFO: found job 16964192 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:22 proc_lsst.multi:146 [INFO] found job 16964193 in provider checkpoint proc_lsst.multi INFO: found job 16964193 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7222e2a9c8d07a56cba94e02d6279b75 with result from task 4982 parsl.dataflow.dflow DEBUG: Task 4982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/73eb85fd-7693-4c83-8f0f-9691d25f1adf_isrForCrosstalkSources_855394_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:22 proc_lsst.multi:146 [INFO] found job 16964194 in provider checkpoint proc_lsst.multi INFO: found job 16964194 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:23 proc_lsst.multi:146 [INFO] found job 16964195 in provider checkpoint proc_lsst.multi INFO: found job 16964195 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7efe11eeee3acb69723f5dc781f4556 with result from task 4999 parsl.dataflow.dflow DEBUG: Task 4999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855437/b646d39b-d624-4cdb-8618-2deff3aa9883_isrForCrosstalkSources_855437_61.stderr parsl.dataflow.dflow INFO: Task 4993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3789066774b1c2013e72659d669f384c with result from task 4993 parsl.dataflow.dflow DEBUG: Task 4993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stdout parsl.dataflow.dflow INFO: Standard error for task 4993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855390/e069bc6e-e7c6-470e-9efb-1a6901eec6af_isrForCrosstalkSources_855390_54.stderr parsl.dataflow.dflow INFO: Task 4992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dbece7a960a0ba2561c883360cc8987 with result from task 4992 parsl.dataflow.dflow DEBUG: Task 4992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stdout parsl.dataflow.dflow INFO: Standard error for task 4992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855424/adbc0e40-c15d-44b3-8424-0791a1a95993_isrForCrosstalkSources_855424_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:23 proc_lsst.multi:146 [INFO] found job 16964196 in provider checkpoint proc_lsst.multi INFO: found job 16964196 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:23 proc_lsst.multi:146 [INFO] found job 16964197 in provider checkpoint proc_lsst.multi INFO: found job 16964197 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 4991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2052f7bc83cfa610552ed96d0085f6 with result from task 4991 parsl.dataflow.dflow DEBUG: Task 4991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stdout parsl.dataflow.dflow INFO: Standard error for task 4991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855389/60902009-547c-4a31-bdc0-25032a22adab_isrForCrosstalkSources_855389_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:23 proc_lsst.multi:146 [INFO] found job 16964198 in provider checkpoint proc_lsst.multi INFO: found job 16964198 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:24 proc_lsst.multi:146 [INFO] found job 16964199 in provider checkpoint proc_lsst.multi INFO: found job 16964199 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:24 proc_lsst.multi:146 [INFO] found job 16964200 in provider checkpoint proc_lsst.multi INFO: found job 16964200 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:24 proc_lsst.multi:146 [INFO] found job 16964201 in provider checkpoint proc_lsst.multi INFO: found job 16964201 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:24 proc_lsst.multi:146 [INFO] found job 16964202 in provider checkpoint proc_lsst.multi INFO: found job 16964202 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:25 proc_lsst.multi:146 [INFO] found job 16964203 in provider checkpoint proc_lsst.multi INFO: found job 16964203 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:25 proc_lsst.multi:146 [INFO] found job 16964204 in provider checkpoint proc_lsst.multi INFO: found job 16964204 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:25 proc_lsst.multi:146 [INFO] found job 16964205 in provider checkpoint proc_lsst.multi INFO: found job 16964205 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:25 proc_lsst.multi:146 [INFO] found job 16964206 in provider checkpoint proc_lsst.multi INFO: found job 16964206 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:26 proc_lsst.multi:146 [INFO] found job 16964207 in provider checkpoint proc_lsst.multi INFO: found job 16964207 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:26 proc_lsst.multi:146 [INFO] found job 16964208 in provider checkpoint proc_lsst.multi INFO: found job 16964208 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:26 proc_lsst.multi:146 [INFO] found job 16964209 in provider checkpoint proc_lsst.multi INFO: found job 16964209 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:26 proc_lsst.multi:146 [INFO] found job 16964210 in provider checkpoint proc_lsst.multi INFO: found job 16964210 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:26 proc_lsst.multi:146 [INFO] found job 16964211 in provider checkpoint proc_lsst.multi INFO: found job 16964211 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:27 proc_lsst.multi:146 [INFO] found job 16964212 in provider checkpoint proc_lsst.multi INFO: found job 16964212 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:27 proc_lsst.multi:146 [INFO] found job 16964213 in provider checkpoint proc_lsst.multi INFO: found job 16964213 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:27 proc_lsst.multi:146 [INFO] found job 16964214 in provider checkpoint proc_lsst.multi INFO: found job 16964214 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964141 PD 16964142 PD 16964143 PD 16964144 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:27 proc_lsst.multi:146 [INFO] found job 16964215 in provider checkpoint proc_lsst.multi INFO: found job 16964215 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:28 proc_lsst.multi:146 [INFO] found job 16964216 in provider checkpoint proc_lsst.multi INFO: found job 16964216 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:28 proc_lsst.multi:146 [INFO] found job 16964217 in provider checkpoint proc_lsst.multi INFO: found job 16964217 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:28 proc_lsst.multi:146 [INFO] found job 16964218 in provider checkpoint proc_lsst.multi INFO: found job 16964218 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:28 proc_lsst.multi:146 [INFO] found job 16964220 in provider checkpoint proc_lsst.multi INFO: found job 16964220 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964130 R 16964131 R 16964132 R 16964133 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:29 proc_lsst.multi:146 [INFO] found job 16964221 in provider checkpoint proc_lsst.multi INFO: found job 16964221 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:29 proc_lsst.multi:146 [INFO] found job 16964222 in provider checkpoint proc_lsst.multi INFO: found job 16964222 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:29 proc_lsst.multi:146 [INFO] found job 16964223 in provider checkpoint proc_lsst.multi INFO: found job 16964223 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:30 proc_lsst.multi:146 [INFO] found job 16964224 in provider checkpoint proc_lsst.multi INFO: found job 16964224 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:30 proc_lsst.multi:146 [INFO] found job 16964225 in provider checkpoint proc_lsst.multi INFO: found job 16964225 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:30 proc_lsst.multi:146 [INFO] found job 16964226 in provider checkpoint proc_lsst.multi INFO: found job 16964226 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:30 proc_lsst.multi:146 [INFO] found job 16964227 in provider checkpoint proc_lsst.multi INFO: found job 16964227 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:31 proc_lsst.multi:146 [INFO] found job 16964228 in provider checkpoint proc_lsst.multi INFO: found job 16964228 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:31 proc_lsst.multi:146 [INFO] found job 16964229 in provider checkpoint proc_lsst.multi INFO: found job 16964229 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:31 proc_lsst.multi:146 [INFO] found job 16964230 in provider checkpoint proc_lsst.multi INFO: found job 16964230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:31 proc_lsst.multi:146 [INFO] found job 16964231 in provider checkpoint proc_lsst.multi INFO: found job 16964231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:32 proc_lsst.multi:146 [INFO] found job 16964232 in provider checkpoint proc_lsst.multi INFO: found job 16964232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:32 proc_lsst.multi:146 [INFO] found job 16964233 in provider checkpoint proc_lsst.multi INFO: found job 16964233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:32 proc_lsst.multi:146 [INFO] found job 16964234 in provider checkpoint proc_lsst.multi INFO: found job 16964234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:32 proc_lsst.multi:146 [INFO] found job 16964235 in provider checkpoint proc_lsst.multi INFO: found job 16964235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:33 proc_lsst.multi:146 [INFO] found job 16964236 in provider checkpoint proc_lsst.multi INFO: found job 16964236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 CG 16964131 CG 16964132 CG 16964133 CG 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:33 proc_lsst.multi:146 [INFO] found job 16964237 in provider checkpoint proc_lsst.multi INFO: found job 16964237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964131 CG 16964132 CG 16964130 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:33 proc_lsst.multi:146 [INFO] found job 16964238 in provider checkpoint proc_lsst.multi INFO: found job 16964238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:33 proc_lsst.multi:146 [INFO] found job 16964239 in provider checkpoint proc_lsst.multi INFO: found job 16964239 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:34 proc_lsst.multi:146 [INFO] found job 16964242 in provider checkpoint proc_lsst.multi INFO: found job 16964242 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:34 proc_lsst.multi:146 [INFO] found job 16964243 in provider checkpoint proc_lsst.multi INFO: found job 16964243 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:34 proc_lsst.multi:146 [INFO] found job 16964244 in provider checkpoint proc_lsst.multi INFO: found job 16964244 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:34 proc_lsst.multi:146 [INFO] found job 16964245 in provider checkpoint proc_lsst.multi INFO: found job 16964245 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:37 proc_lsst.multi:146 [INFO] found job 16964246 in provider checkpoint proc_lsst.multi INFO: found job 16964246 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:37 proc_lsst.multi:146 [INFO] found job 16964247 in provider checkpoint proc_lsst.multi INFO: found job 16964247 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:37 proc_lsst.multi:146 [INFO] found job 16964248 in provider checkpoint proc_lsst.multi INFO: found job 16964248 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:38 proc_lsst.multi:146 [INFO] found job 16964249 in provider checkpoint proc_lsst.multi INFO: found job 16964249 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:39 proc_lsst.multi:146 [INFO] found job 16964250 in provider checkpoint proc_lsst.multi INFO: found job 16964250 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:39 proc_lsst.multi:146 [INFO] found job 16964251 in provider checkpoint proc_lsst.multi INFO: found job 16964251 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:39 proc_lsst.multi:146 [INFO] found job 16964252 in provider checkpoint proc_lsst.multi INFO: found job 16964252 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:39 proc_lsst.multi:146 [INFO] found job 16964253 in provider checkpoint proc_lsst.multi INFO: found job 16964253 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:40 proc_lsst.multi:146 [INFO] found job 16964254 in provider checkpoint proc_lsst.multi INFO: found job 16964254 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:40 proc_lsst.multi:146 [INFO] found job 16964255 in provider checkpoint proc_lsst.multi INFO: found job 16964255 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:40 proc_lsst.multi:146 [INFO] found job 16964256 in provider checkpoint proc_lsst.multi INFO: found job 16964256 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:40 proc_lsst.multi:146 [INFO] found job 16964257 in provider checkpoint proc_lsst.multi INFO: found job 16964257 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:41 proc_lsst.multi:146 [INFO] found job 16964258 in provider checkpoint proc_lsst.multi INFO: found job 16964258 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:41 proc_lsst.multi:146 [INFO] found job 16964259 in provider checkpoint proc_lsst.multi INFO: found job 16964259 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:41 proc_lsst.multi:146 [INFO] found job 16964260 in provider checkpoint proc_lsst.multi INFO: found job 16964260 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:41 proc_lsst.multi:146 [INFO] found job 16964261 in provider checkpoint proc_lsst.multi INFO: found job 16964261 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:41 proc_lsst.multi:146 [INFO] found job 16964262 in provider checkpoint proc_lsst.multi INFO: found job 16964262 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:42 proc_lsst.multi:146 [INFO] found job 16964263 in provider checkpoint proc_lsst.multi INFO: found job 16964263 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:42 proc_lsst.multi:146 [INFO] found job 16964264 in provider checkpoint proc_lsst.multi INFO: found job 16964264 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:42 proc_lsst.multi:146 [INFO] found job 16964265 in provider checkpoint proc_lsst.multi INFO: found job 16964265 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:42 proc_lsst.multi:146 [INFO] found job 16964266 in provider checkpoint proc_lsst.multi INFO: found job 16964266 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:43 proc_lsst.multi:146 [INFO] found job 16964267 in provider checkpoint proc_lsst.multi INFO: found job 16964267 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:43 proc_lsst.multi:146 [INFO] found job 16964268 in provider checkpoint proc_lsst.multi INFO: found job 16964268 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:43 proc_lsst.multi:146 [INFO] found job 16964269 in provider checkpoint proc_lsst.multi INFO: found job 16964269 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:43 proc_lsst.multi:146 [INFO] found job 16964270 in provider checkpoint proc_lsst.multi INFO: found job 16964270 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:44 proc_lsst.multi:146 [INFO] found job 16964271 in provider checkpoint proc_lsst.multi INFO: found job 16964271 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:44 proc_lsst.multi:146 [INFO] found job 16964272 in provider checkpoint proc_lsst.multi INFO: found job 16964272 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:44 proc_lsst.multi:146 [INFO] found job 16964273 in provider checkpoint proc_lsst.multi INFO: found job 16964273 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:44 proc_lsst.multi:146 [INFO] found job 16964274 in provider checkpoint proc_lsst.multi INFO: found job 16964274 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:45 proc_lsst.multi:146 [INFO] found job 16964275 in provider checkpoint proc_lsst.multi INFO: found job 16964275 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:45 proc_lsst.multi:146 [INFO] found job 16964276 in provider checkpoint proc_lsst.multi INFO: found job 16964276 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:45 proc_lsst.multi:146 [INFO] found job 16964277 in provider checkpoint proc_lsst.multi INFO: found job 16964277 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:45 proc_lsst.multi:146 [INFO] found job 16964278 in provider checkpoint proc_lsst.multi INFO: found job 16964278 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:46 proc_lsst.multi:146 [INFO] found job 16964284 in provider checkpoint proc_lsst.multi INFO: found job 16964284 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:46 proc_lsst.multi:146 [INFO] found job 16964286 in provider checkpoint proc_lsst.multi INFO: found job 16964286 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:46 proc_lsst.multi:146 [INFO] found job 16964287 in provider checkpoint proc_lsst.multi INFO: found job 16964287 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:46 proc_lsst.multi:146 [INFO] found job 16964288 in provider checkpoint proc_lsst.multi INFO: found job 16964288 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:47 proc_lsst.multi:146 [INFO] found job 16964289 in provider checkpoint proc_lsst.multi INFO: found job 16964289 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:47 proc_lsst.multi:146 [INFO] found job 16964290 in provider checkpoint proc_lsst.multi INFO: found job 16964290 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:47 proc_lsst.multi:146 [INFO] found job 16964291 in provider checkpoint proc_lsst.multi INFO: found job 16964291 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:47 proc_lsst.multi:146 [INFO] found job 16964292 in provider checkpoint proc_lsst.multi INFO: found job 16964292 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:48 proc_lsst.multi:146 [INFO] found job 16964293 in provider checkpoint proc_lsst.multi INFO: found job 16964293 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5011 try 0 failed parsl.dataflow.dflow INFO: Task 5011 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5011 has memoization hash 4023f2cd117b8b85ee3a45c069d2ddfe parsl.dataflow.memoization INFO: Task 5011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5011 try 1 launched on executor multi with executor id 5071 parsl.dataflow.dflow INFO: Standard output for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5012 try 0 failed parsl.dataflow.dflow INFO: Task 5012 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stdout parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.dataflow.dflow INFO: Standard error for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Task 5012 has memoization hash d0ad46b40fa77e0b506efeadeb768904 parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization INFO: Task 5012 had no result in cache parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Parsl task 5012 try 1 launched on executor multi with executor id 5072 parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.app.errors DEBUG: Reraising exception of type parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 5018 try 0 failed parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 5018 marked for retry parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Task 5018 has memoization hash fc6691b4abf432ea6e9e37df7fb1b982 parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization INFO: Task 5018 had no result in cache parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1459a569d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Parsl task 5018 try 1 launched on executor multi with executor id 5073 parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.app.errors DEBUG: Reraising exception of type parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 803 try 1 failed parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow ERROR: Task 803 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 564, in _unwrap_remote_exception_wrapper result = future.result() ^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/high_throughput/executor.py", line 443, in _queue_management_worker s.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/high_throughput/interchange.py", line 572, in expire_bad_managers raise ManagerLost(manager_id, m['hostname']) ^^^^^^^^^^^^^^^^^ parsl.executors.high_throughput.interchange.ManagerLost: Task failure due to loss of manager c041dc9ff788 on host g3066 parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 972c49fedc3efbbf9f277a960b822c55 with result from task 803 parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855451/052b0d36-c08a-447d-abcf-1f201f6e706d_isrForCrosstalkSources_855451_44.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:48 proc_lsst.multi:146 [INFO] found job 16964294 in provider checkpoint proc_lsst.multi INFO: found job 16964294 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:48 proc_lsst.multi:146 [INFO] found job 16964295 in provider checkpoint proc_lsst.multi INFO: found job 16964295 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:48 proc_lsst.multi:146 [INFO] found job 16964296 in provider checkpoint proc_lsst.multi INFO: found job 16964296 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:49 proc_lsst.multi:146 [INFO] found job 16964297 in provider checkpoint proc_lsst.multi INFO: found job 16964297 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:49 proc_lsst.multi:146 [INFO] found job 16964298 in provider checkpoint proc_lsst.multi INFO: found job 16964298 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:49 proc_lsst.multi:146 [INFO] found job 16964299 in provider checkpoint proc_lsst.multi INFO: found job 16964299 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:49 proc_lsst.multi:146 [INFO] found job 16964300 in provider checkpoint proc_lsst.multi INFO: found job 16964300 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:50 proc_lsst.multi:146 [INFO] found job 16964301 in provider checkpoint proc_lsst.multi INFO: found job 16964301 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:50 proc_lsst.multi:146 [INFO] found job 16964304 in provider checkpoint proc_lsst.multi INFO: found job 16964304 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:50 proc_lsst.multi:146 [INFO] found job 16964305 in provider checkpoint proc_lsst.multi INFO: found job 16964305 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:50 proc_lsst.multi:146 [INFO] found job 16964306 in provider checkpoint proc_lsst.multi INFO: found job 16964306 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:51 proc_lsst.multi:146 [INFO] found job 16964307 in provider checkpoint proc_lsst.multi INFO: found job 16964307 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:51 proc_lsst.multi:146 [INFO] found job 16964308 in provider checkpoint proc_lsst.multi INFO: found job 16964308 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:51 proc_lsst.multi:146 [INFO] found job 16964309 in provider checkpoint proc_lsst.multi INFO: found job 16964309 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:51 proc_lsst.multi:146 [INFO] found job 16964310 in provider checkpoint proc_lsst.multi INFO: found job 16964310 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:52 proc_lsst.multi:146 [INFO] found job 16964311 in provider checkpoint proc_lsst.multi INFO: found job 16964311 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:52 proc_lsst.multi:146 [INFO] found job 16964312 in provider checkpoint proc_lsst.multi INFO: found job 16964312 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:52 proc_lsst.multi:146 [INFO] found job 16964313 in provider checkpoint proc_lsst.multi INFO: found job 16964313 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:52 proc_lsst.multi:146 [INFO] found job 16964314 in provider checkpoint proc_lsst.multi INFO: found job 16964314 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:53 proc_lsst.multi:146 [INFO] found job 16964315 in provider checkpoint proc_lsst.multi INFO: found job 16964315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:53 proc_lsst.multi:146 [INFO] found job 16964316 in provider checkpoint proc_lsst.multi INFO: found job 16964316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:53 proc_lsst.multi:146 [INFO] found job 16964317 in provider checkpoint proc_lsst.multi INFO: found job 16964317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:53 proc_lsst.multi:146 [INFO] found job 16964318 in provider checkpoint proc_lsst.multi INFO: found job 16964318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:54 proc_lsst.multi:146 [INFO] found job 16964319 in provider checkpoint proc_lsst.multi INFO: found job 16964319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:54 proc_lsst.multi:146 [INFO] found job 16964320 in provider checkpoint proc_lsst.multi INFO: found job 16964320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:54 proc_lsst.multi:146 [INFO] found job 16964321 in provider checkpoint proc_lsst.multi INFO: found job 16964321 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:54 proc_lsst.multi:146 [INFO] found job 16964322 in provider checkpoint proc_lsst.multi INFO: found job 16964322 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:55 proc_lsst.multi:146 [INFO] found job 16964323 in provider checkpoint proc_lsst.multi INFO: found job 16964323 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:55 proc_lsst.multi:146 [INFO] found job 16964324 in provider checkpoint proc_lsst.multi INFO: found job 16964324 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:55 proc_lsst.multi:146 [INFO] found job 16964325 in provider checkpoint proc_lsst.multi INFO: found job 16964325 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:55 proc_lsst.multi:146 [INFO] found job 16964326 in provider checkpoint proc_lsst.multi INFO: found job 16964326 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:56 proc_lsst.multi:146 [INFO] found job 16964327 in provider checkpoint proc_lsst.multi INFO: found job 16964327 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:56 proc_lsst.multi:146 [INFO] found job 16964328 in provider checkpoint proc_lsst.multi INFO: found job 16964328 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:56 proc_lsst.multi:146 [INFO] found job 16964329 in provider checkpoint proc_lsst.multi INFO: found job 16964329 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:56 proc_lsst.multi:146 [INFO] found job 16964330 in provider checkpoint proc_lsst.multi INFO: found job 16964330 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:57 proc_lsst.multi:146 [INFO] found job 16964331 in provider checkpoint proc_lsst.multi INFO: found job 16964331 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:57 proc_lsst.multi:146 [INFO] found job 16964332 in provider checkpoint proc_lsst.multi INFO: found job 16964332 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:57 proc_lsst.multi:146 [INFO] found job 16964333 in provider checkpoint proc_lsst.multi INFO: found job 16964333 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:57 proc_lsst.multi:146 [INFO] found job 16964334 in provider checkpoint proc_lsst.multi INFO: found job 16964334 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:57 proc_lsst.multi:146 [INFO] found job 16964335 in provider checkpoint proc_lsst.multi INFO: found job 16964335 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:58 proc_lsst.multi:146 [INFO] found job 16964336 in provider checkpoint proc_lsst.multi INFO: found job 16964336 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:58 proc_lsst.multi:146 [INFO] found job 16964337 in provider checkpoint proc_lsst.multi INFO: found job 16964337 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.dataflow.dflow INFO: Task 5012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0ad46b40fa77e0b506efeadeb768904 with result from task 5012 parsl.dataflow.dflow DEBUG: Task 5012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stdout parsl.dataflow.dflow INFO: Standard error for task 5012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855425/f2766cbc-cc08-42bc-95a2-de44c88be9be_isrForCrosstalkSources_855425_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:58 proc_lsst.multi:146 [INFO] found job 16964338 in provider checkpoint proc_lsst.multi INFO: found job 16964338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:58 proc_lsst.multi:146 [INFO] found job 16964339 in provider checkpoint proc_lsst.multi INFO: found job 16964339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:59 proc_lsst.multi:146 [INFO] found job 16964340 in provider checkpoint proc_lsst.multi INFO: found job 16964340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:59 proc_lsst.multi:146 [INFO] found job 16964341 in provider checkpoint proc_lsst.multi INFO: found job 16964341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:59 proc_lsst.multi:146 [INFO] found job 16964342 in provider checkpoint proc_lsst.multi INFO: found job 16964342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:58:59 proc_lsst.multi:146 [INFO] found job 16964343 in provider checkpoint proc_lsst.multi INFO: found job 16964343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:00 proc_lsst.multi:146 [INFO] found job 16964344 in provider checkpoint proc_lsst.multi INFO: found job 16964344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:00 proc_lsst.multi:146 [INFO] found job 16964345 in provider checkpoint proc_lsst.multi INFO: found job 16964345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:00 proc_lsst.multi:146 [INFO] found job 16964346 in provider checkpoint proc_lsst.multi INFO: found job 16964346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:00 proc_lsst.multi:146 [INFO] found job 16964347 in provider checkpoint proc_lsst.multi INFO: found job 16964347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:01 proc_lsst.multi:146 [INFO] found job 16964348 in provider checkpoint proc_lsst.multi INFO: found job 16964348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964138 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 312, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 131/181 running/pending blocks, and 657 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22375328864144 2024-03-09 14:59:01 proc_lsst.multi:146 [INFO] found job 77954 in provider local proc_lsst.multi INFO: found job 77954 in provider local 2024-03-09 14:59:01 proc_lsst.multi:146 [INFO] found job 16964026 in provider astro proc_lsst.multi INFO: found job 16964026 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:01 proc_lsst.multi:146 [INFO] found job 16964027 in provider astro proc_lsst.multi INFO: found job 16964027 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:01 proc_lsst.multi:146 [INFO] found job 16964028 in provider astro proc_lsst.multi INFO: found job 16964028 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:02 proc_lsst.multi:146 [INFO] found job 16964029 in provider astro proc_lsst.multi INFO: found job 16964029 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:02 proc_lsst.multi:146 [INFO] found job 16964030 in provider astro proc_lsst.multi INFO: found job 16964030 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:02 proc_lsst.multi:146 [INFO] found job 16964031 in provider astro proc_lsst.multi INFO: found job 16964031 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:02 proc_lsst.multi:146 [INFO] found job 16964032 in provider astro proc_lsst.multi INFO: found job 16964032 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:02 proc_lsst.multi:146 [INFO] found job 16964033 in provider astro proc_lsst.multi INFO: found job 16964033 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 5011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4023f2cd117b8b85ee3a45c069d2ddfe with result from task 5011 parsl.dataflow.dflow DEBUG: Task 5011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855401/590dccaa-6227-4509-8bb0-1f226779d777_isrForCrosstalkSources_855401_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:03 proc_lsst.multi:146 [INFO] found job 16964034 in provider astro proc_lsst.multi INFO: found job 16964034 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:03 proc_lsst.multi:146 [INFO] found job 16964035 in provider astro proc_lsst.multi INFO: found job 16964035 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:03 proc_lsst.multi:146 [INFO] found job 16964036 in provider astro proc_lsst.multi INFO: found job 16964036 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:03 proc_lsst.multi:146 [INFO] found job 16964037 in provider astro proc_lsst.multi INFO: found job 16964037 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:04 proc_lsst.multi:146 [INFO] found job 16964038 in provider astro proc_lsst.multi INFO: found job 16964038 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:04 proc_lsst.multi:146 [INFO] found job 16964039 in provider astro proc_lsst.multi INFO: found job 16964039 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:04 proc_lsst.multi:146 [INFO] found job 16964040 in provider astro proc_lsst.multi INFO: found job 16964040 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:04 proc_lsst.multi:146 [INFO] found job 16964041 in provider astro proc_lsst.multi INFO: found job 16964041 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:04 proc_lsst.multi:146 [INFO] found job 16964042 in provider astro proc_lsst.multi INFO: found job 16964042 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:05 proc_lsst.multi:146 [INFO] found job 16964043 in provider astro proc_lsst.multi INFO: found job 16964043 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:05 proc_lsst.multi:146 [INFO] found job 16964044 in provider astro proc_lsst.multi INFO: found job 16964044 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:05 proc_lsst.multi:146 [INFO] found job 16964045 in provider astro proc_lsst.multi INFO: found job 16964045 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:05 proc_lsst.multi:146 [INFO] found job 16964046 in provider astro proc_lsst.multi INFO: found job 16964046 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:05 proc_lsst.multi:146 [INFO] found job 16964047 in provider astro proc_lsst.multi INFO: found job 16964047 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:06 proc_lsst.multi:146 [INFO] found job 16964048 in provider astro proc_lsst.multi INFO: found job 16964048 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:06 proc_lsst.multi:146 [INFO] found job 16964049 in provider astro proc_lsst.multi INFO: found job 16964049 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:06 proc_lsst.multi:146 [INFO] found job 16964050 in provider astro proc_lsst.multi INFO: found job 16964050 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.dataflow.dflow INFO: Task 5018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6691b4abf432ea6e9e37df7fb1b982 with result from task 5018 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 5021 parsl.dataflow.dflow INFO: Tasks in state States.failed: 1 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 5018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stdout parsl.dataflow.dflow INFO: Standard error for task 5018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/submit/DEEP/20190505/A1c/science#step0/20240309T144841Z/logs/isrForCrosstalkSources/855394/a8defe6e-8fbf-4f10-9a33-a165ba7c8747_isrForCrosstalkSources_855394_41.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:06 proc_lsst.multi:146 [INFO] found job 16964051 in provider astro proc_lsst.multi INFO: found job 16964051 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:07 proc_lsst.multi:146 [INFO] found job 16964052 in provider astro proc_lsst.multi INFO: found job 16964052 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:07 proc_lsst.multi:146 [INFO] found job 16964053 in provider astro proc_lsst.multi INFO: found job 16964053 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:07 proc_lsst.multi:146 [INFO] found job 16964054 in provider astro proc_lsst.multi INFO: found job 16964054 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:07 proc_lsst.multi:146 [INFO] found job 16964055 in provider astro proc_lsst.multi INFO: found job 16964055 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964026,16964027,16964028,16964029,16964030,16964031,16964032,16964033,16964034,16964035,16964036,16964037,16964038,16964039,16964040,16964041,16964042,16964043,16964044,16964045,16964046,16964047,16964048,16964049,16964050,16964051,16964052,16964053,16964054,16964055' parsl.providers.slurm.slurm DEBUG: squeue returned 16964054 R 16964055 R 16964048 R 16964049 R 16964050 R 16964051 R 16964052 R 16964053 R 16964043 R 16964044 R 16964045 R 16964046 R 16964047 R 16964038 R 16964039 R 16964040 R 16964041 R 16964042 R 16964035 R 16964036 R 16964037 R 16964033 R 16964034 R 16964026 R 16964027 R 16964028 R 16964029 R 16964030 R 16964031 R 16964032 R parsl.providers.slurm.slurm DEBUG: Updating job 16964054 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964055 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964048 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964049 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964050 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964051 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964052 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964053 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964043 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964044 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964045 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964046 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964047 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964038 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964039 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964040 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964041 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964042 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964035 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964036 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964037 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964033 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964034 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964026 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964027 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964028 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964029 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964030 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964031 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964032 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:07 proc_lsst.multi:146 [INFO] found job 16964056 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964056 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:08 proc_lsst.multi:146 [INFO] found job 16964057 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964057 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:08 proc_lsst.multi:146 [INFO] found job 16964058 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964058 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:08 proc_lsst.multi:146 [INFO] found job 16964059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964059 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:10 proc_lsst.multi:146 [INFO] found job 16964060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964060 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964056,16964057,16964058,16964059,16964060' parsl.providers.slurm.slurm DEBUG: squeue returned 16964058 R 16964059 R 16964060 R 16964056 R 16964057 R parsl.providers.slurm.slurm DEBUG: Updating job 16964058 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964059 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964060 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964056 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964057 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:10 proc_lsst.multi:146 [INFO] found job 16964061 in provider escience_a40 proc_lsst.multi INFO: found job 16964061 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:11 proc_lsst.multi:146 [INFO] found job 16964062 in provider escience_a40 proc_lsst.multi INFO: found job 16964062 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:11 proc_lsst.multi:146 [INFO] found job 16964063 in provider escience_a40 proc_lsst.multi INFO: found job 16964063 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:11 proc_lsst.multi:146 [INFO] found job 16964064 in provider escience_a40 proc_lsst.multi INFO: found job 16964064 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:11 proc_lsst.multi:146 [INFO] found job 16964065 in provider escience_a40 proc_lsst.multi INFO: found job 16964065 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:11 proc_lsst.multi:146 [INFO] found job 16964066 in provider escience_a40 proc_lsst.multi INFO: found job 16964066 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:12 proc_lsst.multi:146 [INFO] found job 16964067 in provider escience_a40 proc_lsst.multi INFO: found job 16964067 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:12 proc_lsst.multi:146 [INFO] found job 16964068 in provider escience_a40 proc_lsst.multi INFO: found job 16964068 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:12 proc_lsst.multi:146 [INFO] found job 16964069 in provider escience_a40 proc_lsst.multi INFO: found job 16964069 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:12 proc_lsst.multi:146 [INFO] found job 16964070 in provider escience_a40 proc_lsst.multi INFO: found job 16964070 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:12 proc_lsst.multi:146 [INFO] found job 16964071 in provider escience_a40 proc_lsst.multi INFO: found job 16964071 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:13 proc_lsst.multi:146 [INFO] found job 16964072 in provider escience_a40 proc_lsst.multi INFO: found job 16964072 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:13 proc_lsst.multi:146 [INFO] found job 16964073 in provider escience_a40 proc_lsst.multi INFO: found job 16964073 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:13 proc_lsst.multi:146 [INFO] found job 16964074 in provider escience_a40 proc_lsst.multi INFO: found job 16964074 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:13 proc_lsst.multi:146 [INFO] found job 16964075 in provider escience_a40 proc_lsst.multi INFO: found job 16964075 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:13 proc_lsst.multi:146 [INFO] found job 16964076 in provider escience_a40 proc_lsst.multi INFO: found job 16964076 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:14 proc_lsst.multi:146 [INFO] found job 16964077 in provider escience_a40 proc_lsst.multi INFO: found job 16964077 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:14 proc_lsst.multi:146 [INFO] found job 16964078 in provider escience_a40 proc_lsst.multi INFO: found job 16964078 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:14 proc_lsst.multi:146 [INFO] found job 16964079 in provider escience_a40 proc_lsst.multi INFO: found job 16964079 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:14 proc_lsst.multi:146 [INFO] found job 16964080 in provider escience_a40 proc_lsst.multi INFO: found job 16964080 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964061,16964062,16964063,16964064,16964065,16964066,16964067,16964068,16964069,16964070,16964071,16964072,16964073,16964074,16964075,16964076,16964077,16964078,16964079,16964080' parsl.providers.slurm.slurm DEBUG: squeue returned 16964079 R 16964080 R 16964073 R 16964074 R 16964075 R 16964076 R 16964077 R 16964078 R 16964069 R 16964070 R 16964071 R 16964072 R 16964064 R 16964065 R 16964066 R 16964067 R 16964068 R 16964061 R 16964062 R 16964063 R parsl.providers.slurm.slurm DEBUG: Updating job 16964079 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964080 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964073 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964074 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964075 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964076 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964077 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964078 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964069 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964070 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964071 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964072 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964064 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964065 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964066 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964067 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964068 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964061 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964062 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964063 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:15 proc_lsst.multi:146 [INFO] found job 16964081 in provider checkpoint proc_lsst.multi INFO: found job 16964081 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:15 proc_lsst.multi:146 [INFO] found job 16964082 in provider checkpoint proc_lsst.multi INFO: found job 16964082 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:15 proc_lsst.multi:146 [INFO] found job 16964083 in provider checkpoint proc_lsst.multi INFO: found job 16964083 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:15 proc_lsst.multi:146 [INFO] found job 16964084 in provider checkpoint proc_lsst.multi INFO: found job 16964084 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:16 proc_lsst.multi:146 [INFO] found job 16964085 in provider checkpoint proc_lsst.multi INFO: found job 16964085 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:16 proc_lsst.multi:146 [INFO] found job 16964086 in provider checkpoint proc_lsst.multi INFO: found job 16964086 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:16 proc_lsst.multi:146 [INFO] found job 16964087 in provider checkpoint proc_lsst.multi INFO: found job 16964087 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:16 proc_lsst.multi:146 [INFO] found job 16964088 in provider checkpoint proc_lsst.multi INFO: found job 16964088 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:17 proc_lsst.multi:146 [INFO] found job 16964089 in provider checkpoint proc_lsst.multi INFO: found job 16964089 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:17 proc_lsst.multi:146 [INFO] found job 16964090 in provider checkpoint proc_lsst.multi INFO: found job 16964090 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:17 proc_lsst.multi:146 [INFO] found job 16964091 in provider checkpoint proc_lsst.multi INFO: found job 16964091 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:17 proc_lsst.multi:146 [INFO] found job 16964092 in provider checkpoint proc_lsst.multi INFO: found job 16964092 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:17 proc_lsst.multi:146 [INFO] found job 16964093 in provider checkpoint proc_lsst.multi INFO: found job 16964093 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:18 proc_lsst.multi:146 [INFO] found job 16964094 in provider checkpoint proc_lsst.multi INFO: found job 16964094 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:18 proc_lsst.multi:146 [INFO] found job 16964095 in provider checkpoint proc_lsst.multi INFO: found job 16964095 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:18 proc_lsst.multi:146 [INFO] found job 16964096 in provider checkpoint proc_lsst.multi INFO: found job 16964096 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:18 proc_lsst.multi:146 [INFO] found job 16964097 in provider checkpoint proc_lsst.multi INFO: found job 16964097 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:19 proc_lsst.multi:146 [INFO] found job 16964099 in provider checkpoint proc_lsst.multi INFO: found job 16964099 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:19 proc_lsst.multi:146 [INFO] found job 16964100 in provider checkpoint proc_lsst.multi INFO: found job 16964100 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:19 proc_lsst.multi:146 [INFO] found job 16964101 in provider checkpoint proc_lsst.multi INFO: found job 16964101 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:19 proc_lsst.multi:146 [INFO] found job 16964102 in provider checkpoint proc_lsst.multi INFO: found job 16964102 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:20 proc_lsst.multi:146 [INFO] found job 16964103 in provider checkpoint proc_lsst.multi INFO: found job 16964103 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:20 proc_lsst.multi:146 [INFO] found job 16964104 in provider checkpoint proc_lsst.multi INFO: found job 16964104 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:20 proc_lsst.multi:146 [INFO] found job 16964105 in provider checkpoint proc_lsst.multi INFO: found job 16964105 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:20 proc_lsst.multi:146 [INFO] found job 16964106 in provider checkpoint proc_lsst.multi INFO: found job 16964106 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:21 proc_lsst.multi:146 [INFO] found job 16964107 in provider checkpoint proc_lsst.multi INFO: found job 16964107 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:21 proc_lsst.multi:146 [INFO] found job 16964108 in provider checkpoint proc_lsst.multi INFO: found job 16964108 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:21 proc_lsst.multi:146 [INFO] found job 16964109 in provider checkpoint proc_lsst.multi INFO: found job 16964109 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:21 proc_lsst.multi:146 [INFO] found job 16964110 in provider checkpoint proc_lsst.multi INFO: found job 16964110 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:22 proc_lsst.multi:146 [INFO] found job 16964111 in provider checkpoint proc_lsst.multi INFO: found job 16964111 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:22 proc_lsst.multi:146 [INFO] found job 16964112 in provider checkpoint proc_lsst.multi INFO: found job 16964112 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:22 proc_lsst.multi:146 [INFO] found job 16964113 in provider checkpoint proc_lsst.multi INFO: found job 16964113 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:22 proc_lsst.multi:146 [INFO] found job 16964114 in provider checkpoint proc_lsst.multi INFO: found job 16964114 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:23 proc_lsst.multi:146 [INFO] found job 16964115 in provider checkpoint proc_lsst.multi INFO: found job 16964115 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:23 proc_lsst.multi:146 [INFO] found job 16964116 in provider checkpoint proc_lsst.multi INFO: found job 16964116 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:23 proc_lsst.multi:146 [INFO] found job 16964117 in provider checkpoint proc_lsst.multi INFO: found job 16964117 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:23 proc_lsst.multi:146 [INFO] found job 16964118 in provider checkpoint proc_lsst.multi INFO: found job 16964118 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:24 proc_lsst.multi:146 [INFO] found job 16964119 in provider checkpoint proc_lsst.multi INFO: found job 16964119 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:24 proc_lsst.multi:146 [INFO] found job 16964120 in provider checkpoint proc_lsst.multi INFO: found job 16964120 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:24 proc_lsst.multi:146 [INFO] found job 16964121 in provider checkpoint proc_lsst.multi INFO: found job 16964121 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:24 proc_lsst.multi:146 [INFO] found job 16964122 in provider checkpoint proc_lsst.multi INFO: found job 16964122 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:25 proc_lsst.multi:146 [INFO] found job 16964123 in provider checkpoint proc_lsst.multi INFO: found job 16964123 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:25 proc_lsst.multi:146 [INFO] found job 16964124 in provider checkpoint proc_lsst.multi INFO: found job 16964124 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:25 proc_lsst.multi:146 [INFO] found job 16964125 in provider checkpoint proc_lsst.multi INFO: found job 16964125 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:25 proc_lsst.multi:146 [INFO] found job 16964126 in provider checkpoint proc_lsst.multi INFO: found job 16964126 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:26 proc_lsst.multi:146 [INFO] found job 16964127 in provider checkpoint proc_lsst.multi INFO: found job 16964127 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:26 proc_lsst.multi:146 [INFO] found job 16964128 in provider checkpoint proc_lsst.multi INFO: found job 16964128 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:26 proc_lsst.multi:146 [INFO] found job 16964129 in provider checkpoint proc_lsst.multi INFO: found job 16964129 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:26 proc_lsst.multi:146 [INFO] found job 16964130 in provider checkpoint proc_lsst.multi INFO: found job 16964130 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:27 proc_lsst.multi:146 [INFO] found job 16964131 in provider checkpoint proc_lsst.multi INFO: found job 16964131 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:27 proc_lsst.multi:146 [INFO] found job 16964132 in provider checkpoint proc_lsst.multi INFO: found job 16964132 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:27 proc_lsst.multi:146 [INFO] found job 16964133 in provider checkpoint proc_lsst.multi INFO: found job 16964133 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:27 proc_lsst.multi:146 [INFO] found job 16964134 in provider checkpoint proc_lsst.multi INFO: found job 16964134 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:28 proc_lsst.multi:146 [INFO] found job 16964135 in provider checkpoint proc_lsst.multi INFO: found job 16964135 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:28 proc_lsst.multi:146 [INFO] found job 16964136 in provider checkpoint proc_lsst.multi INFO: found job 16964136 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:28 proc_lsst.multi:146 [INFO] found job 16964137 in provider checkpoint proc_lsst.multi INFO: found job 16964137 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:28 proc_lsst.multi:146 [INFO] found job 16964138 in provider checkpoint proc_lsst.multi INFO: found job 16964138 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:29 proc_lsst.multi:146 [INFO] found job 16964139 in provider checkpoint proc_lsst.multi INFO: found job 16964139 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:29 proc_lsst.multi:146 [INFO] found job 16964140 in provider checkpoint proc_lsst.multi INFO: found job 16964140 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:29 proc_lsst.multi:146 [INFO] found job 16964141 in provider checkpoint proc_lsst.multi INFO: found job 16964141 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:29 proc_lsst.multi:146 [INFO] found job 16964142 in provider checkpoint proc_lsst.multi INFO: found job 16964142 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:30 proc_lsst.multi:146 [INFO] found job 16964143 in provider checkpoint proc_lsst.multi INFO: found job 16964143 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:30 proc_lsst.multi:146 [INFO] found job 16964144 in provider checkpoint proc_lsst.multi INFO: found job 16964144 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:30 proc_lsst.multi:146 [INFO] found job 16964145 in provider checkpoint proc_lsst.multi INFO: found job 16964145 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:30 proc_lsst.multi:146 [INFO] found job 16964146 in provider checkpoint proc_lsst.multi INFO: found job 16964146 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:31 proc_lsst.multi:146 [INFO] found job 16964147 in provider checkpoint proc_lsst.multi INFO: found job 16964147 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:31 proc_lsst.multi:146 [INFO] found job 16964148 in provider checkpoint proc_lsst.multi INFO: found job 16964148 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:31 proc_lsst.multi:146 [INFO] found job 16964149 in provider checkpoint proc_lsst.multi INFO: found job 16964149 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:31 proc_lsst.multi:146 [INFO] found job 16964150 in provider checkpoint proc_lsst.multi INFO: found job 16964150 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:32 proc_lsst.multi:146 [INFO] found job 16964151 in provider checkpoint proc_lsst.multi INFO: found job 16964151 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:32 proc_lsst.multi:146 [INFO] found job 16964152 in provider checkpoint proc_lsst.multi INFO: found job 16964152 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:32 proc_lsst.multi:146 [INFO] found job 16964153 in provider checkpoint proc_lsst.multi INFO: found job 16964153 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:32 proc_lsst.multi:146 [INFO] found job 16964154 in provider checkpoint proc_lsst.multi INFO: found job 16964154 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:33 proc_lsst.multi:146 [INFO] found job 16964155 in provider checkpoint proc_lsst.multi INFO: found job 16964155 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:33 proc_lsst.multi:146 [INFO] found job 16964156 in provider checkpoint proc_lsst.multi INFO: found job 16964156 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:33 proc_lsst.multi:146 [INFO] found job 16964157 in provider checkpoint proc_lsst.multi INFO: found job 16964157 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:33 proc_lsst.multi:146 [INFO] found job 16964158 in provider checkpoint proc_lsst.multi INFO: found job 16964158 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:34 proc_lsst.multi:146 [INFO] found job 16964159 in provider checkpoint proc_lsst.multi INFO: found job 16964159 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:34 proc_lsst.multi:146 [INFO] found job 16964160 in provider checkpoint proc_lsst.multi INFO: found job 16964160 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:34 proc_lsst.multi:146 [INFO] found job 16964161 in provider checkpoint proc_lsst.multi INFO: found job 16964161 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:34 proc_lsst.multi:146 [INFO] found job 16964162 in provider checkpoint proc_lsst.multi INFO: found job 16964162 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:35 proc_lsst.multi:146 [INFO] found job 16964163 in provider checkpoint proc_lsst.multi INFO: found job 16964163 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:35 proc_lsst.multi:146 [INFO] found job 16964164 in provider checkpoint proc_lsst.multi INFO: found job 16964164 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:35 proc_lsst.multi:146 [INFO] found job 16964165 in provider checkpoint proc_lsst.multi INFO: found job 16964165 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:35 proc_lsst.multi:146 [INFO] found job 16964166 in provider checkpoint proc_lsst.multi INFO: found job 16964166 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:36 proc_lsst.multi:146 [INFO] found job 16964167 in provider checkpoint proc_lsst.multi INFO: found job 16964167 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:36 proc_lsst.multi:146 [INFO] found job 16964168 in provider checkpoint proc_lsst.multi INFO: found job 16964168 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:36 proc_lsst.multi:146 [INFO] found job 16964169 in provider checkpoint proc_lsst.multi INFO: found job 16964169 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:36 proc_lsst.multi:146 [INFO] found job 16964170 in provider checkpoint proc_lsst.multi INFO: found job 16964170 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:37 proc_lsst.multi:146 [INFO] found job 16964171 in provider checkpoint proc_lsst.multi INFO: found job 16964171 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:37 proc_lsst.multi:146 [INFO] found job 16964172 in provider checkpoint proc_lsst.multi INFO: found job 16964172 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:37 proc_lsst.multi:146 [INFO] found job 16964173 in provider checkpoint proc_lsst.multi INFO: found job 16964173 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:37 proc_lsst.multi:146 [INFO] found job 16964174 in provider checkpoint proc_lsst.multi INFO: found job 16964174 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:38 proc_lsst.multi:146 [INFO] found job 16964175 in provider checkpoint proc_lsst.multi INFO: found job 16964175 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:38 proc_lsst.multi:146 [INFO] found job 16964176 in provider checkpoint proc_lsst.multi INFO: found job 16964176 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:38 proc_lsst.multi:146 [INFO] found job 16964177 in provider checkpoint proc_lsst.multi INFO: found job 16964177 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:38 proc_lsst.multi:146 [INFO] found job 16964178 in provider checkpoint proc_lsst.multi INFO: found job 16964178 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:39 proc_lsst.multi:146 [INFO] found job 16964179 in provider checkpoint proc_lsst.multi INFO: found job 16964179 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:39 proc_lsst.multi:146 [INFO] found job 16964180 in provider checkpoint proc_lsst.multi INFO: found job 16964180 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:39 proc_lsst.multi:146 [INFO] found job 16964181 in provider checkpoint proc_lsst.multi INFO: found job 16964181 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:39 proc_lsst.multi:146 [INFO] found job 16964182 in provider checkpoint proc_lsst.multi INFO: found job 16964182 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:39 proc_lsst.multi:146 [INFO] found job 16964183 in provider checkpoint proc_lsst.multi INFO: found job 16964183 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:40 proc_lsst.multi:146 [INFO] found job 16964184 in provider checkpoint proc_lsst.multi INFO: found job 16964184 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:40 proc_lsst.multi:146 [INFO] found job 16964185 in provider checkpoint proc_lsst.multi INFO: found job 16964185 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:40 proc_lsst.multi:146 [INFO] found job 16964186 in provider checkpoint proc_lsst.multi INFO: found job 16964186 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:40 proc_lsst.multi:146 [INFO] found job 16964187 in provider checkpoint proc_lsst.multi INFO: found job 16964187 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:41 proc_lsst.multi:146 [INFO] found job 16964188 in provider checkpoint proc_lsst.multi INFO: found job 16964188 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:43 proc_lsst.multi:146 [INFO] found job 16964189 in provider checkpoint proc_lsst.multi INFO: found job 16964189 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:43 proc_lsst.multi:146 [INFO] found job 16964190 in provider checkpoint proc_lsst.multi INFO: found job 16964190 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:43 proc_lsst.multi:146 [INFO] found job 16964191 in provider checkpoint proc_lsst.multi INFO: found job 16964191 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:45 proc_lsst.multi:146 [INFO] found job 16964192 in provider checkpoint proc_lsst.multi INFO: found job 16964192 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:46 proc_lsst.multi:146 [INFO] found job 16964193 in provider checkpoint proc_lsst.multi INFO: found job 16964193 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:46 proc_lsst.multi:146 [INFO] found job 16964194 in provider checkpoint proc_lsst.multi INFO: found job 16964194 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:46 proc_lsst.multi:146 [INFO] found job 16964195 in provider checkpoint proc_lsst.multi INFO: found job 16964195 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:46 proc_lsst.multi:146 [INFO] found job 16964196 in provider checkpoint proc_lsst.multi INFO: found job 16964196 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:47 proc_lsst.multi:146 [INFO] found job 16964197 in provider checkpoint proc_lsst.multi INFO: found job 16964197 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:47 proc_lsst.multi:146 [INFO] found job 16964198 in provider checkpoint proc_lsst.multi INFO: found job 16964198 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:47 proc_lsst.multi:146 [INFO] found job 16964199 in provider checkpoint proc_lsst.multi INFO: found job 16964199 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:47 proc_lsst.multi:146 [INFO] found job 16964200 in provider checkpoint proc_lsst.multi INFO: found job 16964200 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:48 proc_lsst.multi:146 [INFO] found job 16964201 in provider checkpoint proc_lsst.multi INFO: found job 16964201 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:48 proc_lsst.multi:146 [INFO] found job 16964202 in provider checkpoint proc_lsst.multi INFO: found job 16964202 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:48 proc_lsst.multi:146 [INFO] found job 16964203 in provider checkpoint proc_lsst.multi INFO: found job 16964203 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:48 proc_lsst.multi:146 [INFO] found job 16964204 in provider checkpoint proc_lsst.multi INFO: found job 16964204 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:49 proc_lsst.multi:146 [INFO] found job 16964205 in provider checkpoint proc_lsst.multi INFO: found job 16964205 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:49 proc_lsst.multi:146 [INFO] found job 16964206 in provider checkpoint proc_lsst.multi INFO: found job 16964206 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:49 proc_lsst.multi:146 [INFO] found job 16964207 in provider checkpoint proc_lsst.multi INFO: found job 16964207 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:49 proc_lsst.multi:146 [INFO] found job 16964208 in provider checkpoint proc_lsst.multi INFO: found job 16964208 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:50 proc_lsst.multi:146 [INFO] found job 16964209 in provider checkpoint proc_lsst.multi INFO: found job 16964209 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:50 proc_lsst.multi:146 [INFO] found job 16964210 in provider checkpoint proc_lsst.multi INFO: found job 16964210 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:50 proc_lsst.multi:146 [INFO] found job 16964211 in provider checkpoint proc_lsst.multi INFO: found job 16964211 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:50 proc_lsst.multi:146 [INFO] found job 16964212 in provider checkpoint proc_lsst.multi INFO: found job 16964212 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:51 proc_lsst.multi:146 [INFO] found job 16964213 in provider checkpoint proc_lsst.multi INFO: found job 16964213 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:51 proc_lsst.multi:146 [INFO] found job 16964214 in provider checkpoint proc_lsst.multi INFO: found job 16964214 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:51 proc_lsst.multi:146 [INFO] found job 16964215 in provider checkpoint proc_lsst.multi INFO: found job 16964215 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:51 proc_lsst.multi:146 [INFO] found job 16964216 in provider checkpoint proc_lsst.multi INFO: found job 16964216 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:52 proc_lsst.multi:146 [INFO] found job 16964217 in provider checkpoint proc_lsst.multi INFO: found job 16964217 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:52 proc_lsst.multi:146 [INFO] found job 16964218 in provider checkpoint proc_lsst.multi INFO: found job 16964218 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:52 proc_lsst.multi:146 [INFO] found job 16964220 in provider checkpoint proc_lsst.multi INFO: found job 16964220 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:52 proc_lsst.multi:146 [INFO] found job 16964221 in provider checkpoint proc_lsst.multi INFO: found job 16964221 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:53 proc_lsst.multi:146 [INFO] found job 16964222 in provider checkpoint proc_lsst.multi INFO: found job 16964222 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:53 proc_lsst.multi:146 [INFO] found job 16964223 in provider checkpoint proc_lsst.multi INFO: found job 16964223 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:53 proc_lsst.multi:146 [INFO] found job 16964224 in provider checkpoint proc_lsst.multi INFO: found job 16964224 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:53 proc_lsst.multi:146 [INFO] found job 16964225 in provider checkpoint proc_lsst.multi INFO: found job 16964225 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:54 proc_lsst.multi:146 [INFO] found job 16964226 in provider checkpoint proc_lsst.multi INFO: found job 16964226 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:54 proc_lsst.multi:146 [INFO] found job 16964227 in provider checkpoint proc_lsst.multi INFO: found job 16964227 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:54 proc_lsst.multi:146 [INFO] found job 16964228 in provider checkpoint proc_lsst.multi INFO: found job 16964228 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:54 proc_lsst.multi:146 [INFO] found job 16964229 in provider checkpoint proc_lsst.multi INFO: found job 16964229 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:55 proc_lsst.multi:146 [INFO] found job 16964230 in provider checkpoint proc_lsst.multi INFO: found job 16964230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:55 proc_lsst.multi:146 [INFO] found job 16964231 in provider checkpoint proc_lsst.multi INFO: found job 16964231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:55 proc_lsst.multi:146 [INFO] found job 16964232 in provider checkpoint proc_lsst.multi INFO: found job 16964232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:55 proc_lsst.multi:146 [INFO] found job 16964233 in provider checkpoint proc_lsst.multi INFO: found job 16964233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:55 proc_lsst.multi:146 [INFO] found job 16964234 in provider checkpoint proc_lsst.multi INFO: found job 16964234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:56 proc_lsst.multi:146 [INFO] found job 16964235 in provider checkpoint proc_lsst.multi INFO: found job 16964235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:56 proc_lsst.multi:146 [INFO] found job 16964236 in provider checkpoint proc_lsst.multi INFO: found job 16964236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:56 proc_lsst.multi:146 [INFO] found job 16964237 in provider checkpoint proc_lsst.multi INFO: found job 16964237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:56 proc_lsst.multi:146 [INFO] found job 16964238 in provider checkpoint proc_lsst.multi INFO: found job 16964238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:57 proc_lsst.multi:146 [INFO] found job 16964239 in provider checkpoint proc_lsst.multi INFO: found job 16964239 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:57 proc_lsst.multi:146 [INFO] found job 16964242 in provider checkpoint proc_lsst.multi INFO: found job 16964242 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:57 proc_lsst.multi:146 [INFO] found job 16964243 in provider checkpoint proc_lsst.multi INFO: found job 16964243 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:57 proc_lsst.multi:146 [INFO] found job 16964244 in provider checkpoint proc_lsst.multi INFO: found job 16964244 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:58 proc_lsst.multi:146 [INFO] found job 16964245 in provider checkpoint proc_lsst.multi INFO: found job 16964245 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:58 proc_lsst.multi:146 [INFO] found job 16964246 in provider checkpoint proc_lsst.multi INFO: found job 16964246 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:58 proc_lsst.multi:146 [INFO] found job 16964247 in provider checkpoint proc_lsst.multi INFO: found job 16964247 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:58 proc_lsst.multi:146 [INFO] found job 16964248 in provider checkpoint proc_lsst.multi INFO: found job 16964248 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:59 proc_lsst.multi:146 [INFO] found job 16964249 in provider checkpoint proc_lsst.multi INFO: found job 16964249 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:59 proc_lsst.multi:146 [INFO] found job 16964250 in provider checkpoint proc_lsst.multi INFO: found job 16964250 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:59 proc_lsst.multi:146 [INFO] found job 16964251 in provider checkpoint proc_lsst.multi INFO: found job 16964251 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 14:59:59 proc_lsst.multi:146 [INFO] found job 16964252 in provider checkpoint proc_lsst.multi INFO: found job 16964252 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:00 proc_lsst.multi:146 [INFO] found job 16964253 in provider checkpoint proc_lsst.multi INFO: found job 16964253 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:00 proc_lsst.multi:146 [INFO] found job 16964254 in provider checkpoint proc_lsst.multi INFO: found job 16964254 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:00 proc_lsst.multi:146 [INFO] found job 16964255 in provider checkpoint proc_lsst.multi INFO: found job 16964255 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:00 proc_lsst.multi:146 [INFO] found job 16964256 in provider checkpoint proc_lsst.multi INFO: found job 16964256 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:01 proc_lsst.multi:146 [INFO] found job 16964257 in provider checkpoint proc_lsst.multi INFO: found job 16964257 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:01 proc_lsst.multi:146 [INFO] found job 16964258 in provider checkpoint proc_lsst.multi INFO: found job 16964258 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:01 proc_lsst.multi:146 [INFO] found job 16964259 in provider checkpoint proc_lsst.multi INFO: found job 16964259 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:02 proc_lsst.multi:146 [INFO] found job 16964260 in provider checkpoint proc_lsst.multi INFO: found job 16964260 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:02 proc_lsst.multi:146 [INFO] found job 16964261 in provider checkpoint proc_lsst.multi INFO: found job 16964261 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:02 proc_lsst.multi:146 [INFO] found job 16964262 in provider checkpoint proc_lsst.multi INFO: found job 16964262 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:02 proc_lsst.multi:146 [INFO] found job 16964263 in provider checkpoint proc_lsst.multi INFO: found job 16964263 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:03 proc_lsst.multi:146 [INFO] found job 16964264 in provider checkpoint proc_lsst.multi INFO: found job 16964264 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:03 proc_lsst.multi:146 [INFO] found job 16964265 in provider checkpoint proc_lsst.multi INFO: found job 16964265 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:03 proc_lsst.multi:146 [INFO] found job 16964266 in provider checkpoint proc_lsst.multi INFO: found job 16964266 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:03 proc_lsst.multi:146 [INFO] found job 16964267 in provider checkpoint proc_lsst.multi INFO: found job 16964267 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:04 proc_lsst.multi:146 [INFO] found job 16964268 in provider checkpoint proc_lsst.multi INFO: found job 16964268 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:04 proc_lsst.multi:146 [INFO] found job 16964269 in provider checkpoint proc_lsst.multi INFO: found job 16964269 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:04 proc_lsst.multi:146 [INFO] found job 16964270 in provider checkpoint proc_lsst.multi INFO: found job 16964270 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:04 proc_lsst.multi:146 [INFO] found job 16964271 in provider checkpoint proc_lsst.multi INFO: found job 16964271 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:05 proc_lsst.multi:146 [INFO] found job 16964272 in provider checkpoint proc_lsst.multi INFO: found job 16964272 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:05 proc_lsst.multi:146 [INFO] found job 16964273 in provider checkpoint proc_lsst.multi INFO: found job 16964273 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:05 proc_lsst.multi:146 [INFO] found job 16964274 in provider checkpoint proc_lsst.multi INFO: found job 16964274 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:05 proc_lsst.multi:146 [INFO] found job 16964275 in provider checkpoint proc_lsst.multi INFO: found job 16964275 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:06 proc_lsst.multi:146 [INFO] found job 16964276 in provider checkpoint proc_lsst.multi INFO: found job 16964276 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:06 proc_lsst.multi:146 [INFO] found job 16964277 in provider checkpoint proc_lsst.multi INFO: found job 16964277 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:06 proc_lsst.multi:146 [INFO] found job 16964278 in provider checkpoint proc_lsst.multi INFO: found job 16964278 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:06 proc_lsst.multi:146 [INFO] found job 16964284 in provider checkpoint proc_lsst.multi INFO: found job 16964284 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:07 proc_lsst.multi:146 [INFO] found job 16964286 in provider checkpoint proc_lsst.multi INFO: found job 16964286 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:07 proc_lsst.multi:146 [INFO] found job 16964287 in provider checkpoint proc_lsst.multi INFO: found job 16964287 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:07 proc_lsst.multi:146 [INFO] found job 16964288 in provider checkpoint proc_lsst.multi INFO: found job 16964288 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:08 proc_lsst.multi:146 [INFO] found job 16964289 in provider checkpoint proc_lsst.multi INFO: found job 16964289 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:08 proc_lsst.multi:146 [INFO] found job 16964290 in provider checkpoint proc_lsst.multi INFO: found job 16964290 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:08 proc_lsst.multi:146 [INFO] found job 16964291 in provider checkpoint proc_lsst.multi INFO: found job 16964291 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:08 proc_lsst.multi:146 [INFO] found job 16964292 in provider checkpoint proc_lsst.multi INFO: found job 16964292 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:09 proc_lsst.multi:146 [INFO] found job 16964293 in provider checkpoint proc_lsst.multi INFO: found job 16964293 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:09 proc_lsst.multi:146 [INFO] found job 16964294 in provider checkpoint proc_lsst.multi INFO: found job 16964294 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:09 proc_lsst.multi:146 [INFO] found job 16964295 in provider checkpoint proc_lsst.multi INFO: found job 16964295 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:09 proc_lsst.multi:146 [INFO] found job 16964296 in provider checkpoint proc_lsst.multi INFO: found job 16964296 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:10 proc_lsst.multi:146 [INFO] found job 16964297 in provider checkpoint proc_lsst.multi INFO: found job 16964297 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:10 proc_lsst.multi:146 [INFO] found job 16964298 in provider checkpoint proc_lsst.multi INFO: found job 16964298 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:10 proc_lsst.multi:146 [INFO] found job 16964299 in provider checkpoint proc_lsst.multi INFO: found job 16964299 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:10 proc_lsst.multi:146 [INFO] found job 16964300 in provider checkpoint proc_lsst.multi INFO: found job 16964300 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:10 proc_lsst.multi:146 [INFO] found job 16964301 in provider checkpoint proc_lsst.multi INFO: found job 16964301 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:11 proc_lsst.multi:146 [INFO] found job 16964304 in provider checkpoint proc_lsst.multi INFO: found job 16964304 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:11 proc_lsst.multi:146 [INFO] found job 16964305 in provider checkpoint proc_lsst.multi INFO: found job 16964305 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:11 proc_lsst.multi:146 [INFO] found job 16964306 in provider checkpoint proc_lsst.multi INFO: found job 16964306 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:11 proc_lsst.multi:146 [INFO] found job 16964307 in provider checkpoint proc_lsst.multi INFO: found job 16964307 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:12 proc_lsst.multi:146 [INFO] found job 16964308 in provider checkpoint proc_lsst.multi INFO: found job 16964308 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:12 proc_lsst.multi:146 [INFO] found job 16964309 in provider checkpoint proc_lsst.multi INFO: found job 16964309 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:12 proc_lsst.multi:146 [INFO] found job 16964310 in provider checkpoint proc_lsst.multi INFO: found job 16964310 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:12 proc_lsst.multi:146 [INFO] found job 16964311 in provider checkpoint proc_lsst.multi INFO: found job 16964311 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:13 proc_lsst.multi:146 [INFO] found job 16964312 in provider checkpoint proc_lsst.multi INFO: found job 16964312 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:13 proc_lsst.multi:146 [INFO] found job 16964313 in provider checkpoint proc_lsst.multi INFO: found job 16964313 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:13 proc_lsst.multi:146 [INFO] found job 16964314 in provider checkpoint proc_lsst.multi INFO: found job 16964314 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:13 proc_lsst.multi:146 [INFO] found job 16964315 in provider checkpoint proc_lsst.multi INFO: found job 16964315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:14 proc_lsst.multi:146 [INFO] found job 16964316 in provider checkpoint proc_lsst.multi INFO: found job 16964316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:14 proc_lsst.multi:146 [INFO] found job 16964317 in provider checkpoint proc_lsst.multi INFO: found job 16964317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:14 proc_lsst.multi:146 [INFO] found job 16964318 in provider checkpoint proc_lsst.multi INFO: found job 16964318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:14 proc_lsst.multi:146 [INFO] found job 16964319 in provider checkpoint proc_lsst.multi INFO: found job 16964319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:15 proc_lsst.multi:146 [INFO] found job 16964320 in provider checkpoint proc_lsst.multi INFO: found job 16964320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:15 proc_lsst.multi:146 [INFO] found job 16964321 in provider checkpoint proc_lsst.multi INFO: found job 16964321 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:15 proc_lsst.multi:146 [INFO] found job 16964322 in provider checkpoint proc_lsst.multi INFO: found job 16964322 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:15 proc_lsst.multi:146 [INFO] found job 16964323 in provider checkpoint proc_lsst.multi INFO: found job 16964323 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:16 proc_lsst.multi:146 [INFO] found job 16964324 in provider checkpoint proc_lsst.multi INFO: found job 16964324 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:16 proc_lsst.multi:146 [INFO] found job 16964325 in provider checkpoint proc_lsst.multi INFO: found job 16964325 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:18 proc_lsst.multi:146 [INFO] found job 16964326 in provider checkpoint proc_lsst.multi INFO: found job 16964326 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:18 proc_lsst.multi:146 [INFO] found job 16964327 in provider checkpoint proc_lsst.multi INFO: found job 16964327 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964213 PD 16964212 PD 16964211 PD 16964210 PD 16964209 PD 16964208 PD 16964207 PD 16964206 PD 16964205 PD 16964204 PD 16964203 PD 16964202 PD 16964201 PD 16964200 PD 16964199 PD 16964198 PD 16964197 PD 16964196 PD 16964195 PD 16964194 PD 16964193 PD 16964192 PD 16964191 PD 16964190 PD 16964189 PD 16964188 PD 16964187 PD 16964186 PD 16964185 PD 16964184 PD 16964183 PD 16964182 PD 16964181 PD 16964180 PD 16964179 PD 16964178 PD 16964177 PD 16964176 PD 16964175 PD 16964174 PD 16964173 PD 16964172 PD 16964171 PD 16964170 PD 16964169 PD 16964168 PD 16964167 PD 16964166 PD 16964165 PD 16964164 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:19 proc_lsst.multi:146 [INFO] found job 16964328 in provider checkpoint proc_lsst.multi INFO: found job 16964328 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:21 proc_lsst.multi:146 [INFO] found job 16964329 in provider checkpoint proc_lsst.multi INFO: found job 16964329 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:21 proc_lsst.multi:146 [INFO] found job 16964330 in provider checkpoint proc_lsst.multi INFO: found job 16964330 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:21 proc_lsst.multi:146 [INFO] found job 16964331 in provider checkpoint proc_lsst.multi INFO: found job 16964331 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:22 proc_lsst.multi:146 [INFO] found job 16964332 in provider checkpoint proc_lsst.multi INFO: found job 16964332 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:22 proc_lsst.multi:146 [INFO] found job 16964333 in provider checkpoint proc_lsst.multi INFO: found job 16964333 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:22 proc_lsst.multi:146 [INFO] found job 16964334 in provider checkpoint proc_lsst.multi INFO: found job 16964334 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:22 proc_lsst.multi:146 [INFO] found job 16964335 in provider checkpoint proc_lsst.multi INFO: found job 16964335 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:23 proc_lsst.multi:146 [INFO] found job 16964336 in provider checkpoint proc_lsst.multi INFO: found job 16964336 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:23 proc_lsst.multi:146 [INFO] found job 16964337 in provider checkpoint proc_lsst.multi INFO: found job 16964337 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:23 proc_lsst.multi:146 [INFO] found job 16964338 in provider checkpoint proc_lsst.multi INFO: found job 16964338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:23 proc_lsst.multi:146 [INFO] found job 16964339 in provider checkpoint proc_lsst.multi INFO: found job 16964339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:24 proc_lsst.multi:146 [INFO] found job 16964340 in provider checkpoint proc_lsst.multi INFO: found job 16964340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:24 proc_lsst.multi:146 [INFO] found job 16964341 in provider checkpoint proc_lsst.multi INFO: found job 16964341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:24 proc_lsst.multi:146 [INFO] found job 16964342 in provider checkpoint proc_lsst.multi INFO: found job 16964342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:24 proc_lsst.multi:146 [INFO] found job 16964343 in provider checkpoint proc_lsst.multi INFO: found job 16964343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:25 proc_lsst.multi:146 [INFO] found job 16964344 in provider checkpoint proc_lsst.multi INFO: found job 16964344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:25 proc_lsst.multi:146 [INFO] found job 16964345 in provider checkpoint proc_lsst.multi INFO: found job 16964345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:25 proc_lsst.multi:146 [INFO] found job 16964346 in provider checkpoint proc_lsst.multi INFO: found job 16964346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:25 proc_lsst.multi:146 [INFO] found job 16964347 in provider checkpoint proc_lsst.multi INFO: found job 16964347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING 2024-03-09 15:00:26 proc_lsst.multi:146 [INFO] found job 16964348 in provider checkpoint proc_lsst.multi INFO: found job 16964348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16964081,16964082,16964083,16964084,16964085,16964086,16964087,16964088,16964089,16964090,16964091,16964092,16964093,16964094,16964095,16964096,16964097,16964099,16964100,16964101,16964102,16964103,16964104,16964105,16964106,16964107,16964108,16964109,16964110,16964111,16964112,16964113,16964114,16964115,16964116,16964117,16964118,16964119,16964120,16964121,16964122,16964123,16964124,16964125,16964126,16964127,16964128,16964129,16964130,16964131,16964132,16964133,16964134,16964135,16964136,16964137,16964138,16964139,16964140,16964141,16964142,16964143,16964144,16964145,16964146,16964147,16964148,16964149,16964150,16964151,16964152,16964153,16964154,16964155,16964156,16964157,16964158,16964159,16964160,16964161,16964162,16964163,16964164,16964165,16964166,16964167,16964168,16964169,16964170,16964171,16964172,16964173,16964174,16964175,16964176,16964177,16964178,16964179,16964180,16964181,16964182,16964183,16964184,16964185,16964186,16964187,16964188,16964189,16964190,16964191,16964192,16964193,16964194,16964195,16964196,16964197,16964198,16964199,16964200,16964201,16964202,16964203,16964204,16964205,16964206,16964207,16964208,16964209,16964210,16964211,16964212,16964213,16964214,16964215,16964216,16964217,16964218,16964220,16964221,16964222,16964223,16964224,16964225,16964226,16964227,16964228,16964229,16964230,16964231,16964232,16964233,16964234,16964235,16964236,16964237,16964238,16964239,16964242,16964243,16964244,16964245,16964246,16964247,16964248,16964249,16964250,16964251,16964252,16964253,16964254,16964255,16964256,16964257,16964258,16964259,16964260,16964261,16964262,16964263,16964264,16964265,16964266,16964267,16964268,16964269,16964270,16964271,16964272,16964273,16964274,16964275,16964276,16964277,16964278,16964284,16964286,16964287,16964288,16964289,16964290,16964291,16964292,16964293,16964294,16964295,16964296,16964297,16964298,16964299,16964300,16964301,16964304,16964305,16964306,16964307,16964308,16964309,16964310,16964311,16964312,16964313,16964314,16964315,16964316,16964317,16964318,16964319,16964320,16964321,16964322,16964323,16964324,16964325,16964326,16964327,16964328,16964329,16964330,16964331,16964332,16964333,16964334,16964335,16964336,16964337,16964338,16964339,16964340,16964341,16964342,16964343,16964344,16964345,16964346,16964347,16964348' parsl.providers.slurm.slurm DEBUG: squeue returned 16964130 PD 16964131 PD 16964132 PD 16964133 PD 16964164 PD 16964165 PD 16964166 PD 16964167 PD 16964168 PD 16964169 PD 16964170 PD 16964171 PD 16964172 PD 16964173 PD 16964174 PD 16964175 PD 16964176 PD 16964177 PD 16964178 PD 16964179 PD 16964180 PD 16964181 PD 16964182 PD 16964183 PD 16964184 PD 16964185 PD 16964186 PD 16964187 PD 16964188 PD 16964189 PD 16964190 PD 16964191 PD 16964192 PD 16964193 PD 16964194 PD 16964195 PD 16964196 PD 16964197 PD 16964198 PD 16964199 PD 16964200 PD 16964201 PD 16964202 PD 16964203 PD 16964204 PD 16964205 PD 16964206 PD 16964207 PD 16964208 PD 16964209 PD 16964210 PD 16964211 PD 16964212 PD 16964213 PD 16964348 PD 16964347 PD 16964346 PD 16964345 PD 16964344 PD 16964343 PD 16964342 PD 16964341 PD 16964340 PD 16964339 PD 16964338 PD 16964337 PD 16964336 PD 16964335 PD 16964334 PD 16964333 PD 16964332 PD 16964331 PD 16964330 PD 16964329 PD 16964328 PD 16964327 PD 16964326 PD 16964325 PD 16964324 PD 16964323 PD 16964322 PD 16964321 PD 16964320 PD 16964319 PD 16964318 PD 16964317 PD 16964316 PD 16964315 PD 16964314 PD 16964313 PD 16964312 PD 16964311 PD 16964310 PD 16964309 PD 16964308 PD 16964307 PD 16964306 PD 16964305 PD 16964304 PD 16964301 PD 16964300 PD 16964299 PD 16964298 PD 16964297 PD 16964296 PD 16964295 PD 16964294 PD 16964293 PD 16964292 PD 16964291 PD 16964290 PD 16964289 PD 16964288 PD 16964287 PD 16964286 PD 16964284 PD 16964278 PD 16964277 PD 16964276 PD 16964275 PD 16964274 PD 16964273 PD 16964272 PD 16964271 PD 16964270 PD 16964269 PD 16964268 PD 16964267 PD 16964266 PD 16964265 PD 16964264 PD 16964263 PD 16964262 PD 16964261 PD 16964260 PD 16964259 PD 16964258 PD 16964257 PD 16964256 PD 16964255 PD 16964254 PD 16964253 PD 16964252 PD 16964251 PD 16964250 PD 16964249 PD 16964248 PD 16964247 PD 16964246 PD 16964245 PD 16964244 PD 16964243 PD 16964242 PD 16964239 PD 16964238 PD 16964237 PD 16964236 PD 16964235 PD 16964234 PD 16964233 PD 16964232 PD 16964231 PD 16964230 PD 16964229 PD 16964228 PD 16964227 PD 16964226 PD 16964225 PD 16964224 PD 16964223 PD 16964222 PD 16964221 PD 16964220 PD 16964218 PD 16964217 PD 16964216 PD 16964215 PD 16964214 PD 16964144 PD 16964143 PD 16964142 PD 16964141 PD 16964138 PD 16964126 PD 16964125 PD 16964163 R 16964153 R 16964154 R 16964155 R 16964156 R 16964157 R 16964158 R 16964159 R 16964160 R 16964161 R 16964162 R 16964152 R 16964124 R 16964127 R 16964128 R 16964129 R 16964134 R 16964135 R 16964136 R 16964137 R 16964139 R 16964140 R 16964145 R 16964146 R 16964147 R 16964148 R 16964149 R 16964150 R 16964151 R 16964081 R 16964082 R 16964083 R 16964084 R 16964085 R 16964086 R 16964087 R 16964088 R 16964089 R 16964090 R 16964091 R 16964092 R 16964093 R 16964094 R 16964095 R 16964096 R 16964097 R 16964099 R 16964100 R 16964101 R 16964102 R 16964103 R 16964104 R 16964105 R 16964106 R 16964107 R 16964108 R 16964109 R 16964110 R 16964111 R 16964112 R 16964113 R 16964114 R 16964115 R 16964116 R 16964117 R 16964118 R 16964119 R 16964120 R 16964121 R 16964122 R 16964123 R parsl.providers.slurm.slurm DEBUG: Updating job 16964130 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964131 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964132 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964133 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964164 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964165 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964166 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964167 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964168 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964169 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964170 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964171 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964172 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964173 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964174 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964175 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964176 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964177 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964178 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964179 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964180 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964181 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964182 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964183 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964184 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964185 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964186 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964187 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964188 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964189 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964190 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964191 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964192 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964193 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964194 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964195 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964196 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964197 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964198 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964199 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964200 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964201 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964202 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964203 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964204 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964205 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964206 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964207 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964208 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964209 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964210 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964211 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964212 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964213 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964348 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964347 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964346 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964345 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964344 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964343 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964342 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964341 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964340 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964339 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964338 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964337 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964336 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964335 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964334 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964333 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964332 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964331 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964330 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964329 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964328 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964327 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964326 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964325 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964324 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964323 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964322 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964321 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964320 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964319 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964318 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964317 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964316 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964315 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964314 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964313 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964312 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964311 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964310 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964309 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964308 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964307 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964306 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964305 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964304 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964301 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964300 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964299 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964298 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964297 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964296 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964295 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964294 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964293 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964292 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964291 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964290 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964289 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964288 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964287 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964286 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964284 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964278 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964277 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964276 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964275 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964274 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964273 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964272 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964271 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964270 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964269 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964268 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964267 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964266 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964265 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964264 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964263 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964262 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964261 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964260 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964259 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964258 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964257 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964256 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964255 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964254 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964253 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964252 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964251 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964250 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964249 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964248 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964247 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964246 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964245 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964244 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964243 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964242 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964239 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964238 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964236 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964235 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964234 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964233 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964232 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964231 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964230 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964229 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964228 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964227 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964226 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964225 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964224 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964223 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964222 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964221 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964220 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964218 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964217 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964216 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964215 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964214 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964144 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964143 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964142 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964141 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964138 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964126 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964125 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16964163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964153 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964154 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964156 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964158 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964160 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964152 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964124 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964128 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964134 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964136 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964139 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964147 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964148 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964150 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964081 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964082 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964083 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964084 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964085 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964086 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964087 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964091 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964092 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964095 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964102 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964105 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964107 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964109 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964111 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964112 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964113 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964114 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964116 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964122 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16964123 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 312, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 127/185 running/pending blocks, and 625 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (312) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 3.0994415283203125e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22375328864144 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=312, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 127 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0733c57c3f40 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0733c57c3f40 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb4bdd9d5000 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb4bdd9d5000 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aed635aa95a6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aed635aa95a6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8ce5d22d149 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8ce5d22d149 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c31e03554530 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c31e03554530 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a0d5f6b92d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a0d5f6b92d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c78cd257216b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c78cd257216b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8e4c5417135c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8e4c5417135c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 676b013a1310 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 676b013a1310 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a405b074aa5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a405b074aa5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3e6ec2357c9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3e6ec2357c9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a1537776714 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a1537776714 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8a69d48938d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8a69d48938d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6ebe348f2d8b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6ebe348f2d8b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0fe7856ffa0e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0fe7856ffa0e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e7e53db4aac5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e7e53db4aac5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c1e0f2a4419f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c1e0f2a4419f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d6858b58faa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d6858b58faa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd7a7e1a85ce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd7a7e1a85ce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe68341a4ce5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe68341a4ce5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 37517df0aaa0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 37517df0aaa0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1fd4d9ccdb24 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1fd4d9ccdb24 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbcf02898c6d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbcf02898c6d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e1c7282d121 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e1c7282d121 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3367210c36a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3367210c36a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c841c0252606 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c841c0252606 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00ee20c18cfc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00ee20c18cfc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41082d299ade parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41082d299ade parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78be259d8b88 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78be259d8b88 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2d472184107 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2d472184107 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 739356491c41 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 739356491c41 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c85694f9c86 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c85694f9c86 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfceff14f627 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfceff14f627 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6682aeb2965e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6682aeb2965e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f7656a39463 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f7656a39463 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9a10d0ed57c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9a10d0ed57c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ab3ee4b6a99c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ab3ee4b6a99c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 26296784f0be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 26296784f0be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e4e248d2a2a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e4e248d2a2a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da014d217da0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da014d217da0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7268ca92ca1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7268ca92ca1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b10fa8b07397 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b10fa8b07397 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a892a2fc7c85 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a892a2fc7c85 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 210f45748382 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 210f45748382 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2539a43370ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2539a43370ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fcf84176f2aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fcf84176f2aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 407e102291b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 407e102291b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1541aeb7e3bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1541aeb7e3bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 332ffe5b2b2b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 332ffe5b2b2b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67359270e40f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67359270e40f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 565248f43c53 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 565248f43c53 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d5ae72dc594 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d5ae72dc594 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75de1d6db92d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75de1d6db92d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3b1457ae4ed6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3b1457ae4ed6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa01740a5216 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa01740a5216 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a39ccffe057 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a39ccffe057 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a80a1f1c138b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a80a1f1c138b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e16a158e26df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e16a158e26df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dae0b99bf4e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dae0b99bf4e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bc4ffcf3be1e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bc4ffcf3be1e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bffab73187ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bffab73187ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2aa725a08eff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2aa725a08eff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 99b3523e4b81 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 99b3523e4b81 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad2b24e4d2e7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad2b24e4d2e7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f219789bb835 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f219789bb835 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a93f7e47f9b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a93f7e47f9b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 929d42ebeb05 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 929d42ebeb05 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c512a98dbe92 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c512a98dbe92 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2d3767e8098e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2d3767e8098e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77eaa6aa0f31 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77eaa6aa0f31 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 90b8cd902eaa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 90b8cd902eaa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a1c1a3a0a997 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a1c1a3a0a997 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bac72ee3dca9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bac72ee3dca9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 147e15649532 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 147e15649532 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bed68f286aa7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bed68f286aa7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dbe208290144 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dbe208290144 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e8667930585 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e8667930585 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce879e9eae15 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce879e9eae15 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ecc55b37f07 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ecc55b37f07 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e5270ea6eb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e5270ea6eb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d36ed4a1d23 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d36ed4a1d23 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 524b2528b0d8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 524b2528b0d8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6196e92e7c8f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6196e92e7c8f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd2296ee3dda parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd2296ee3dda parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ac1371c802cb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ac1371c802cb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 217cf9ec127d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 217cf9ec127d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 637d327d35b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 637d327d35b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f3f5620b0ba9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f3f5620b0ba9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 432d9007fee3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 432d9007fee3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abdb35f74e59 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abdb35f74e59 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 622356fb9ac6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 622356fb9ac6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87970b05a7ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87970b05a7ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 467a465e6a68 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 467a465e6a68 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5e6b34e2ed2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5e6b34e2ed2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8656da0953ec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8656da0953ec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e54fb6eac438 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e54fb6eac438 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2fd1bf73f31b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2fd1bf73f31b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 605956606465 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 605956606465 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0b84ac3cb2b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0b84ac3cb2b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fc34fe9ac777 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fc34fe9ac777 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eeeb1a4d45d6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eeeb1a4d45d6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f8d8ea4725d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f8d8ea4725d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff745047609e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff745047609e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e3a7ff318c0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e3a7ff318c0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: afc4d8c60776 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: afc4d8c60776 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2c01c37cb351 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2c01c37cb351 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 255dfda1a5ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 255dfda1a5ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4b588f95043 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4b588f95043 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51275d6d81f7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51275d6d81f7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76bb4cfc4701 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76bb4cfc4701 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a7e6887de4d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a7e6887de4d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ed1b56d7605 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ed1b56d7605 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6c318b974f6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6c318b974f6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68eeade29ce0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68eeade29ce0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 460b2959e8d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 460b2959e8d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 89d4dfa8dfac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 89d4dfa8dfac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 34745a774a4a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 34745a774a4a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f3dbe2729f5e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f3dbe2729f5e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec379dd988d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec379dd988d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d13406d5665 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d13406d5665 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f128027ad85 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f128027ad85 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f026358b4f4e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f026358b4f4e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc1687d24b07 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc1687d24b07 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 523211cb0bb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 523211cb0bb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a7e1ad7d18cf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a7e1ad7d18cf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79d125e47a55 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79d125e47a55 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 28c7a5e18d45 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 28c7a5e18d45 2024-03-09 15:00:26 proc_lsst.multi:146 [INFO] found job 16964099 in provider checkpoint proc_lsst.multi INFO: found job 16964099 in provider checkpoint 2024-03-09 15:00:26 proc_lsst.multi:201 [INFO] cancelling 16964099 on provider checkpoint proc_lsst.multi INFO: cancelling 16964099 on provider checkpoint 2024-03-09 15:00:26 proc_lsst.multi:146 [INFO] found job 16964089 in provider checkpoint proc_lsst.multi INFO: found job 16964089 in provider checkpoint 2024-03-09 15:00:26 proc_lsst.multi:201 [INFO] cancelling 16964089 on provider checkpoint proc_lsst.multi INFO: cancelling 16964089 on provider checkpoint 2024-03-09 15:00:26 proc_lsst.multi:146 [INFO] found job 16964067 in provider escience_a40 proc_lsst.multi INFO: found job 16964067 in provider escience_a40 2024-03-09 15:00:26 proc_lsst.multi:201 [INFO] cancelling 16964067 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964067 on provider escience_a40 2024-03-09 15:00:27 proc_lsst.multi:146 [INFO] found job 16964163 in provider checkpoint proc_lsst.multi INFO: found job 16964163 in provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:201 [INFO] cancelling 16964163 on provider checkpoint proc_lsst.multi INFO: cancelling 16964163 on provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:146 [INFO] found job 16964137 in provider checkpoint proc_lsst.multi INFO: found job 16964137 in provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:201 [INFO] cancelling 16964137 on provider checkpoint proc_lsst.multi INFO: cancelling 16964137 on provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:146 [INFO] found job 16964101 in provider checkpoint proc_lsst.multi INFO: found job 16964101 in provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:201 [INFO] cancelling 16964101 on provider checkpoint proc_lsst.multi INFO: cancelling 16964101 on provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:146 [INFO] found job 16964116 in provider checkpoint proc_lsst.multi INFO: found job 16964116 in provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:201 [INFO] cancelling 16964116 on provider checkpoint proc_lsst.multi INFO: cancelling 16964116 on provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:146 [INFO] found job 16964070 in provider escience_a40 proc_lsst.multi INFO: found job 16964070 in provider escience_a40 2024-03-09 15:00:27 proc_lsst.multi:201 [INFO] cancelling 16964070 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964070 on provider escience_a40 2024-03-09 15:00:27 proc_lsst.multi:146 [INFO] found job 16964083 in provider checkpoint proc_lsst.multi INFO: found job 16964083 in provider checkpoint 2024-03-09 15:00:27 proc_lsst.multi:201 [INFO] cancelling 16964083 on provider checkpoint proc_lsst.multi INFO: cancelling 16964083 on provider checkpoint 2024-03-09 15:00:28 proc_lsst.multi:146 [INFO] found job 16964075 in provider escience_a40 proc_lsst.multi INFO: found job 16964075 in provider escience_a40 2024-03-09 15:00:28 proc_lsst.multi:201 [INFO] cancelling 16964075 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964075 on provider escience_a40 2024-03-09 15:00:28 proc_lsst.multi:146 [INFO] found job 16964078 in provider escience_a40 proc_lsst.multi INFO: found job 16964078 in provider escience_a40 2024-03-09 15:00:28 proc_lsst.multi:201 [INFO] cancelling 16964078 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964078 on provider escience_a40 2024-03-09 15:00:28 proc_lsst.multi:146 [INFO] found job 16964156 in provider checkpoint proc_lsst.multi INFO: found job 16964156 in provider checkpoint 2024-03-09 15:00:28 proc_lsst.multi:201 [INFO] cancelling 16964156 on provider checkpoint proc_lsst.multi INFO: cancelling 16964156 on provider checkpoint 2024-03-09 15:00:28 proc_lsst.multi:146 [INFO] found job 16964157 in provider checkpoint proc_lsst.multi INFO: found job 16964157 in provider checkpoint 2024-03-09 15:00:28 proc_lsst.multi:201 [INFO] cancelling 16964157 on provider checkpoint proc_lsst.multi INFO: cancelling 16964157 on provider checkpoint 2024-03-09 15:00:28 proc_lsst.multi:146 [INFO] found job 16964158 in provider checkpoint proc_lsst.multi INFO: found job 16964158 in provider checkpoint 2024-03-09 15:00:28 proc_lsst.multi:201 [INFO] cancelling 16964158 on provider checkpoint proc_lsst.multi INFO: cancelling 16964158 on provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:146 [INFO] found job 16964159 in provider checkpoint proc_lsst.multi INFO: found job 16964159 in provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:201 [INFO] cancelling 16964159 on provider checkpoint proc_lsst.multi INFO: cancelling 16964159 on provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:146 [INFO] found job 16964155 in provider checkpoint proc_lsst.multi INFO: found job 16964155 in provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:201 [INFO] cancelling 16964155 on provider checkpoint proc_lsst.multi INFO: cancelling 16964155 on provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:146 [INFO] found job 16964161 in provider checkpoint proc_lsst.multi INFO: found job 16964161 in provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:201 [INFO] cancelling 16964161 on provider checkpoint proc_lsst.multi INFO: cancelling 16964161 on provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:146 [INFO] found job 16964160 in provider checkpoint proc_lsst.multi INFO: found job 16964160 in provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:201 [INFO] cancelling 16964160 on provider checkpoint proc_lsst.multi INFO: cancelling 16964160 on provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:146 [INFO] found job 16964140 in provider checkpoint proc_lsst.multi INFO: found job 16964140 in provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:201 [INFO] cancelling 16964140 on provider checkpoint proc_lsst.multi INFO: cancelling 16964140 on provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:146 [INFO] found job 16964139 in provider checkpoint proc_lsst.multi INFO: found job 16964139 in provider checkpoint 2024-03-09 15:00:29 proc_lsst.multi:201 [INFO] cancelling 16964139 on provider checkpoint proc_lsst.multi INFO: cancelling 16964139 on provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:146 [INFO] found job 16964061 in provider escience_a40 proc_lsst.multi INFO: found job 16964061 in provider escience_a40 2024-03-09 15:00:30 proc_lsst.multi:201 [INFO] cancelling 16964061 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964061 on provider escience_a40 2024-03-09 15:00:30 proc_lsst.multi:146 [INFO] found job 16964150 in provider checkpoint proc_lsst.multi INFO: found job 16964150 in provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:201 [INFO] cancelling 16964150 on provider checkpoint proc_lsst.multi INFO: cancelling 16964150 on provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:146 [INFO] found job 16964151 in provider checkpoint proc_lsst.multi INFO: found job 16964151 in provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:201 [INFO] cancelling 16964151 on provider checkpoint proc_lsst.multi INFO: cancelling 16964151 on provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:146 [INFO] found job 16964149 in provider checkpoint proc_lsst.multi INFO: found job 16964149 in provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:201 [INFO] cancelling 16964149 on provider checkpoint proc_lsst.multi INFO: cancelling 16964149 on provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:146 [INFO] found job 16964109 in provider checkpoint proc_lsst.multi INFO: found job 16964109 in provider checkpoint 2024-03-09 15:00:30 proc_lsst.multi:201 [INFO] cancelling 16964109 on provider checkpoint proc_lsst.multi INFO: cancelling 16964109 on provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:146 [INFO] found job 16964084 in provider checkpoint proc_lsst.multi INFO: found job 16964084 in provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:201 [INFO] cancelling 16964084 on provider checkpoint proc_lsst.multi INFO: cancelling 16964084 on provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:146 [INFO] found job 16964110 in provider checkpoint proc_lsst.multi INFO: found job 16964110 in provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:201 [INFO] cancelling 16964110 on provider checkpoint proc_lsst.multi INFO: cancelling 16964110 on provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:146 [INFO] found job 16964111 in provider checkpoint proc_lsst.multi INFO: found job 16964111 in provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:201 [INFO] cancelling 16964111 on provider checkpoint proc_lsst.multi INFO: cancelling 16964111 on provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:146 [INFO] found job 16964123 in provider checkpoint proc_lsst.multi INFO: found job 16964123 in provider checkpoint 2024-03-09 15:00:31 proc_lsst.multi:201 [INFO] cancelling 16964123 on provider checkpoint proc_lsst.multi INFO: cancelling 16964123 on provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:146 [INFO] found job 16964122 in provider checkpoint proc_lsst.multi INFO: found job 16964122 in provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:201 [INFO] cancelling 16964122 on provider checkpoint proc_lsst.multi INFO: cancelling 16964122 on provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:146 [INFO] found job 16964145 in provider checkpoint proc_lsst.multi INFO: found job 16964145 in provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:201 [INFO] cancelling 16964145 on provider checkpoint proc_lsst.multi INFO: cancelling 16964145 on provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:146 [INFO] found job 16964119 in provider checkpoint proc_lsst.multi INFO: found job 16964119 in provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:201 [INFO] cancelling 16964119 on provider checkpoint proc_lsst.multi INFO: cancelling 16964119 on provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:146 [INFO] found job 16964081 in provider checkpoint proc_lsst.multi INFO: found job 16964081 in provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:201 [INFO] cancelling 16964081 on provider checkpoint proc_lsst.multi INFO: cancelling 16964081 on provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:146 [INFO] found job 16964085 in provider checkpoint proc_lsst.multi INFO: found job 16964085 in provider checkpoint 2024-03-09 15:00:32 proc_lsst.multi:201 [INFO] cancelling 16964085 on provider checkpoint proc_lsst.multi INFO: cancelling 16964085 on provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:146 [INFO] found job 16964121 in provider checkpoint proc_lsst.multi INFO: found job 16964121 in provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:201 [INFO] cancelling 16964121 on provider checkpoint proc_lsst.multi INFO: cancelling 16964121 on provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:146 [INFO] found job 16964086 in provider checkpoint proc_lsst.multi INFO: found job 16964086 in provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:201 [INFO] cancelling 16964086 on provider checkpoint proc_lsst.multi INFO: cancelling 16964086 on provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:146 [INFO] found job 16964088 in provider checkpoint proc_lsst.multi INFO: found job 16964088 in provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:201 [INFO] cancelling 16964088 on provider checkpoint proc_lsst.multi INFO: cancelling 16964088 on provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:146 [INFO] found job 16964162 in provider checkpoint proc_lsst.multi INFO: found job 16964162 in provider checkpoint 2024-03-09 15:00:33 proc_lsst.multi:201 [INFO] cancelling 16964162 on provider checkpoint proc_lsst.multi INFO: cancelling 16964162 on provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:146 [INFO] found job 16964087 in provider checkpoint proc_lsst.multi INFO: found job 16964087 in provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:201 [INFO] cancelling 16964087 on provider checkpoint proc_lsst.multi INFO: cancelling 16964087 on provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:146 [INFO] found job 16964107 in provider checkpoint proc_lsst.multi INFO: found job 16964107 in provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:201 [INFO] cancelling 16964107 on provider checkpoint proc_lsst.multi INFO: cancelling 16964107 on provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:146 [INFO] found job 16964106 in provider checkpoint proc_lsst.multi INFO: found job 16964106 in provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:201 [INFO] cancelling 16964106 on provider checkpoint proc_lsst.multi INFO: cancelling 16964106 on provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:146 [INFO] found job 16964094 in provider checkpoint proc_lsst.multi INFO: found job 16964094 in provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:201 [INFO] cancelling 16964094 on provider checkpoint proc_lsst.multi INFO: cancelling 16964094 on provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:146 [INFO] found job 16964103 in provider checkpoint proc_lsst.multi INFO: found job 16964103 in provider checkpoint 2024-03-09 15:00:34 proc_lsst.multi:201 [INFO] cancelling 16964103 on provider checkpoint proc_lsst.multi INFO: cancelling 16964103 on provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:146 [INFO] found job 16964135 in provider checkpoint proc_lsst.multi INFO: found job 16964135 in provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:201 [INFO] cancelling 16964135 on provider checkpoint proc_lsst.multi INFO: cancelling 16964135 on provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:146 [INFO] found job 16964146 in provider checkpoint proc_lsst.multi INFO: found job 16964146 in provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:201 [INFO] cancelling 16964146 on provider checkpoint proc_lsst.multi INFO: cancelling 16964146 on provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:146 [INFO] found job 16964136 in provider checkpoint proc_lsst.multi INFO: found job 16964136 in provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:201 [INFO] cancelling 16964136 on provider checkpoint proc_lsst.multi INFO: cancelling 16964136 on provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:146 [INFO] found job 16964095 in provider checkpoint proc_lsst.multi INFO: found job 16964095 in provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:201 [INFO] cancelling 16964095 on provider checkpoint proc_lsst.multi INFO: cancelling 16964095 on provider checkpoint 2024-03-09 15:00:35 proc_lsst.multi:146 [INFO] found job 16964036 in provider astro proc_lsst.multi INFO: found job 16964036 in provider astro 2024-03-09 15:00:35 proc_lsst.multi:201 [INFO] cancelling 16964036 on provider astro proc_lsst.multi INFO: cancelling 16964036 on provider astro 2024-03-09 15:00:36 proc_lsst.multi:146 [INFO] found job 16964127 in provider checkpoint proc_lsst.multi INFO: found job 16964127 in provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:201 [INFO] cancelling 16964127 on provider checkpoint proc_lsst.multi INFO: cancelling 16964127 on provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:146 [INFO] found job 16964128 in provider checkpoint proc_lsst.multi INFO: found job 16964128 in provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:201 [INFO] cancelling 16964128 on provider checkpoint proc_lsst.multi INFO: cancelling 16964128 on provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:146 [INFO] found job 16964105 in provider checkpoint proc_lsst.multi INFO: found job 16964105 in provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:201 [INFO] cancelling 16964105 on provider checkpoint proc_lsst.multi INFO: cancelling 16964105 on provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:146 [INFO] found job 16964153 in provider checkpoint proc_lsst.multi INFO: found job 16964153 in provider checkpoint 2024-03-09 15:00:36 proc_lsst.multi:201 [INFO] cancelling 16964153 on provider checkpoint proc_lsst.multi INFO: cancelling 16964153 on provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:146 [INFO] found job 16964071 in provider escience_a40 proc_lsst.multi INFO: found job 16964071 in provider escience_a40 2024-03-09 15:00:37 proc_lsst.multi:201 [INFO] cancelling 16964071 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964071 on provider escience_a40 2024-03-09 15:00:37 proc_lsst.multi:146 [INFO] found job 16964108 in provider checkpoint proc_lsst.multi INFO: found job 16964108 in provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:201 [INFO] cancelling 16964108 on provider checkpoint proc_lsst.multi INFO: cancelling 16964108 on provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:146 [INFO] found job 16964104 in provider checkpoint proc_lsst.multi INFO: found job 16964104 in provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:201 [INFO] cancelling 16964104 on provider checkpoint proc_lsst.multi INFO: cancelling 16964104 on provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:146 [INFO] found job 16964058 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964058 in provider escience_rtx6k 2024-03-09 15:00:37 proc_lsst.multi:201 [INFO] cancelling 16964058 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964058 on provider escience_rtx6k 2024-03-09 15:00:37 proc_lsst.multi:146 [INFO] found job 16964154 in provider checkpoint proc_lsst.multi INFO: found job 16964154 in provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:201 [INFO] cancelling 16964154 on provider checkpoint proc_lsst.multi INFO: cancelling 16964154 on provider checkpoint 2024-03-09 15:00:37 proc_lsst.multi:146 [INFO] found job 16964057 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964057 in provider escience_rtx6k 2024-03-09 15:00:37 proc_lsst.multi:201 [INFO] cancelling 16964057 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964057 on provider escience_rtx6k 2024-03-09 15:00:38 proc_lsst.multi:146 [INFO] found job 16964134 in provider checkpoint proc_lsst.multi INFO: found job 16964134 in provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:201 [INFO] cancelling 16964134 on provider checkpoint proc_lsst.multi INFO: cancelling 16964134 on provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:146 [INFO] found job 16964114 in provider checkpoint proc_lsst.multi INFO: found job 16964114 in provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:201 [INFO] cancelling 16964114 on provider checkpoint proc_lsst.multi INFO: cancelling 16964114 on provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:146 [INFO] found job 16964100 in provider checkpoint proc_lsst.multi INFO: found job 16964100 in provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:201 [INFO] cancelling 16964100 on provider checkpoint proc_lsst.multi INFO: cancelling 16964100 on provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:146 [INFO] found job 16964152 in provider checkpoint proc_lsst.multi INFO: found job 16964152 in provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:201 [INFO] cancelling 16964152 on provider checkpoint proc_lsst.multi INFO: cancelling 16964152 on provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:146 [INFO] found job 16964082 in provider checkpoint proc_lsst.multi INFO: found job 16964082 in provider checkpoint 2024-03-09 15:00:38 proc_lsst.multi:201 [INFO] cancelling 16964082 on provider checkpoint proc_lsst.multi INFO: cancelling 16964082 on provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:146 [INFO] found job 16964097 in provider checkpoint proc_lsst.multi INFO: found job 16964097 in provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:201 [INFO] cancelling 16964097 on provider checkpoint proc_lsst.multi INFO: cancelling 16964097 on provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:146 [INFO] found job 16964129 in provider checkpoint proc_lsst.multi INFO: found job 16964129 in provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:201 [INFO] cancelling 16964129 on provider checkpoint proc_lsst.multi INFO: cancelling 16964129 on provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:146 [INFO] found job 16964072 in provider escience_a40 proc_lsst.multi INFO: found job 16964072 in provider escience_a40 2024-03-09 15:00:39 proc_lsst.multi:201 [INFO] cancelling 16964072 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964072 on provider escience_a40 2024-03-09 15:00:39 proc_lsst.multi:146 [INFO] found job 16964102 in provider checkpoint proc_lsst.multi INFO: found job 16964102 in provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:201 [INFO] cancelling 16964102 on provider checkpoint proc_lsst.multi INFO: cancelling 16964102 on provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:146 [INFO] found job 16964059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964059 in provider escience_rtx6k 2024-03-09 15:00:39 proc_lsst.multi:201 [INFO] cancelling 16964059 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964059 on provider escience_rtx6k 2024-03-09 15:00:39 proc_lsst.multi:146 [INFO] found job 16964113 in provider checkpoint proc_lsst.multi INFO: found job 16964113 in provider checkpoint 2024-03-09 15:00:39 proc_lsst.multi:201 [INFO] cancelling 16964113 on provider checkpoint proc_lsst.multi INFO: cancelling 16964113 on provider checkpoint 2024-03-09 15:00:40 proc_lsst.multi:146 [INFO] found job 16964112 in provider checkpoint proc_lsst.multi INFO: found job 16964112 in provider checkpoint 2024-03-09 15:00:40 proc_lsst.multi:201 [INFO] cancelling 16964112 on provider checkpoint proc_lsst.multi INFO: cancelling 16964112 on provider checkpoint 2024-03-09 15:00:40 proc_lsst.multi:146 [INFO] found job 16964039 in provider astro proc_lsst.multi INFO: found job 16964039 in provider astro 2024-03-09 15:00:40 proc_lsst.multi:201 [INFO] cancelling 16964039 on provider astro proc_lsst.multi INFO: cancelling 16964039 on provider astro 2024-03-09 15:00:40 proc_lsst.multi:146 [INFO] found job 16964049 in provider astro proc_lsst.multi INFO: found job 16964049 in provider astro 2024-03-09 15:00:40 proc_lsst.multi:201 [INFO] cancelling 16964049 on provider astro proc_lsst.multi INFO: cancelling 16964049 on provider astro 2024-03-09 15:00:40 proc_lsst.multi:146 [INFO] found job 16964028 in provider astro proc_lsst.multi INFO: found job 16964028 in provider astro 2024-03-09 15:00:40 proc_lsst.multi:201 [INFO] cancelling 16964028 on provider astro proc_lsst.multi INFO: cancelling 16964028 on provider astro 2024-03-09 15:00:40 proc_lsst.multi:146 [INFO] found job 16964027 in provider astro proc_lsst.multi INFO: found job 16964027 in provider astro 2024-03-09 15:00:40 proc_lsst.multi:201 [INFO] cancelling 16964027 on provider astro proc_lsst.multi INFO: cancelling 16964027 on provider astro 2024-03-09 15:00:41 proc_lsst.multi:146 [INFO] found job 16964118 in provider checkpoint proc_lsst.multi INFO: found job 16964118 in provider checkpoint 2024-03-09 15:00:41 proc_lsst.multi:201 [INFO] cancelling 16964118 on provider checkpoint proc_lsst.multi INFO: cancelling 16964118 on provider checkpoint 2024-03-09 15:00:41 proc_lsst.multi:146 [INFO] found job 77954 in provider local proc_lsst.multi INFO: found job 77954 in provider local 2024-03-09 15:00:41 proc_lsst.multi:201 [INFO] cancelling 77954 on provider local proc_lsst.multi INFO: cancelling 77954 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 77954 2024-03-09 15:00:41 proc_lsst.multi:146 [INFO] found job 16964117 in provider checkpoint proc_lsst.multi INFO: found job 16964117 in provider checkpoint 2024-03-09 15:00:41 proc_lsst.multi:201 [INFO] cancelling 16964117 on provider checkpoint proc_lsst.multi INFO: cancelling 16964117 on provider checkpoint 2024-03-09 15:00:41 proc_lsst.multi:146 [INFO] found job 16964063 in provider escience_a40 proc_lsst.multi INFO: found job 16964063 in provider escience_a40 2024-03-09 15:00:41 proc_lsst.multi:201 [INFO] cancelling 16964063 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964063 on provider escience_a40 2024-03-09 15:00:41 proc_lsst.multi:146 [INFO] found job 16964026 in provider astro proc_lsst.multi INFO: found job 16964026 in provider astro 2024-03-09 15:00:41 proc_lsst.multi:201 [INFO] cancelling 16964026 on provider astro proc_lsst.multi INFO: cancelling 16964026 on provider astro 2024-03-09 15:00:42 proc_lsst.multi:146 [INFO] found job 16964041 in provider astro proc_lsst.multi INFO: found job 16964041 in provider astro 2024-03-09 15:00:42 proc_lsst.multi:201 [INFO] cancelling 16964041 on provider astro proc_lsst.multi INFO: cancelling 16964041 on provider astro 2024-03-09 15:00:42 proc_lsst.multi:146 [INFO] found job 16964050 in provider astro proc_lsst.multi INFO: found job 16964050 in provider astro 2024-03-09 15:00:42 proc_lsst.multi:201 [INFO] cancelling 16964050 on provider astro proc_lsst.multi INFO: cancelling 16964050 on provider astro 2024-03-09 15:00:42 proc_lsst.multi:146 [INFO] found job 16964068 in provider escience_a40 proc_lsst.multi INFO: found job 16964068 in provider escience_a40 2024-03-09 15:00:42 proc_lsst.multi:201 [INFO] cancelling 16964068 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964068 on provider escience_a40 2024-03-09 15:00:42 proc_lsst.multi:146 [INFO] found job 16964064 in provider escience_a40 proc_lsst.multi INFO: found job 16964064 in provider escience_a40 2024-03-09 15:00:42 proc_lsst.multi:201 [INFO] cancelling 16964064 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964064 on provider escience_a40 2024-03-09 15:00:42 proc_lsst.multi:146 [INFO] found job 16964045 in provider astro proc_lsst.multi INFO: found job 16964045 in provider astro 2024-03-09 15:00:42 proc_lsst.multi:201 [INFO] cancelling 16964045 on provider astro proc_lsst.multi INFO: cancelling 16964045 on provider astro 2024-03-09 15:00:42 proc_lsst.multi:146 [INFO] found job 16964090 in provider checkpoint proc_lsst.multi INFO: found job 16964090 in provider checkpoint 2024-03-09 15:00:42 proc_lsst.multi:201 [INFO] cancelling 16964090 on provider checkpoint proc_lsst.multi INFO: cancelling 16964090 on provider checkpoint 2024-03-09 15:00:43 proc_lsst.multi:146 [INFO] found job 16964029 in provider astro proc_lsst.multi INFO: found job 16964029 in provider astro 2024-03-09 15:00:43 proc_lsst.multi:201 [INFO] cancelling 16964029 on provider astro proc_lsst.multi INFO: cancelling 16964029 on provider astro 2024-03-09 15:00:43 proc_lsst.multi:146 [INFO] found job 16964091 in provider checkpoint proc_lsst.multi INFO: found job 16964091 in provider checkpoint 2024-03-09 15:00:43 proc_lsst.multi:201 [INFO] cancelling 16964091 on provider checkpoint proc_lsst.multi INFO: cancelling 16964091 on provider checkpoint 2024-03-09 15:00:43 proc_lsst.multi:146 [INFO] found job 16964093 in provider checkpoint proc_lsst.multi INFO: found job 16964093 in provider checkpoint 2024-03-09 15:00:43 proc_lsst.multi:201 [INFO] cancelling 16964093 on provider checkpoint proc_lsst.multi INFO: cancelling 16964093 on provider checkpoint 2024-03-09 15:00:43 proc_lsst.multi:146 [INFO] found job 16964044 in provider astro proc_lsst.multi INFO: found job 16964044 in provider astro 2024-03-09 15:00:43 proc_lsst.multi:201 [INFO] cancelling 16964044 on provider astro proc_lsst.multi INFO: cancelling 16964044 on provider astro 2024-03-09 15:00:43 proc_lsst.multi:146 [INFO] found job 16964031 in provider astro proc_lsst.multi INFO: found job 16964031 in provider astro 2024-03-09 15:00:43 proc_lsst.multi:201 [INFO] cancelling 16964031 on provider astro proc_lsst.multi INFO: cancelling 16964031 on provider astro 2024-03-09 15:00:43 proc_lsst.multi:146 [INFO] found job 16964115 in provider checkpoint proc_lsst.multi INFO: found job 16964115 in provider checkpoint 2024-03-09 15:00:43 proc_lsst.multi:201 [INFO] cancelling 16964115 on provider checkpoint proc_lsst.multi INFO: cancelling 16964115 on provider checkpoint 2024-03-09 15:00:44 proc_lsst.multi:146 [INFO] found job 16964033 in provider astro proc_lsst.multi INFO: found job 16964033 in provider astro 2024-03-09 15:00:44 proc_lsst.multi:201 [INFO] cancelling 16964033 on provider astro proc_lsst.multi INFO: cancelling 16964033 on provider astro 2024-03-09 15:00:44 proc_lsst.multi:146 [INFO] found job 16964092 in provider checkpoint proc_lsst.multi INFO: found job 16964092 in provider checkpoint 2024-03-09 15:00:44 proc_lsst.multi:201 [INFO] cancelling 16964092 on provider checkpoint proc_lsst.multi INFO: cancelling 16964092 on provider checkpoint 2024-03-09 15:00:44 proc_lsst.multi:146 [INFO] found job 16964124 in provider checkpoint proc_lsst.multi INFO: found job 16964124 in provider checkpoint 2024-03-09 15:00:44 proc_lsst.multi:201 [INFO] cancelling 16964124 on provider checkpoint proc_lsst.multi INFO: cancelling 16964124 on provider checkpoint 2024-03-09 15:00:44 proc_lsst.multi:146 [INFO] found job 16964062 in provider escience_a40 proc_lsst.multi INFO: found job 16964062 in provider escience_a40 2024-03-09 15:00:44 proc_lsst.multi:201 [INFO] cancelling 16964062 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964062 on provider escience_a40 2024-03-09 15:00:44 proc_lsst.multi:146 [INFO] found job 16964047 in provider astro proc_lsst.multi INFO: found job 16964047 in provider astro 2024-03-09 15:00:44 proc_lsst.multi:201 [INFO] cancelling 16964047 on provider astro proc_lsst.multi INFO: cancelling 16964047 on provider astro 2024-03-09 15:00:45 proc_lsst.multi:146 [INFO] found job 16964048 in provider astro proc_lsst.multi INFO: found job 16964048 in provider astro 2024-03-09 15:00:45 proc_lsst.multi:201 [INFO] cancelling 16964048 on provider astro proc_lsst.multi INFO: cancelling 16964048 on provider astro 2024-03-09 15:00:45 proc_lsst.multi:146 [INFO] found job 16964076 in provider escience_a40 proc_lsst.multi INFO: found job 16964076 in provider escience_a40 2024-03-09 15:00:45 proc_lsst.multi:201 [INFO] cancelling 16964076 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964076 on provider escience_a40 2024-03-09 15:00:45 proc_lsst.multi:146 [INFO] found job 16964034 in provider astro proc_lsst.multi INFO: found job 16964034 in provider astro 2024-03-09 15:00:45 proc_lsst.multi:201 [INFO] cancelling 16964034 on provider astro proc_lsst.multi INFO: cancelling 16964034 on provider astro 2024-03-09 15:00:45 proc_lsst.multi:146 [INFO] found job 16964080 in provider escience_a40 proc_lsst.multi INFO: found job 16964080 in provider escience_a40 2024-03-09 15:00:45 proc_lsst.multi:201 [INFO] cancelling 16964080 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964080 on provider escience_a40 2024-03-09 15:00:45 proc_lsst.multi:146 [INFO] found job 16964060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964060 in provider escience_rtx6k 2024-03-09 15:00:45 proc_lsst.multi:201 [INFO] cancelling 16964060 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964060 on provider escience_rtx6k 2024-03-09 15:00:45 proc_lsst.multi:146 [INFO] found job 16964096 in provider checkpoint proc_lsst.multi INFO: found job 16964096 in provider checkpoint 2024-03-09 15:00:45 proc_lsst.multi:201 [INFO] cancelling 16964096 on provider checkpoint proc_lsst.multi INFO: cancelling 16964096 on provider checkpoint 2024-03-09 15:00:46 proc_lsst.multi:146 [INFO] found job 16964120 in provider checkpoint proc_lsst.multi INFO: found job 16964120 in provider checkpoint 2024-03-09 15:00:46 proc_lsst.multi:201 [INFO] cancelling 16964120 on provider checkpoint proc_lsst.multi INFO: cancelling 16964120 on provider checkpoint 2024-03-09 15:00:46 proc_lsst.multi:146 [INFO] found job 16964055 in provider astro proc_lsst.multi INFO: found job 16964055 in provider astro 2024-03-09 15:00:46 proc_lsst.multi:201 [INFO] cancelling 16964055 on provider astro proc_lsst.multi INFO: cancelling 16964055 on provider astro 2024-03-09 15:00:46 proc_lsst.multi:146 [INFO] found job 16964051 in provider astro proc_lsst.multi INFO: found job 16964051 in provider astro 2024-03-09 15:00:46 proc_lsst.multi:201 [INFO] cancelling 16964051 on provider astro proc_lsst.multi INFO: cancelling 16964051 on provider astro 2024-03-09 15:00:46 proc_lsst.multi:146 [INFO] found job 16964148 in provider checkpoint proc_lsst.multi INFO: found job 16964148 in provider checkpoint 2024-03-09 15:00:46 proc_lsst.multi:201 [INFO] cancelling 16964148 on provider checkpoint proc_lsst.multi INFO: cancelling 16964148 on provider checkpoint 2024-03-09 15:00:46 proc_lsst.multi:146 [INFO] found job 16964147 in provider checkpoint proc_lsst.multi INFO: found job 16964147 in provider checkpoint 2024-03-09 15:00:46 proc_lsst.multi:201 [INFO] cancelling 16964147 on provider checkpoint proc_lsst.multi INFO: cancelling 16964147 on provider checkpoint 2024-03-09 15:00:47 proc_lsst.multi:146 [INFO] found job 16964037 in provider astro proc_lsst.multi INFO: found job 16964037 in provider astro 2024-03-09 15:00:47 proc_lsst.multi:201 [INFO] cancelling 16964037 on provider astro proc_lsst.multi INFO: cancelling 16964037 on provider astro 2024-03-09 15:00:47 proc_lsst.multi:146 [INFO] found job 16964052 in provider astro proc_lsst.multi INFO: found job 16964052 in provider astro 2024-03-09 15:00:47 proc_lsst.multi:201 [INFO] cancelling 16964052 on provider astro proc_lsst.multi INFO: cancelling 16964052 on provider astro 2024-03-09 15:00:47 proc_lsst.multi:146 [INFO] found job 16964040 in provider astro proc_lsst.multi INFO: found job 16964040 in provider astro 2024-03-09 15:00:47 proc_lsst.multi:201 [INFO] cancelling 16964040 on provider astro proc_lsst.multi INFO: cancelling 16964040 on provider astro 2024-03-09 15:00:47 proc_lsst.multi:146 [INFO] found job 16964038 in provider astro proc_lsst.multi INFO: found job 16964038 in provider astro 2024-03-09 15:00:47 proc_lsst.multi:201 [INFO] cancelling 16964038 on provider astro proc_lsst.multi INFO: cancelling 16964038 on provider astro 2024-03-09 15:00:47 proc_lsst.multi:146 [INFO] found job 16964035 in provider astro proc_lsst.multi INFO: found job 16964035 in provider astro 2024-03-09 15:00:47 proc_lsst.multi:201 [INFO] cancelling 16964035 on provider astro proc_lsst.multi INFO: cancelling 16964035 on provider astro 2024-03-09 15:00:47 proc_lsst.multi:146 [INFO] found job 16964043 in provider astro proc_lsst.multi INFO: found job 16964043 in provider astro 2024-03-09 15:00:47 proc_lsst.multi:201 [INFO] cancelling 16964043 on provider astro proc_lsst.multi INFO: cancelling 16964043 on provider astro 2024-03-09 15:00:48 proc_lsst.multi:146 [INFO] found job 16964073 in provider escience_a40 proc_lsst.multi INFO: found job 16964073 in provider escience_a40 2024-03-09 15:00:48 proc_lsst.multi:201 [INFO] cancelling 16964073 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964073 on provider escience_a40 2024-03-09 15:00:48 proc_lsst.multi:146 [INFO] found job 16964042 in provider astro proc_lsst.multi INFO: found job 16964042 in provider astro 2024-03-09 15:00:48 proc_lsst.multi:201 [INFO] cancelling 16964042 on provider astro proc_lsst.multi INFO: cancelling 16964042 on provider astro 2024-03-09 15:00:48 proc_lsst.multi:146 [INFO] found job 16964030 in provider astro proc_lsst.multi INFO: found job 16964030 in provider astro 2024-03-09 15:00:48 proc_lsst.multi:201 [INFO] cancelling 16964030 on provider astro proc_lsst.multi INFO: cancelling 16964030 on provider astro 2024-03-09 15:00:48 proc_lsst.multi:146 [INFO] found job 16964054 in provider astro proc_lsst.multi INFO: found job 16964054 in provider astro 2024-03-09 15:00:48 proc_lsst.multi:201 [INFO] cancelling 16964054 on provider astro proc_lsst.multi INFO: cancelling 16964054 on provider astro 2024-03-09 15:00:48 proc_lsst.multi:146 [INFO] found job 16964032 in provider astro proc_lsst.multi INFO: found job 16964032 in provider astro 2024-03-09 15:00:48 proc_lsst.multi:201 [INFO] cancelling 16964032 on provider astro proc_lsst.multi INFO: cancelling 16964032 on provider astro 2024-03-09 15:00:48 proc_lsst.multi:146 [INFO] found job 16964046 in provider astro proc_lsst.multi INFO: found job 16964046 in provider astro 2024-03-09 15:00:48 proc_lsst.multi:201 [INFO] cancelling 16964046 on provider astro proc_lsst.multi INFO: cancelling 16964046 on provider astro 2024-03-09 15:00:49 proc_lsst.multi:146 [INFO] found job 16964074 in provider escience_a40 proc_lsst.multi INFO: found job 16964074 in provider escience_a40 2024-03-09 15:00:49 proc_lsst.multi:201 [INFO] cancelling 16964074 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964074 on provider escience_a40 2024-03-09 15:00:49 proc_lsst.multi:146 [INFO] found job 16964077 in provider escience_a40 proc_lsst.multi INFO: found job 16964077 in provider escience_a40 2024-03-09 15:00:49 proc_lsst.multi:201 [INFO] cancelling 16964077 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964077 on provider escience_a40 2024-03-09 15:00:49 proc_lsst.multi:146 [INFO] found job 16964079 in provider escience_a40 proc_lsst.multi INFO: found job 16964079 in provider escience_a40 2024-03-09 15:00:49 proc_lsst.multi:201 [INFO] cancelling 16964079 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964079 on provider escience_a40 2024-03-09 15:00:49 proc_lsst.multi:146 [INFO] found job 16964056 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964056 in provider escience_rtx6k 2024-03-09 15:00:49 proc_lsst.multi:201 [INFO] cancelling 16964056 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964056 on provider escience_rtx6k 2024-03-09 15:00:49 proc_lsst.multi:146 [INFO] found job 16964053 in provider astro proc_lsst.multi INFO: found job 16964053 in provider astro 2024-03-09 15:00:49 proc_lsst.multi:201 [INFO] cancelling 16964053 on provider astro proc_lsst.multi INFO: cancelling 16964053 on provider astro 2024-03-09 15:00:50 proc_lsst.multi:146 [INFO] found job 16964069 in provider escience_a40 proc_lsst.multi INFO: found job 16964069 in provider escience_a40 2024-03-09 15:00:50 proc_lsst.multi:201 [INFO] cancelling 16964069 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964069 on provider escience_a40 2024-03-09 15:00:50 proc_lsst.multi:146 [INFO] found job 16964066 in provider escience_a40 proc_lsst.multi INFO: found job 16964066 in provider escience_a40 2024-03-09 15:00:50 proc_lsst.multi:201 [INFO] cancelling 16964066 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964066 on provider escience_a40 2024-03-09 15:00:50 proc_lsst.multi:146 [INFO] found job 16964065 in provider escience_a40 proc_lsst.multi INFO: found job 16964065 in provider escience_a40 2024-03-09 15:00:50 proc_lsst.multi:201 [INFO] cancelling 16964065 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964065 on provider escience_a40 parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-09 15:00:50 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-09 15:00:50 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16964081', '16964082', '16964083', '16964084', '16964085', '16964086', '16964087', '16964088', '16964089', '16964090', '16964091', '16964092', '16964093', '16964094', '16964095', '16964096', '16964097', '16964099', '16964100', '16964101', '16964102', '16964103', '16964104', '16964105', '16964106', '16964107', '16964108', '16964109', '16964110', '16964111', '16964112', '16964113', '16964114', '16964115', '16964116', '16964117', '16964118', '16964119', '16964120', '16964121', '16964122', '16964123', '16964124', '16964125', '16964126', '16964127', '16964128', '16964129', '16964130', '16964131', '16964132', '16964133', '16964134', '16964135', '16964136', '16964137', '16964138', '16964139', '16964140', '16964141', '16964142', '16964143', '16964144', '16964145', '16964146', '16964147', '16964148', '16964149', '16964150', '16964151', '16964152', '16964153', '16964154', '16964155', '16964156', '16964157', '16964158', '16964159', '16964160', '16964161', '16964162', '16964163', '16964164', '16964165', '16964166', '16964167', '16964168', '16964169', '16964170', '16964171', '16964172', '16964173', '16964174', '16964175', '16964176', '16964177', '16964178', '16964179', '16964180', '16964181', '16964182', '16964183', '16964184', '16964185', '16964186', '16964187', '16964188', '16964189', '16964190', '16964191', '16964192', '16964193', '16964194', '16964195', '16964196', '16964197', '16964198', '16964199', '16964200', '16964201', '16964202', '16964203', '16964204', '16964205', '16964206', '16964207', '16964208', '16964209', '16964210', '16964211', '16964212', '16964213', '16964214', '16964215', '16964216', '16964217', '16964218', '16964220', '16964221', '16964222', '16964223', '16964224', '16964225', '16964226', '16964227', '16964228', '16964229', '16964230', '16964231', '16964232', '16964233', '16964234', '16964235', '16964236', '16964237', '16964238', '16964239', '16964242', '16964243', '16964244', '16964245', '16964246', '16964247', '16964248', '16964249', '16964250', '16964251', '16964252', '16964253', '16964254', '16964255', '16964256', '16964257', '16964258', '16964259', '16964260', '16964261', '16964262', '16964263', '16964264', '16964265', '16964266', '16964267', '16964268', '16964269', '16964270', '16964271', '16964272', '16964273', '16964274', '16964275', '16964276', '16964277', '16964278', '16964284', '16964286', '16964287', '16964288', '16964289', '16964290', '16964291', '16964292', '16964293', '16964294', '16964295', '16964296', '16964297', '16964298', '16964299', '16964300', '16964301', '16964304', '16964305', '16964306', '16964307', '16964308', '16964309', '16964310', '16964311', '16964312', '16964313', '16964314', '16964315', '16964316', '16964317', '16964318', '16964319', '16964320', '16964321', '16964322', '16964323', '16964324', '16964325', '16964326', '16964327', '16964328', '16964329', '16964330', '16964331', '16964332', '16964333', '16964334', '16964335', '16964336', '16964337', '16964338', '16964339', '16964340', '16964341', '16964342', '16964343', '16964344', '16964345', '16964346', '16964347', '16964348', '16964026', '16964027', '16964028', '16964029', '16964030', '16964031', '16964032', '16964033', '16964034', '16964035', '16964036', '16964037', '16964038', '16964039', '16964040', '16964041', '16964042', '16964043', '16964044', '16964045', '16964046', '16964047', '16964048', '16964049', '16964050', '16964051', '16964052', '16964053', '16964054', '16964055', '16964061', '16964062', '16964063', '16964064', '16964065', '16964066', '16964067', '16964068', '16964069', '16964070', '16964071', '16964072', '16964073', '16964074', '16964075', '16964076', '16964077', '16964078', '16964079', '16964080', '16964056', '16964057', '16964058', '16964059', '16964060', '77954'] proc_lsst.multi INFO: new jobs since last cancel ['16964081', '16964082', '16964083', '16964084', '16964085', '16964086', '16964087', '16964088', '16964089', '16964090', '16964091', '16964092', '16964093', '16964094', '16964095', '16964096', '16964097', '16964099', '16964100', '16964101', '16964102', '16964103', '16964104', '16964105', '16964106', '16964107', '16964108', '16964109', '16964110', '16964111', '16964112', '16964113', '16964114', '16964115', '16964116', '16964117', '16964118', '16964119', '16964120', '16964121', '16964122', '16964123', '16964124', '16964125', '16964126', '16964127', '16964128', '16964129', '16964130', '16964131', '16964132', '16964133', '16964134', '16964135', '16964136', '16964137', '16964138', '16964139', '16964140', '16964141', '16964142', '16964143', '16964144', '16964145', '16964146', '16964147', '16964148', '16964149', '16964150', '16964151', '16964152', '16964153', '16964154', '16964155', '16964156', '16964157', '16964158', '16964159', '16964160', '16964161', '16964162', '16964163', '16964164', '16964165', '16964166', '16964167', '16964168', '16964169', '16964170', '16964171', '16964172', '16964173', '16964174', '16964175', '16964176', '16964177', '16964178', '16964179', '16964180', '16964181', '16964182', '16964183', '16964184', '16964185', '16964186', '16964187', '16964188', '16964189', '16964190', '16964191', '16964192', '16964193', '16964194', '16964195', '16964196', '16964197', '16964198', '16964199', '16964200', '16964201', '16964202', '16964203', '16964204', '16964205', '16964206', '16964207', '16964208', '16964209', '16964210', '16964211', '16964212', '16964213', '16964214', '16964215', '16964216', '16964217', '16964218', '16964220', '16964221', '16964222', '16964223', '16964224', '16964225', '16964226', '16964227', '16964228', '16964229', '16964230', '16964231', '16964232', '16964233', '16964234', '16964235', '16964236', '16964237', '16964238', '16964239', '16964242', '16964243', '16964244', '16964245', '16964246', '16964247', '16964248', '16964249', '16964250', '16964251', '16964252', '16964253', '16964254', '16964255', '16964256', '16964257', '16964258', '16964259', '16964260', '16964261', '16964262', '16964263', '16964264', '16964265', '16964266', '16964267', '16964268', '16964269', '16964270', '16964271', '16964272', '16964273', '16964274', '16964275', '16964276', '16964277', '16964278', '16964284', '16964286', '16964287', '16964288', '16964289', '16964290', '16964291', '16964292', '16964293', '16964294', '16964295', '16964296', '16964297', '16964298', '16964299', '16964300', '16964301', '16964304', '16964305', '16964306', '16964307', '16964308', '16964309', '16964310', '16964311', '16964312', '16964313', '16964314', '16964315', '16964316', '16964317', '16964318', '16964319', '16964320', '16964321', '16964322', '16964323', '16964324', '16964325', '16964326', '16964327', '16964328', '16964329', '16964330', '16964331', '16964332', '16964333', '16964334', '16964335', '16964336', '16964337', '16964338', '16964339', '16964340', '16964341', '16964342', '16964343', '16964344', '16964345', '16964346', '16964347', '16964348', '16964026', '16964027', '16964028', '16964029', '16964030', '16964031', '16964032', '16964033', '16964034', '16964035', '16964036', '16964037', '16964038', '16964039', '16964040', '16964041', '16964042', '16964043', '16964044', '16964045', '16964046', '16964047', '16964048', '16964049', '16964050', '16964051', '16964052', '16964053', '16964054', '16964055', '16964061', '16964062', '16964063', '16964064', '16964065', '16964066', '16964067', '16964068', '16964069', '16964070', '16964071', '16964072', '16964073', '16964074', '16964075', '16964076', '16964077', '16964078', '16964079', '16964080', '16964056', '16964057', '16964058', '16964059', '16964060', '77954'] 2024-03-09 15:00:50 proc_lsst.multi:146 [INFO] found job 16964081 in provider checkpoint proc_lsst.multi INFO: found job 16964081 in provider checkpoint 2024-03-09 15:00:50 proc_lsst.multi:201 [INFO] cancelling 16964081 on provider checkpoint proc_lsst.multi INFO: cancelling 16964081 on provider checkpoint 2024-03-09 15:00:50 proc_lsst.multi:146 [INFO] found job 16964082 in provider checkpoint proc_lsst.multi INFO: found job 16964082 in provider checkpoint 2024-03-09 15:00:50 proc_lsst.multi:201 [INFO] cancelling 16964082 on provider checkpoint proc_lsst.multi INFO: cancelling 16964082 on provider checkpoint 2024-03-09 15:00:50 proc_lsst.multi:146 [INFO] found job 16964083 in provider checkpoint proc_lsst.multi INFO: found job 16964083 in provider checkpoint 2024-03-09 15:00:50 proc_lsst.multi:201 [INFO] cancelling 16964083 on provider checkpoint proc_lsst.multi INFO: cancelling 16964083 on provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:146 [INFO] found job 16964084 in provider checkpoint proc_lsst.multi INFO: found job 16964084 in provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:201 [INFO] cancelling 16964084 on provider checkpoint proc_lsst.multi INFO: cancelling 16964084 on provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:146 [INFO] found job 16964085 in provider checkpoint proc_lsst.multi INFO: found job 16964085 in provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:201 [INFO] cancelling 16964085 on provider checkpoint proc_lsst.multi INFO: cancelling 16964085 on provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:146 [INFO] found job 16964086 in provider checkpoint proc_lsst.multi INFO: found job 16964086 in provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:201 [INFO] cancelling 16964086 on provider checkpoint proc_lsst.multi INFO: cancelling 16964086 on provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:146 [INFO] found job 16964087 in provider checkpoint proc_lsst.multi INFO: found job 16964087 in provider checkpoint 2024-03-09 15:00:51 proc_lsst.multi:201 [INFO] cancelling 16964087 on provider checkpoint proc_lsst.multi INFO: cancelling 16964087 on provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:146 [INFO] found job 16964088 in provider checkpoint proc_lsst.multi INFO: found job 16964088 in provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:201 [INFO] cancelling 16964088 on provider checkpoint proc_lsst.multi INFO: cancelling 16964088 on provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:146 [INFO] found job 16964089 in provider checkpoint proc_lsst.multi INFO: found job 16964089 in provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:201 [INFO] cancelling 16964089 on provider checkpoint proc_lsst.multi INFO: cancelling 16964089 on provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:146 [INFO] found job 16964090 in provider checkpoint proc_lsst.multi INFO: found job 16964090 in provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:201 [INFO] cancelling 16964090 on provider checkpoint proc_lsst.multi INFO: cancelling 16964090 on provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:146 [INFO] found job 16964091 in provider checkpoint proc_lsst.multi INFO: found job 16964091 in provider checkpoint 2024-03-09 15:00:53 proc_lsst.multi:201 [INFO] cancelling 16964091 on provider checkpoint proc_lsst.multi INFO: cancelling 16964091 on provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:146 [INFO] found job 16964092 in provider checkpoint proc_lsst.multi INFO: found job 16964092 in provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:201 [INFO] cancelling 16964092 on provider checkpoint proc_lsst.multi INFO: cancelling 16964092 on provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:146 [INFO] found job 16964093 in provider checkpoint proc_lsst.multi INFO: found job 16964093 in provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:201 [INFO] cancelling 16964093 on provider checkpoint proc_lsst.multi INFO: cancelling 16964093 on provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:146 [INFO] found job 16964094 in provider checkpoint proc_lsst.multi INFO: found job 16964094 in provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:201 [INFO] cancelling 16964094 on provider checkpoint proc_lsst.multi INFO: cancelling 16964094 on provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:146 [INFO] found job 16964095 in provider checkpoint proc_lsst.multi INFO: found job 16964095 in provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:201 [INFO] cancelling 16964095 on provider checkpoint proc_lsst.multi INFO: cancelling 16964095 on provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:146 [INFO] found job 16964096 in provider checkpoint proc_lsst.multi INFO: found job 16964096 in provider checkpoint 2024-03-09 15:00:54 proc_lsst.multi:201 [INFO] cancelling 16964096 on provider checkpoint proc_lsst.multi INFO: cancelling 16964096 on provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:146 [INFO] found job 16964097 in provider checkpoint proc_lsst.multi INFO: found job 16964097 in provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:201 [INFO] cancelling 16964097 on provider checkpoint proc_lsst.multi INFO: cancelling 16964097 on provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:146 [INFO] found job 16964099 in provider checkpoint proc_lsst.multi INFO: found job 16964099 in provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:201 [INFO] cancelling 16964099 on provider checkpoint proc_lsst.multi INFO: cancelling 16964099 on provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:146 [INFO] found job 16964100 in provider checkpoint proc_lsst.multi INFO: found job 16964100 in provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:201 [INFO] cancelling 16964100 on provider checkpoint proc_lsst.multi INFO: cancelling 16964100 on provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:146 [INFO] found job 16964101 in provider checkpoint proc_lsst.multi INFO: found job 16964101 in provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:201 [INFO] cancelling 16964101 on provider checkpoint proc_lsst.multi INFO: cancelling 16964101 on provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:146 [INFO] found job 16964102 in provider checkpoint proc_lsst.multi INFO: found job 16964102 in provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:201 [INFO] cancelling 16964102 on provider checkpoint proc_lsst.multi INFO: cancelling 16964102 on provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:146 [INFO] found job 16964103 in provider checkpoint proc_lsst.multi INFO: found job 16964103 in provider checkpoint 2024-03-09 15:00:55 proc_lsst.multi:201 [INFO] cancelling 16964103 on provider checkpoint proc_lsst.multi INFO: cancelling 16964103 on provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:146 [INFO] found job 16964104 in provider checkpoint proc_lsst.multi INFO: found job 16964104 in provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:201 [INFO] cancelling 16964104 on provider checkpoint proc_lsst.multi INFO: cancelling 16964104 on provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:146 [INFO] found job 16964105 in provider checkpoint proc_lsst.multi INFO: found job 16964105 in provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:201 [INFO] cancelling 16964105 on provider checkpoint proc_lsst.multi INFO: cancelling 16964105 on provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:146 [INFO] found job 16964106 in provider checkpoint proc_lsst.multi INFO: found job 16964106 in provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:201 [INFO] cancelling 16964106 on provider checkpoint proc_lsst.multi INFO: cancelling 16964106 on provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:146 [INFO] found job 16964107 in provider checkpoint proc_lsst.multi INFO: found job 16964107 in provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:201 [INFO] cancelling 16964107 on provider checkpoint proc_lsst.multi INFO: cancelling 16964107 on provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:146 [INFO] found job 16964108 in provider checkpoint proc_lsst.multi INFO: found job 16964108 in provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:201 [INFO] cancelling 16964108 on provider checkpoint proc_lsst.multi INFO: cancelling 16964108 on provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:146 [INFO] found job 16964109 in provider checkpoint proc_lsst.multi INFO: found job 16964109 in provider checkpoint 2024-03-09 15:00:56 proc_lsst.multi:201 [INFO] cancelling 16964109 on provider checkpoint proc_lsst.multi INFO: cancelling 16964109 on provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:146 [INFO] found job 16964110 in provider checkpoint proc_lsst.multi INFO: found job 16964110 in provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:201 [INFO] cancelling 16964110 on provider checkpoint proc_lsst.multi INFO: cancelling 16964110 on provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:146 [INFO] found job 16964111 in provider checkpoint proc_lsst.multi INFO: found job 16964111 in provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:201 [INFO] cancelling 16964111 on provider checkpoint proc_lsst.multi INFO: cancelling 16964111 on provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:146 [INFO] found job 16964112 in provider checkpoint proc_lsst.multi INFO: found job 16964112 in provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:201 [INFO] cancelling 16964112 on provider checkpoint proc_lsst.multi INFO: cancelling 16964112 on provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:146 [INFO] found job 16964113 in provider checkpoint proc_lsst.multi INFO: found job 16964113 in provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:201 [INFO] cancelling 16964113 on provider checkpoint proc_lsst.multi INFO: cancelling 16964113 on provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:146 [INFO] found job 16964114 in provider checkpoint proc_lsst.multi INFO: found job 16964114 in provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:201 [INFO] cancelling 16964114 on provider checkpoint proc_lsst.multi INFO: cancelling 16964114 on provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:146 [INFO] found job 16964115 in provider checkpoint proc_lsst.multi INFO: found job 16964115 in provider checkpoint 2024-03-09 15:00:57 proc_lsst.multi:201 [INFO] cancelling 16964115 on provider checkpoint proc_lsst.multi INFO: cancelling 16964115 on provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:146 [INFO] found job 16964116 in provider checkpoint proc_lsst.multi INFO: found job 16964116 in provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:201 [INFO] cancelling 16964116 on provider checkpoint proc_lsst.multi INFO: cancelling 16964116 on provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:146 [INFO] found job 16964117 in provider checkpoint proc_lsst.multi INFO: found job 16964117 in provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:201 [INFO] cancelling 16964117 on provider checkpoint proc_lsst.multi INFO: cancelling 16964117 on provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:146 [INFO] found job 16964118 in provider checkpoint proc_lsst.multi INFO: found job 16964118 in provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:201 [INFO] cancelling 16964118 on provider checkpoint proc_lsst.multi INFO: cancelling 16964118 on provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:146 [INFO] found job 16964119 in provider checkpoint proc_lsst.multi INFO: found job 16964119 in provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:201 [INFO] cancelling 16964119 on provider checkpoint proc_lsst.multi INFO: cancelling 16964119 on provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:146 [INFO] found job 16964120 in provider checkpoint proc_lsst.multi INFO: found job 16964120 in provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:201 [INFO] cancelling 16964120 on provider checkpoint proc_lsst.multi INFO: cancelling 16964120 on provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:146 [INFO] found job 16964121 in provider checkpoint proc_lsst.multi INFO: found job 16964121 in provider checkpoint 2024-03-09 15:00:58 proc_lsst.multi:201 [INFO] cancelling 16964121 on provider checkpoint proc_lsst.multi INFO: cancelling 16964121 on provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:146 [INFO] found job 16964122 in provider checkpoint proc_lsst.multi INFO: found job 16964122 in provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:201 [INFO] cancelling 16964122 on provider checkpoint proc_lsst.multi INFO: cancelling 16964122 on provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:146 [INFO] found job 16964123 in provider checkpoint proc_lsst.multi INFO: found job 16964123 in provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:201 [INFO] cancelling 16964123 on provider checkpoint proc_lsst.multi INFO: cancelling 16964123 on provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:146 [INFO] found job 16964124 in provider checkpoint proc_lsst.multi INFO: found job 16964124 in provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:201 [INFO] cancelling 16964124 on provider checkpoint proc_lsst.multi INFO: cancelling 16964124 on provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:146 [INFO] found job 16964125 in provider checkpoint proc_lsst.multi INFO: found job 16964125 in provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:201 [INFO] cancelling 16964125 on provider checkpoint proc_lsst.multi INFO: cancelling 16964125 on provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:146 [INFO] found job 16964126 in provider checkpoint proc_lsst.multi INFO: found job 16964126 in provider checkpoint 2024-03-09 15:00:59 proc_lsst.multi:201 [INFO] cancelling 16964126 on provider checkpoint proc_lsst.multi INFO: cancelling 16964126 on provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:146 [INFO] found job 16964127 in provider checkpoint proc_lsst.multi INFO: found job 16964127 in provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:201 [INFO] cancelling 16964127 on provider checkpoint proc_lsst.multi INFO: cancelling 16964127 on provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:146 [INFO] found job 16964128 in provider checkpoint proc_lsst.multi INFO: found job 16964128 in provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:201 [INFO] cancelling 16964128 on provider checkpoint proc_lsst.multi INFO: cancelling 16964128 on provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:146 [INFO] found job 16964129 in provider checkpoint proc_lsst.multi INFO: found job 16964129 in provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:201 [INFO] cancelling 16964129 on provider checkpoint proc_lsst.multi INFO: cancelling 16964129 on provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:146 [INFO] found job 16964130 in provider checkpoint proc_lsst.multi INFO: found job 16964130 in provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:201 [INFO] cancelling 16964130 on provider checkpoint proc_lsst.multi INFO: cancelling 16964130 on provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:146 [INFO] found job 16964131 in provider checkpoint proc_lsst.multi INFO: found job 16964131 in provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:201 [INFO] cancelling 16964131 on provider checkpoint proc_lsst.multi INFO: cancelling 16964131 on provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:146 [INFO] found job 16964132 in provider checkpoint proc_lsst.multi INFO: found job 16964132 in provider checkpoint 2024-03-09 15:01:00 proc_lsst.multi:201 [INFO] cancelling 16964132 on provider checkpoint proc_lsst.multi INFO: cancelling 16964132 on provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:146 [INFO] found job 16964133 in provider checkpoint proc_lsst.multi INFO: found job 16964133 in provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:201 [INFO] cancelling 16964133 on provider checkpoint proc_lsst.multi INFO: cancelling 16964133 on provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:146 [INFO] found job 16964134 in provider checkpoint proc_lsst.multi INFO: found job 16964134 in provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:201 [INFO] cancelling 16964134 on provider checkpoint proc_lsst.multi INFO: cancelling 16964134 on provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:146 [INFO] found job 16964135 in provider checkpoint proc_lsst.multi INFO: found job 16964135 in provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:201 [INFO] cancelling 16964135 on provider checkpoint proc_lsst.multi INFO: cancelling 16964135 on provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:146 [INFO] found job 16964136 in provider checkpoint proc_lsst.multi INFO: found job 16964136 in provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:201 [INFO] cancelling 16964136 on provider checkpoint proc_lsst.multi INFO: cancelling 16964136 on provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:146 [INFO] found job 16964137 in provider checkpoint proc_lsst.multi INFO: found job 16964137 in provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:201 [INFO] cancelling 16964137 on provider checkpoint proc_lsst.multi INFO: cancelling 16964137 on provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:146 [INFO] found job 16964138 in provider checkpoint proc_lsst.multi INFO: found job 16964138 in provider checkpoint 2024-03-09 15:01:01 proc_lsst.multi:201 [INFO] cancelling 16964138 on provider checkpoint proc_lsst.multi INFO: cancelling 16964138 on provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:146 [INFO] found job 16964139 in provider checkpoint proc_lsst.multi INFO: found job 16964139 in provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:201 [INFO] cancelling 16964139 on provider checkpoint proc_lsst.multi INFO: cancelling 16964139 on provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:146 [INFO] found job 16964140 in provider checkpoint proc_lsst.multi INFO: found job 16964140 in provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:201 [INFO] cancelling 16964140 on provider checkpoint proc_lsst.multi INFO: cancelling 16964140 on provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:146 [INFO] found job 16964141 in provider checkpoint proc_lsst.multi INFO: found job 16964141 in provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:201 [INFO] cancelling 16964141 on provider checkpoint proc_lsst.multi INFO: cancelling 16964141 on provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:146 [INFO] found job 16964142 in provider checkpoint proc_lsst.multi INFO: found job 16964142 in provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:201 [INFO] cancelling 16964142 on provider checkpoint proc_lsst.multi INFO: cancelling 16964142 on provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:146 [INFO] found job 16964143 in provider checkpoint proc_lsst.multi INFO: found job 16964143 in provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:201 [INFO] cancelling 16964143 on provider checkpoint proc_lsst.multi INFO: cancelling 16964143 on provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:146 [INFO] found job 16964144 in provider checkpoint proc_lsst.multi INFO: found job 16964144 in provider checkpoint 2024-03-09 15:01:02 proc_lsst.multi:201 [INFO] cancelling 16964144 on provider checkpoint proc_lsst.multi INFO: cancelling 16964144 on provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:146 [INFO] found job 16964145 in provider checkpoint proc_lsst.multi INFO: found job 16964145 in provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:201 [INFO] cancelling 16964145 on provider checkpoint proc_lsst.multi INFO: cancelling 16964145 on provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:146 [INFO] found job 16964146 in provider checkpoint proc_lsst.multi INFO: found job 16964146 in provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:201 [INFO] cancelling 16964146 on provider checkpoint proc_lsst.multi INFO: cancelling 16964146 on provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:146 [INFO] found job 16964147 in provider checkpoint proc_lsst.multi INFO: found job 16964147 in provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:201 [INFO] cancelling 16964147 on provider checkpoint proc_lsst.multi INFO: cancelling 16964147 on provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:146 [INFO] found job 16964148 in provider checkpoint proc_lsst.multi INFO: found job 16964148 in provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:201 [INFO] cancelling 16964148 on provider checkpoint proc_lsst.multi INFO: cancelling 16964148 on provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:146 [INFO] found job 16964149 in provider checkpoint proc_lsst.multi INFO: found job 16964149 in provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:201 [INFO] cancelling 16964149 on provider checkpoint proc_lsst.multi INFO: cancelling 16964149 on provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:146 [INFO] found job 16964150 in provider checkpoint proc_lsst.multi INFO: found job 16964150 in provider checkpoint 2024-03-09 15:01:03 proc_lsst.multi:201 [INFO] cancelling 16964150 on provider checkpoint proc_lsst.multi INFO: cancelling 16964150 on provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:146 [INFO] found job 16964151 in provider checkpoint proc_lsst.multi INFO: found job 16964151 in provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:201 [INFO] cancelling 16964151 on provider checkpoint proc_lsst.multi INFO: cancelling 16964151 on provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:146 [INFO] found job 16964152 in provider checkpoint proc_lsst.multi INFO: found job 16964152 in provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:201 [INFO] cancelling 16964152 on provider checkpoint proc_lsst.multi INFO: cancelling 16964152 on provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:146 [INFO] found job 16964153 in provider checkpoint proc_lsst.multi INFO: found job 16964153 in provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:201 [INFO] cancelling 16964153 on provider checkpoint proc_lsst.multi INFO: cancelling 16964153 on provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:146 [INFO] found job 16964154 in provider checkpoint proc_lsst.multi INFO: found job 16964154 in provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:201 [INFO] cancelling 16964154 on provider checkpoint proc_lsst.multi INFO: cancelling 16964154 on provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:146 [INFO] found job 16964155 in provider checkpoint proc_lsst.multi INFO: found job 16964155 in provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:201 [INFO] cancelling 16964155 on provider checkpoint proc_lsst.multi INFO: cancelling 16964155 on provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:146 [INFO] found job 16964156 in provider checkpoint proc_lsst.multi INFO: found job 16964156 in provider checkpoint 2024-03-09 15:01:04 proc_lsst.multi:201 [INFO] cancelling 16964156 on provider checkpoint proc_lsst.multi INFO: cancelling 16964156 on provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:146 [INFO] found job 16964157 in provider checkpoint proc_lsst.multi INFO: found job 16964157 in provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:201 [INFO] cancelling 16964157 on provider checkpoint proc_lsst.multi INFO: cancelling 16964157 on provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:146 [INFO] found job 16964158 in provider checkpoint proc_lsst.multi INFO: found job 16964158 in provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:201 [INFO] cancelling 16964158 on provider checkpoint proc_lsst.multi INFO: cancelling 16964158 on provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:146 [INFO] found job 16964159 in provider checkpoint proc_lsst.multi INFO: found job 16964159 in provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:201 [INFO] cancelling 16964159 on provider checkpoint proc_lsst.multi INFO: cancelling 16964159 on provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:146 [INFO] found job 16964160 in provider checkpoint proc_lsst.multi INFO: found job 16964160 in provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:201 [INFO] cancelling 16964160 on provider checkpoint proc_lsst.multi INFO: cancelling 16964160 on provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:146 [INFO] found job 16964161 in provider checkpoint proc_lsst.multi INFO: found job 16964161 in provider checkpoint 2024-03-09 15:01:05 proc_lsst.multi:201 [INFO] cancelling 16964161 on provider checkpoint proc_lsst.multi INFO: cancelling 16964161 on provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:146 [INFO] found job 16964162 in provider checkpoint proc_lsst.multi INFO: found job 16964162 in provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:201 [INFO] cancelling 16964162 on provider checkpoint proc_lsst.multi INFO: cancelling 16964162 on provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:146 [INFO] found job 16964163 in provider checkpoint proc_lsst.multi INFO: found job 16964163 in provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:201 [INFO] cancelling 16964163 on provider checkpoint proc_lsst.multi INFO: cancelling 16964163 on provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:146 [INFO] found job 16964164 in provider checkpoint proc_lsst.multi INFO: found job 16964164 in provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:201 [INFO] cancelling 16964164 on provider checkpoint proc_lsst.multi INFO: cancelling 16964164 on provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:146 [INFO] found job 16964165 in provider checkpoint proc_lsst.multi INFO: found job 16964165 in provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:201 [INFO] cancelling 16964165 on provider checkpoint proc_lsst.multi INFO: cancelling 16964165 on provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:146 [INFO] found job 16964166 in provider checkpoint proc_lsst.multi INFO: found job 16964166 in provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:201 [INFO] cancelling 16964166 on provider checkpoint proc_lsst.multi INFO: cancelling 16964166 on provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:146 [INFO] found job 16964167 in provider checkpoint proc_lsst.multi INFO: found job 16964167 in provider checkpoint 2024-03-09 15:01:06 proc_lsst.multi:201 [INFO] cancelling 16964167 on provider checkpoint proc_lsst.multi INFO: cancelling 16964167 on provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:146 [INFO] found job 16964168 in provider checkpoint proc_lsst.multi INFO: found job 16964168 in provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:201 [INFO] cancelling 16964168 on provider checkpoint proc_lsst.multi INFO: cancelling 16964168 on provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:146 [INFO] found job 16964169 in provider checkpoint proc_lsst.multi INFO: found job 16964169 in provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:201 [INFO] cancelling 16964169 on provider checkpoint proc_lsst.multi INFO: cancelling 16964169 on provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:146 [INFO] found job 16964170 in provider checkpoint proc_lsst.multi INFO: found job 16964170 in provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:201 [INFO] cancelling 16964170 on provider checkpoint proc_lsst.multi INFO: cancelling 16964170 on provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:146 [INFO] found job 16964171 in provider checkpoint proc_lsst.multi INFO: found job 16964171 in provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:201 [INFO] cancelling 16964171 on provider checkpoint proc_lsst.multi INFO: cancelling 16964171 on provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:146 [INFO] found job 16964172 in provider checkpoint proc_lsst.multi INFO: found job 16964172 in provider checkpoint 2024-03-09 15:01:07 proc_lsst.multi:201 [INFO] cancelling 16964172 on provider checkpoint proc_lsst.multi INFO: cancelling 16964172 on provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:146 [INFO] found job 16964173 in provider checkpoint proc_lsst.multi INFO: found job 16964173 in provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:201 [INFO] cancelling 16964173 on provider checkpoint proc_lsst.multi INFO: cancelling 16964173 on provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:146 [INFO] found job 16964174 in provider checkpoint proc_lsst.multi INFO: found job 16964174 in provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:201 [INFO] cancelling 16964174 on provider checkpoint proc_lsst.multi INFO: cancelling 16964174 on provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:146 [INFO] found job 16964175 in provider checkpoint proc_lsst.multi INFO: found job 16964175 in provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:201 [INFO] cancelling 16964175 on provider checkpoint proc_lsst.multi INFO: cancelling 16964175 on provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:146 [INFO] found job 16964176 in provider checkpoint proc_lsst.multi INFO: found job 16964176 in provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:201 [INFO] cancelling 16964176 on provider checkpoint proc_lsst.multi INFO: cancelling 16964176 on provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:146 [INFO] found job 16964177 in provider checkpoint proc_lsst.multi INFO: found job 16964177 in provider checkpoint 2024-03-09 15:01:08 proc_lsst.multi:201 [INFO] cancelling 16964177 on provider checkpoint proc_lsst.multi INFO: cancelling 16964177 on provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:146 [INFO] found job 16964178 in provider checkpoint proc_lsst.multi INFO: found job 16964178 in provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:201 [INFO] cancelling 16964178 on provider checkpoint proc_lsst.multi INFO: cancelling 16964178 on provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:146 [INFO] found job 16964179 in provider checkpoint proc_lsst.multi INFO: found job 16964179 in provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:201 [INFO] cancelling 16964179 on provider checkpoint proc_lsst.multi INFO: cancelling 16964179 on provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:146 [INFO] found job 16964180 in provider checkpoint proc_lsst.multi INFO: found job 16964180 in provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:201 [INFO] cancelling 16964180 on provider checkpoint proc_lsst.multi INFO: cancelling 16964180 on provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:146 [INFO] found job 16964181 in provider checkpoint proc_lsst.multi INFO: found job 16964181 in provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:201 [INFO] cancelling 16964181 on provider checkpoint proc_lsst.multi INFO: cancelling 16964181 on provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:146 [INFO] found job 16964182 in provider checkpoint proc_lsst.multi INFO: found job 16964182 in provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:201 [INFO] cancelling 16964182 on provider checkpoint proc_lsst.multi INFO: cancelling 16964182 on provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:146 [INFO] found job 16964183 in provider checkpoint proc_lsst.multi INFO: found job 16964183 in provider checkpoint 2024-03-09 15:01:09 proc_lsst.multi:201 [INFO] cancelling 16964183 on provider checkpoint proc_lsst.multi INFO: cancelling 16964183 on provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:146 [INFO] found job 16964184 in provider checkpoint proc_lsst.multi INFO: found job 16964184 in provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:201 [INFO] cancelling 16964184 on provider checkpoint proc_lsst.multi INFO: cancelling 16964184 on provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:146 [INFO] found job 16964185 in provider checkpoint proc_lsst.multi INFO: found job 16964185 in provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:201 [INFO] cancelling 16964185 on provider checkpoint proc_lsst.multi INFO: cancelling 16964185 on provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:146 [INFO] found job 16964186 in provider checkpoint proc_lsst.multi INFO: found job 16964186 in provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:201 [INFO] cancelling 16964186 on provider checkpoint proc_lsst.multi INFO: cancelling 16964186 on provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:146 [INFO] found job 16964187 in provider checkpoint proc_lsst.multi INFO: found job 16964187 in provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:201 [INFO] cancelling 16964187 on provider checkpoint proc_lsst.multi INFO: cancelling 16964187 on provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:146 [INFO] found job 16964188 in provider checkpoint proc_lsst.multi INFO: found job 16964188 in provider checkpoint 2024-03-09 15:01:10 proc_lsst.multi:201 [INFO] cancelling 16964188 on provider checkpoint proc_lsst.multi INFO: cancelling 16964188 on provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:146 [INFO] found job 16964189 in provider checkpoint proc_lsst.multi INFO: found job 16964189 in provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:201 [INFO] cancelling 16964189 on provider checkpoint proc_lsst.multi INFO: cancelling 16964189 on provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:146 [INFO] found job 16964190 in provider checkpoint proc_lsst.multi INFO: found job 16964190 in provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:201 [INFO] cancelling 16964190 on provider checkpoint proc_lsst.multi INFO: cancelling 16964190 on provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:146 [INFO] found job 16964191 in provider checkpoint proc_lsst.multi INFO: found job 16964191 in provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:201 [INFO] cancelling 16964191 on provider checkpoint proc_lsst.multi INFO: cancelling 16964191 on provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:146 [INFO] found job 16964192 in provider checkpoint proc_lsst.multi INFO: found job 16964192 in provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:201 [INFO] cancelling 16964192 on provider checkpoint proc_lsst.multi INFO: cancelling 16964192 on provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:146 [INFO] found job 16964193 in provider checkpoint proc_lsst.multi INFO: found job 16964193 in provider checkpoint 2024-03-09 15:01:11 proc_lsst.multi:201 [INFO] cancelling 16964193 on provider checkpoint proc_lsst.multi INFO: cancelling 16964193 on provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:146 [INFO] found job 16964194 in provider checkpoint proc_lsst.multi INFO: found job 16964194 in provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:201 [INFO] cancelling 16964194 on provider checkpoint proc_lsst.multi INFO: cancelling 16964194 on provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:146 [INFO] found job 16964195 in provider checkpoint proc_lsst.multi INFO: found job 16964195 in provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:201 [INFO] cancelling 16964195 on provider checkpoint proc_lsst.multi INFO: cancelling 16964195 on provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:146 [INFO] found job 16964196 in provider checkpoint proc_lsst.multi INFO: found job 16964196 in provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:201 [INFO] cancelling 16964196 on provider checkpoint proc_lsst.multi INFO: cancelling 16964196 on provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:146 [INFO] found job 16964197 in provider checkpoint proc_lsst.multi INFO: found job 16964197 in provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:201 [INFO] cancelling 16964197 on provider checkpoint proc_lsst.multi INFO: cancelling 16964197 on provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:146 [INFO] found job 16964198 in provider checkpoint proc_lsst.multi INFO: found job 16964198 in provider checkpoint 2024-03-09 15:01:12 proc_lsst.multi:201 [INFO] cancelling 16964198 on provider checkpoint proc_lsst.multi INFO: cancelling 16964198 on provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:146 [INFO] found job 16964199 in provider checkpoint proc_lsst.multi INFO: found job 16964199 in provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:201 [INFO] cancelling 16964199 on provider checkpoint proc_lsst.multi INFO: cancelling 16964199 on provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:146 [INFO] found job 16964200 in provider checkpoint proc_lsst.multi INFO: found job 16964200 in provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:201 [INFO] cancelling 16964200 on provider checkpoint proc_lsst.multi INFO: cancelling 16964200 on provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:146 [INFO] found job 16964201 in provider checkpoint proc_lsst.multi INFO: found job 16964201 in provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:201 [INFO] cancelling 16964201 on provider checkpoint proc_lsst.multi INFO: cancelling 16964201 on provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:146 [INFO] found job 16964202 in provider checkpoint proc_lsst.multi INFO: found job 16964202 in provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:201 [INFO] cancelling 16964202 on provider checkpoint proc_lsst.multi INFO: cancelling 16964202 on provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:146 [INFO] found job 16964203 in provider checkpoint proc_lsst.multi INFO: found job 16964203 in provider checkpoint 2024-03-09 15:01:13 proc_lsst.multi:201 [INFO] cancelling 16964203 on provider checkpoint proc_lsst.multi INFO: cancelling 16964203 on provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:146 [INFO] found job 16964204 in provider checkpoint proc_lsst.multi INFO: found job 16964204 in provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:201 [INFO] cancelling 16964204 on provider checkpoint proc_lsst.multi INFO: cancelling 16964204 on provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:146 [INFO] found job 16964205 in provider checkpoint proc_lsst.multi INFO: found job 16964205 in provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:201 [INFO] cancelling 16964205 on provider checkpoint proc_lsst.multi INFO: cancelling 16964205 on provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:146 [INFO] found job 16964206 in provider checkpoint proc_lsst.multi INFO: found job 16964206 in provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:201 [INFO] cancelling 16964206 on provider checkpoint proc_lsst.multi INFO: cancelling 16964206 on provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:146 [INFO] found job 16964207 in provider checkpoint proc_lsst.multi INFO: found job 16964207 in provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:201 [INFO] cancelling 16964207 on provider checkpoint proc_lsst.multi INFO: cancelling 16964207 on provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:146 [INFO] found job 16964208 in provider checkpoint proc_lsst.multi INFO: found job 16964208 in provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:201 [INFO] cancelling 16964208 on provider checkpoint proc_lsst.multi INFO: cancelling 16964208 on provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:146 [INFO] found job 16964209 in provider checkpoint proc_lsst.multi INFO: found job 16964209 in provider checkpoint 2024-03-09 15:01:14 proc_lsst.multi:201 [INFO] cancelling 16964209 on provider checkpoint proc_lsst.multi INFO: cancelling 16964209 on provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:146 [INFO] found job 16964210 in provider checkpoint proc_lsst.multi INFO: found job 16964210 in provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:201 [INFO] cancelling 16964210 on provider checkpoint proc_lsst.multi INFO: cancelling 16964210 on provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:146 [INFO] found job 16964211 in provider checkpoint proc_lsst.multi INFO: found job 16964211 in provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:201 [INFO] cancelling 16964211 on provider checkpoint proc_lsst.multi INFO: cancelling 16964211 on provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:146 [INFO] found job 16964212 in provider checkpoint proc_lsst.multi INFO: found job 16964212 in provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:201 [INFO] cancelling 16964212 on provider checkpoint proc_lsst.multi INFO: cancelling 16964212 on provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:146 [INFO] found job 16964213 in provider checkpoint proc_lsst.multi INFO: found job 16964213 in provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:201 [INFO] cancelling 16964213 on provider checkpoint proc_lsst.multi INFO: cancelling 16964213 on provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:146 [INFO] found job 16964214 in provider checkpoint proc_lsst.multi INFO: found job 16964214 in provider checkpoint 2024-03-09 15:01:15 proc_lsst.multi:201 [INFO] cancelling 16964214 on provider checkpoint proc_lsst.multi INFO: cancelling 16964214 on provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:146 [INFO] found job 16964215 in provider checkpoint proc_lsst.multi INFO: found job 16964215 in provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:201 [INFO] cancelling 16964215 on provider checkpoint proc_lsst.multi INFO: cancelling 16964215 on provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:146 [INFO] found job 16964216 in provider checkpoint proc_lsst.multi INFO: found job 16964216 in provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:201 [INFO] cancelling 16964216 on provider checkpoint proc_lsst.multi INFO: cancelling 16964216 on provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:146 [INFO] found job 16964217 in provider checkpoint proc_lsst.multi INFO: found job 16964217 in provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:201 [INFO] cancelling 16964217 on provider checkpoint proc_lsst.multi INFO: cancelling 16964217 on provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:146 [INFO] found job 16964218 in provider checkpoint proc_lsst.multi INFO: found job 16964218 in provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:201 [INFO] cancelling 16964218 on provider checkpoint proc_lsst.multi INFO: cancelling 16964218 on provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:146 [INFO] found job 16964220 in provider checkpoint proc_lsst.multi INFO: found job 16964220 in provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:201 [INFO] cancelling 16964220 on provider checkpoint proc_lsst.multi INFO: cancelling 16964220 on provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:146 [INFO] found job 16964221 in provider checkpoint proc_lsst.multi INFO: found job 16964221 in provider checkpoint 2024-03-09 15:01:16 proc_lsst.multi:201 [INFO] cancelling 16964221 on provider checkpoint proc_lsst.multi INFO: cancelling 16964221 on provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:146 [INFO] found job 16964222 in provider checkpoint proc_lsst.multi INFO: found job 16964222 in provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:201 [INFO] cancelling 16964222 on provider checkpoint proc_lsst.multi INFO: cancelling 16964222 on provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:146 [INFO] found job 16964223 in provider checkpoint proc_lsst.multi INFO: found job 16964223 in provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:201 [INFO] cancelling 16964223 on provider checkpoint proc_lsst.multi INFO: cancelling 16964223 on provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:146 [INFO] found job 16964224 in provider checkpoint proc_lsst.multi INFO: found job 16964224 in provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:201 [INFO] cancelling 16964224 on provider checkpoint proc_lsst.multi INFO: cancelling 16964224 on provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:146 [INFO] found job 16964225 in provider checkpoint proc_lsst.multi INFO: found job 16964225 in provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:201 [INFO] cancelling 16964225 on provider checkpoint proc_lsst.multi INFO: cancelling 16964225 on provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:146 [INFO] found job 16964226 in provider checkpoint proc_lsst.multi INFO: found job 16964226 in provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:201 [INFO] cancelling 16964226 on provider checkpoint proc_lsst.multi INFO: cancelling 16964226 on provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:146 [INFO] found job 16964227 in provider checkpoint proc_lsst.multi INFO: found job 16964227 in provider checkpoint 2024-03-09 15:01:17 proc_lsst.multi:201 [INFO] cancelling 16964227 on provider checkpoint proc_lsst.multi INFO: cancelling 16964227 on provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:146 [INFO] found job 16964228 in provider checkpoint proc_lsst.multi INFO: found job 16964228 in provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:201 [INFO] cancelling 16964228 on provider checkpoint proc_lsst.multi INFO: cancelling 16964228 on provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:146 [INFO] found job 16964229 in provider checkpoint proc_lsst.multi INFO: found job 16964229 in provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:201 [INFO] cancelling 16964229 on provider checkpoint proc_lsst.multi INFO: cancelling 16964229 on provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:146 [INFO] found job 16964230 in provider checkpoint proc_lsst.multi INFO: found job 16964230 in provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:201 [INFO] cancelling 16964230 on provider checkpoint proc_lsst.multi INFO: cancelling 16964230 on provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:146 [INFO] found job 16964231 in provider checkpoint proc_lsst.multi INFO: found job 16964231 in provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:201 [INFO] cancelling 16964231 on provider checkpoint proc_lsst.multi INFO: cancelling 16964231 on provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:146 [INFO] found job 16964232 in provider checkpoint proc_lsst.multi INFO: found job 16964232 in provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:201 [INFO] cancelling 16964232 on provider checkpoint proc_lsst.multi INFO: cancelling 16964232 on provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:146 [INFO] found job 16964233 in provider checkpoint proc_lsst.multi INFO: found job 16964233 in provider checkpoint 2024-03-09 15:01:18 proc_lsst.multi:201 [INFO] cancelling 16964233 on provider checkpoint proc_lsst.multi INFO: cancelling 16964233 on provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:146 [INFO] found job 16964234 in provider checkpoint proc_lsst.multi INFO: found job 16964234 in provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:201 [INFO] cancelling 16964234 on provider checkpoint proc_lsst.multi INFO: cancelling 16964234 on provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:146 [INFO] found job 16964235 in provider checkpoint proc_lsst.multi INFO: found job 16964235 in provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:201 [INFO] cancelling 16964235 on provider checkpoint proc_lsst.multi INFO: cancelling 16964235 on provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:146 [INFO] found job 16964236 in provider checkpoint proc_lsst.multi INFO: found job 16964236 in provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:201 [INFO] cancelling 16964236 on provider checkpoint proc_lsst.multi INFO: cancelling 16964236 on provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:146 [INFO] found job 16964237 in provider checkpoint proc_lsst.multi INFO: found job 16964237 in provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:201 [INFO] cancelling 16964237 on provider checkpoint proc_lsst.multi INFO: cancelling 16964237 on provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:146 [INFO] found job 16964238 in provider checkpoint proc_lsst.multi INFO: found job 16964238 in provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:201 [INFO] cancelling 16964238 on provider checkpoint proc_lsst.multi INFO: cancelling 16964238 on provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:146 [INFO] found job 16964239 in provider checkpoint proc_lsst.multi INFO: found job 16964239 in provider checkpoint 2024-03-09 15:01:19 proc_lsst.multi:201 [INFO] cancelling 16964239 on provider checkpoint proc_lsst.multi INFO: cancelling 16964239 on provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:146 [INFO] found job 16964242 in provider checkpoint proc_lsst.multi INFO: found job 16964242 in provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:201 [INFO] cancelling 16964242 on provider checkpoint proc_lsst.multi INFO: cancelling 16964242 on provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:146 [INFO] found job 16964243 in provider checkpoint proc_lsst.multi INFO: found job 16964243 in provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:201 [INFO] cancelling 16964243 on provider checkpoint proc_lsst.multi INFO: cancelling 16964243 on provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:146 [INFO] found job 16964244 in provider checkpoint proc_lsst.multi INFO: found job 16964244 in provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:201 [INFO] cancelling 16964244 on provider checkpoint proc_lsst.multi INFO: cancelling 16964244 on provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:146 [INFO] found job 16964245 in provider checkpoint proc_lsst.multi INFO: found job 16964245 in provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:201 [INFO] cancelling 16964245 on provider checkpoint proc_lsst.multi INFO: cancelling 16964245 on provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:146 [INFO] found job 16964246 in provider checkpoint proc_lsst.multi INFO: found job 16964246 in provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:201 [INFO] cancelling 16964246 on provider checkpoint proc_lsst.multi INFO: cancelling 16964246 on provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:146 [INFO] found job 16964247 in provider checkpoint proc_lsst.multi INFO: found job 16964247 in provider checkpoint 2024-03-09 15:01:20 proc_lsst.multi:201 [INFO] cancelling 16964247 on provider checkpoint proc_lsst.multi INFO: cancelling 16964247 on provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:146 [INFO] found job 16964248 in provider checkpoint proc_lsst.multi INFO: found job 16964248 in provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:201 [INFO] cancelling 16964248 on provider checkpoint proc_lsst.multi INFO: cancelling 16964248 on provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:146 [INFO] found job 16964249 in provider checkpoint proc_lsst.multi INFO: found job 16964249 in provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:201 [INFO] cancelling 16964249 on provider checkpoint proc_lsst.multi INFO: cancelling 16964249 on provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:146 [INFO] found job 16964250 in provider checkpoint proc_lsst.multi INFO: found job 16964250 in provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:201 [INFO] cancelling 16964250 on provider checkpoint proc_lsst.multi INFO: cancelling 16964250 on provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:146 [INFO] found job 16964251 in provider checkpoint proc_lsst.multi INFO: found job 16964251 in provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:201 [INFO] cancelling 16964251 on provider checkpoint proc_lsst.multi INFO: cancelling 16964251 on provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:146 [INFO] found job 16964252 in provider checkpoint proc_lsst.multi INFO: found job 16964252 in provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:201 [INFO] cancelling 16964252 on provider checkpoint proc_lsst.multi INFO: cancelling 16964252 on provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:146 [INFO] found job 16964253 in provider checkpoint proc_lsst.multi INFO: found job 16964253 in provider checkpoint 2024-03-09 15:01:21 proc_lsst.multi:201 [INFO] cancelling 16964253 on provider checkpoint proc_lsst.multi INFO: cancelling 16964253 on provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:146 [INFO] found job 16964254 in provider checkpoint proc_lsst.multi INFO: found job 16964254 in provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:201 [INFO] cancelling 16964254 on provider checkpoint proc_lsst.multi INFO: cancelling 16964254 on provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:146 [INFO] found job 16964255 in provider checkpoint proc_lsst.multi INFO: found job 16964255 in provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:201 [INFO] cancelling 16964255 on provider checkpoint proc_lsst.multi INFO: cancelling 16964255 on provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:146 [INFO] found job 16964256 in provider checkpoint proc_lsst.multi INFO: found job 16964256 in provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:201 [INFO] cancelling 16964256 on provider checkpoint proc_lsst.multi INFO: cancelling 16964256 on provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:146 [INFO] found job 16964257 in provider checkpoint proc_lsst.multi INFO: found job 16964257 in provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:201 [INFO] cancelling 16964257 on provider checkpoint proc_lsst.multi INFO: cancelling 16964257 on provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:146 [INFO] found job 16964258 in provider checkpoint proc_lsst.multi INFO: found job 16964258 in provider checkpoint 2024-03-09 15:01:22 proc_lsst.multi:201 [INFO] cancelling 16964258 on provider checkpoint proc_lsst.multi INFO: cancelling 16964258 on provider checkpoint 2024-03-09 15:01:23 proc_lsst.multi:146 [INFO] found job 16964259 in provider checkpoint proc_lsst.multi INFO: found job 16964259 in provider checkpoint 2024-03-09 15:01:23 proc_lsst.multi:201 [INFO] cancelling 16964259 on provider checkpoint proc_lsst.multi INFO: cancelling 16964259 on provider checkpoint 2024-03-09 15:01:23 proc_lsst.multi:146 [INFO] found job 16964260 in provider checkpoint proc_lsst.multi INFO: found job 16964260 in provider checkpoint 2024-03-09 15:01:23 proc_lsst.multi:201 [INFO] cancelling 16964260 on provider checkpoint proc_lsst.multi INFO: cancelling 16964260 on provider checkpoint 2024-03-09 15:01:24 proc_lsst.multi:146 [INFO] found job 16964261 in provider checkpoint proc_lsst.multi INFO: found job 16964261 in provider checkpoint 2024-03-09 15:01:24 proc_lsst.multi:201 [INFO] cancelling 16964261 on provider checkpoint proc_lsst.multi INFO: cancelling 16964261 on provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:146 [INFO] found job 16964262 in provider checkpoint proc_lsst.multi INFO: found job 16964262 in provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:201 [INFO] cancelling 16964262 on provider checkpoint proc_lsst.multi INFO: cancelling 16964262 on provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:146 [INFO] found job 16964263 in provider checkpoint proc_lsst.multi INFO: found job 16964263 in provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:201 [INFO] cancelling 16964263 on provider checkpoint proc_lsst.multi INFO: cancelling 16964263 on provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:146 [INFO] found job 16964264 in provider checkpoint proc_lsst.multi INFO: found job 16964264 in provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:201 [INFO] cancelling 16964264 on provider checkpoint proc_lsst.multi INFO: cancelling 16964264 on provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:146 [INFO] found job 16964265 in provider checkpoint proc_lsst.multi INFO: found job 16964265 in provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:201 [INFO] cancelling 16964265 on provider checkpoint proc_lsst.multi INFO: cancelling 16964265 on provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:146 [INFO] found job 16964266 in provider checkpoint proc_lsst.multi INFO: found job 16964266 in provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:201 [INFO] cancelling 16964266 on provider checkpoint proc_lsst.multi INFO: cancelling 16964266 on provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:146 [INFO] found job 16964267 in provider checkpoint proc_lsst.multi INFO: found job 16964267 in provider checkpoint 2024-03-09 15:01:25 proc_lsst.multi:201 [INFO] cancelling 16964267 on provider checkpoint proc_lsst.multi INFO: cancelling 16964267 on provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:146 [INFO] found job 16964268 in provider checkpoint proc_lsst.multi INFO: found job 16964268 in provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:201 [INFO] cancelling 16964268 on provider checkpoint proc_lsst.multi INFO: cancelling 16964268 on provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:146 [INFO] found job 16964269 in provider checkpoint proc_lsst.multi INFO: found job 16964269 in provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:201 [INFO] cancelling 16964269 on provider checkpoint proc_lsst.multi INFO: cancelling 16964269 on provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:146 [INFO] found job 16964270 in provider checkpoint proc_lsst.multi INFO: found job 16964270 in provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:201 [INFO] cancelling 16964270 on provider checkpoint proc_lsst.multi INFO: cancelling 16964270 on provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:146 [INFO] found job 16964271 in provider checkpoint proc_lsst.multi INFO: found job 16964271 in provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:201 [INFO] cancelling 16964271 on provider checkpoint proc_lsst.multi INFO: cancelling 16964271 on provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:146 [INFO] found job 16964272 in provider checkpoint proc_lsst.multi INFO: found job 16964272 in provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:201 [INFO] cancelling 16964272 on provider checkpoint proc_lsst.multi INFO: cancelling 16964272 on provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:146 [INFO] found job 16964273 in provider checkpoint proc_lsst.multi INFO: found job 16964273 in provider checkpoint 2024-03-09 15:01:26 proc_lsst.multi:201 [INFO] cancelling 16964273 on provider checkpoint proc_lsst.multi INFO: cancelling 16964273 on provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:146 [INFO] found job 16964274 in provider checkpoint proc_lsst.multi INFO: found job 16964274 in provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:201 [INFO] cancelling 16964274 on provider checkpoint proc_lsst.multi INFO: cancelling 16964274 on provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:146 [INFO] found job 16964275 in provider checkpoint proc_lsst.multi INFO: found job 16964275 in provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:201 [INFO] cancelling 16964275 on provider checkpoint proc_lsst.multi INFO: cancelling 16964275 on provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:146 [INFO] found job 16964276 in provider checkpoint proc_lsst.multi INFO: found job 16964276 in provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:201 [INFO] cancelling 16964276 on provider checkpoint proc_lsst.multi INFO: cancelling 16964276 on provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:146 [INFO] found job 16964277 in provider checkpoint proc_lsst.multi INFO: found job 16964277 in provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:201 [INFO] cancelling 16964277 on provider checkpoint proc_lsst.multi INFO: cancelling 16964277 on provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:146 [INFO] found job 16964278 in provider checkpoint proc_lsst.multi INFO: found job 16964278 in provider checkpoint 2024-03-09 15:01:27 proc_lsst.multi:201 [INFO] cancelling 16964278 on provider checkpoint proc_lsst.multi INFO: cancelling 16964278 on provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:146 [INFO] found job 16964284 in provider checkpoint proc_lsst.multi INFO: found job 16964284 in provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:201 [INFO] cancelling 16964284 on provider checkpoint proc_lsst.multi INFO: cancelling 16964284 on provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:146 [INFO] found job 16964286 in provider checkpoint proc_lsst.multi INFO: found job 16964286 in provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:201 [INFO] cancelling 16964286 on provider checkpoint proc_lsst.multi INFO: cancelling 16964286 on provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:146 [INFO] found job 16964287 in provider checkpoint proc_lsst.multi INFO: found job 16964287 in provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:201 [INFO] cancelling 16964287 on provider checkpoint proc_lsst.multi INFO: cancelling 16964287 on provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:146 [INFO] found job 16964288 in provider checkpoint proc_lsst.multi INFO: found job 16964288 in provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:201 [INFO] cancelling 16964288 on provider checkpoint proc_lsst.multi INFO: cancelling 16964288 on provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:146 [INFO] found job 16964289 in provider checkpoint proc_lsst.multi INFO: found job 16964289 in provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:201 [INFO] cancelling 16964289 on provider checkpoint proc_lsst.multi INFO: cancelling 16964289 on provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:146 [INFO] found job 16964290 in provider checkpoint proc_lsst.multi INFO: found job 16964290 in provider checkpoint 2024-03-09 15:01:28 proc_lsst.multi:201 [INFO] cancelling 16964290 on provider checkpoint proc_lsst.multi INFO: cancelling 16964290 on provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:146 [INFO] found job 16964291 in provider checkpoint proc_lsst.multi INFO: found job 16964291 in provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:201 [INFO] cancelling 16964291 on provider checkpoint proc_lsst.multi INFO: cancelling 16964291 on provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:146 [INFO] found job 16964292 in provider checkpoint proc_lsst.multi INFO: found job 16964292 in provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:201 [INFO] cancelling 16964292 on provider checkpoint proc_lsst.multi INFO: cancelling 16964292 on provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:146 [INFO] found job 16964293 in provider checkpoint proc_lsst.multi INFO: found job 16964293 in provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:201 [INFO] cancelling 16964293 on provider checkpoint proc_lsst.multi INFO: cancelling 16964293 on provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:146 [INFO] found job 16964294 in provider checkpoint proc_lsst.multi INFO: found job 16964294 in provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:201 [INFO] cancelling 16964294 on provider checkpoint proc_lsst.multi INFO: cancelling 16964294 on provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:146 [INFO] found job 16964295 in provider checkpoint proc_lsst.multi INFO: found job 16964295 in provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:201 [INFO] cancelling 16964295 on provider checkpoint proc_lsst.multi INFO: cancelling 16964295 on provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:146 [INFO] found job 16964296 in provider checkpoint proc_lsst.multi INFO: found job 16964296 in provider checkpoint 2024-03-09 15:01:29 proc_lsst.multi:201 [INFO] cancelling 16964296 on provider checkpoint proc_lsst.multi INFO: cancelling 16964296 on provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:146 [INFO] found job 16964297 in provider checkpoint proc_lsst.multi INFO: found job 16964297 in provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:201 [INFO] cancelling 16964297 on provider checkpoint proc_lsst.multi INFO: cancelling 16964297 on provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:146 [INFO] found job 16964298 in provider checkpoint proc_lsst.multi INFO: found job 16964298 in provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:201 [INFO] cancelling 16964298 on provider checkpoint proc_lsst.multi INFO: cancelling 16964298 on provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:146 [INFO] found job 16964299 in provider checkpoint proc_lsst.multi INFO: found job 16964299 in provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:201 [INFO] cancelling 16964299 on provider checkpoint proc_lsst.multi INFO: cancelling 16964299 on provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:146 [INFO] found job 16964300 in provider checkpoint proc_lsst.multi INFO: found job 16964300 in provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:201 [INFO] cancelling 16964300 on provider checkpoint proc_lsst.multi INFO: cancelling 16964300 on provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:146 [INFO] found job 16964301 in provider checkpoint proc_lsst.multi INFO: found job 16964301 in provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:201 [INFO] cancelling 16964301 on provider checkpoint proc_lsst.multi INFO: cancelling 16964301 on provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:146 [INFO] found job 16964304 in provider checkpoint proc_lsst.multi INFO: found job 16964304 in provider checkpoint 2024-03-09 15:01:30 proc_lsst.multi:201 [INFO] cancelling 16964304 on provider checkpoint proc_lsst.multi INFO: cancelling 16964304 on provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:146 [INFO] found job 16964305 in provider checkpoint proc_lsst.multi INFO: found job 16964305 in provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:201 [INFO] cancelling 16964305 on provider checkpoint proc_lsst.multi INFO: cancelling 16964305 on provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:146 [INFO] found job 16964306 in provider checkpoint proc_lsst.multi INFO: found job 16964306 in provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:201 [INFO] cancelling 16964306 on provider checkpoint proc_lsst.multi INFO: cancelling 16964306 on provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:146 [INFO] found job 16964307 in provider checkpoint proc_lsst.multi INFO: found job 16964307 in provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:201 [INFO] cancelling 16964307 on provider checkpoint proc_lsst.multi INFO: cancelling 16964307 on provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:146 [INFO] found job 16964308 in provider checkpoint proc_lsst.multi INFO: found job 16964308 in provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:201 [INFO] cancelling 16964308 on provider checkpoint proc_lsst.multi INFO: cancelling 16964308 on provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:146 [INFO] found job 16964309 in provider checkpoint proc_lsst.multi INFO: found job 16964309 in provider checkpoint 2024-03-09 15:01:31 proc_lsst.multi:201 [INFO] cancelling 16964309 on provider checkpoint proc_lsst.multi INFO: cancelling 16964309 on provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:146 [INFO] found job 16964310 in provider checkpoint proc_lsst.multi INFO: found job 16964310 in provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:201 [INFO] cancelling 16964310 on provider checkpoint proc_lsst.multi INFO: cancelling 16964310 on provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:146 [INFO] found job 16964311 in provider checkpoint proc_lsst.multi INFO: found job 16964311 in provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:201 [INFO] cancelling 16964311 on provider checkpoint proc_lsst.multi INFO: cancelling 16964311 on provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:146 [INFO] found job 16964312 in provider checkpoint proc_lsst.multi INFO: found job 16964312 in provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:201 [INFO] cancelling 16964312 on provider checkpoint proc_lsst.multi INFO: cancelling 16964312 on provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:146 [INFO] found job 16964313 in provider checkpoint proc_lsst.multi INFO: found job 16964313 in provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:201 [INFO] cancelling 16964313 on provider checkpoint proc_lsst.multi INFO: cancelling 16964313 on provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:146 [INFO] found job 16964314 in provider checkpoint proc_lsst.multi INFO: found job 16964314 in provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:201 [INFO] cancelling 16964314 on provider checkpoint proc_lsst.multi INFO: cancelling 16964314 on provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:146 [INFO] found job 16964315 in provider checkpoint proc_lsst.multi INFO: found job 16964315 in provider checkpoint 2024-03-09 15:01:32 proc_lsst.multi:201 [INFO] cancelling 16964315 on provider checkpoint proc_lsst.multi INFO: cancelling 16964315 on provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:146 [INFO] found job 16964316 in provider checkpoint proc_lsst.multi INFO: found job 16964316 in provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:201 [INFO] cancelling 16964316 on provider checkpoint proc_lsst.multi INFO: cancelling 16964316 on provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:146 [INFO] found job 16964317 in provider checkpoint proc_lsst.multi INFO: found job 16964317 in provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:201 [INFO] cancelling 16964317 on provider checkpoint proc_lsst.multi INFO: cancelling 16964317 on provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:146 [INFO] found job 16964318 in provider checkpoint proc_lsst.multi INFO: found job 16964318 in provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:201 [INFO] cancelling 16964318 on provider checkpoint proc_lsst.multi INFO: cancelling 16964318 on provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:146 [INFO] found job 16964319 in provider checkpoint proc_lsst.multi INFO: found job 16964319 in provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:201 [INFO] cancelling 16964319 on provider checkpoint proc_lsst.multi INFO: cancelling 16964319 on provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:146 [INFO] found job 16964320 in provider checkpoint proc_lsst.multi INFO: found job 16964320 in provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:201 [INFO] cancelling 16964320 on provider checkpoint proc_lsst.multi INFO: cancelling 16964320 on provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:146 [INFO] found job 16964321 in provider checkpoint proc_lsst.multi INFO: found job 16964321 in provider checkpoint 2024-03-09 15:01:33 proc_lsst.multi:201 [INFO] cancelling 16964321 on provider checkpoint proc_lsst.multi INFO: cancelling 16964321 on provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:146 [INFO] found job 16964322 in provider checkpoint proc_lsst.multi INFO: found job 16964322 in provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:201 [INFO] cancelling 16964322 on provider checkpoint proc_lsst.multi INFO: cancelling 16964322 on provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:146 [INFO] found job 16964323 in provider checkpoint proc_lsst.multi INFO: found job 16964323 in provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:201 [INFO] cancelling 16964323 on provider checkpoint proc_lsst.multi INFO: cancelling 16964323 on provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:146 [INFO] found job 16964324 in provider checkpoint proc_lsst.multi INFO: found job 16964324 in provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:201 [INFO] cancelling 16964324 on provider checkpoint proc_lsst.multi INFO: cancelling 16964324 on provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:146 [INFO] found job 16964325 in provider checkpoint proc_lsst.multi INFO: found job 16964325 in provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:201 [INFO] cancelling 16964325 on provider checkpoint proc_lsst.multi INFO: cancelling 16964325 on provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:146 [INFO] found job 16964326 in provider checkpoint proc_lsst.multi INFO: found job 16964326 in provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:201 [INFO] cancelling 16964326 on provider checkpoint proc_lsst.multi INFO: cancelling 16964326 on provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:146 [INFO] found job 16964327 in provider checkpoint proc_lsst.multi INFO: found job 16964327 in provider checkpoint 2024-03-09 15:01:34 proc_lsst.multi:201 [INFO] cancelling 16964327 on provider checkpoint proc_lsst.multi INFO: cancelling 16964327 on provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:146 [INFO] found job 16964328 in provider checkpoint proc_lsst.multi INFO: found job 16964328 in provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:201 [INFO] cancelling 16964328 on provider checkpoint proc_lsst.multi INFO: cancelling 16964328 on provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:146 [INFO] found job 16964329 in provider checkpoint proc_lsst.multi INFO: found job 16964329 in provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:201 [INFO] cancelling 16964329 on provider checkpoint proc_lsst.multi INFO: cancelling 16964329 on provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:146 [INFO] found job 16964330 in provider checkpoint proc_lsst.multi INFO: found job 16964330 in provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:201 [INFO] cancelling 16964330 on provider checkpoint proc_lsst.multi INFO: cancelling 16964330 on provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:146 [INFO] found job 16964331 in provider checkpoint proc_lsst.multi INFO: found job 16964331 in provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:201 [INFO] cancelling 16964331 on provider checkpoint proc_lsst.multi INFO: cancelling 16964331 on provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:146 [INFO] found job 16964332 in provider checkpoint proc_lsst.multi INFO: found job 16964332 in provider checkpoint 2024-03-09 15:01:35 proc_lsst.multi:201 [INFO] cancelling 16964332 on provider checkpoint proc_lsst.multi INFO: cancelling 16964332 on provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:146 [INFO] found job 16964333 in provider checkpoint proc_lsst.multi INFO: found job 16964333 in provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:201 [INFO] cancelling 16964333 on provider checkpoint proc_lsst.multi INFO: cancelling 16964333 on provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:146 [INFO] found job 16964334 in provider checkpoint proc_lsst.multi INFO: found job 16964334 in provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:201 [INFO] cancelling 16964334 on provider checkpoint proc_lsst.multi INFO: cancelling 16964334 on provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:146 [INFO] found job 16964335 in provider checkpoint proc_lsst.multi INFO: found job 16964335 in provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:201 [INFO] cancelling 16964335 on provider checkpoint proc_lsst.multi INFO: cancelling 16964335 on provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:146 [INFO] found job 16964336 in provider checkpoint proc_lsst.multi INFO: found job 16964336 in provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:201 [INFO] cancelling 16964336 on provider checkpoint proc_lsst.multi INFO: cancelling 16964336 on provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:146 [INFO] found job 16964337 in provider checkpoint proc_lsst.multi INFO: found job 16964337 in provider checkpoint 2024-03-09 15:01:36 proc_lsst.multi:201 [INFO] cancelling 16964337 on provider checkpoint proc_lsst.multi INFO: cancelling 16964337 on provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:146 [INFO] found job 16964338 in provider checkpoint proc_lsst.multi INFO: found job 16964338 in provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:201 [INFO] cancelling 16964338 on provider checkpoint proc_lsst.multi INFO: cancelling 16964338 on provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:146 [INFO] found job 16964339 in provider checkpoint proc_lsst.multi INFO: found job 16964339 in provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:201 [INFO] cancelling 16964339 on provider checkpoint proc_lsst.multi INFO: cancelling 16964339 on provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:146 [INFO] found job 16964340 in provider checkpoint proc_lsst.multi INFO: found job 16964340 in provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:201 [INFO] cancelling 16964340 on provider checkpoint proc_lsst.multi INFO: cancelling 16964340 on provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:146 [INFO] found job 16964341 in provider checkpoint proc_lsst.multi INFO: found job 16964341 in provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:201 [INFO] cancelling 16964341 on provider checkpoint proc_lsst.multi INFO: cancelling 16964341 on provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:146 [INFO] found job 16964342 in provider checkpoint proc_lsst.multi INFO: found job 16964342 in provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:201 [INFO] cancelling 16964342 on provider checkpoint proc_lsst.multi INFO: cancelling 16964342 on provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:146 [INFO] found job 16964343 in provider checkpoint proc_lsst.multi INFO: found job 16964343 in provider checkpoint 2024-03-09 15:01:37 proc_lsst.multi:201 [INFO] cancelling 16964343 on provider checkpoint proc_lsst.multi INFO: cancelling 16964343 on provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:146 [INFO] found job 16964344 in provider checkpoint proc_lsst.multi INFO: found job 16964344 in provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:201 [INFO] cancelling 16964344 on provider checkpoint proc_lsst.multi INFO: cancelling 16964344 on provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:146 [INFO] found job 16964345 in provider checkpoint proc_lsst.multi INFO: found job 16964345 in provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:201 [INFO] cancelling 16964345 on provider checkpoint proc_lsst.multi INFO: cancelling 16964345 on provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:146 [INFO] found job 16964346 in provider checkpoint proc_lsst.multi INFO: found job 16964346 in provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:201 [INFO] cancelling 16964346 on provider checkpoint proc_lsst.multi INFO: cancelling 16964346 on provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:146 [INFO] found job 16964347 in provider checkpoint proc_lsst.multi INFO: found job 16964347 in provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:201 [INFO] cancelling 16964347 on provider checkpoint proc_lsst.multi INFO: cancelling 16964347 on provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:146 [INFO] found job 16964348 in provider checkpoint proc_lsst.multi INFO: found job 16964348 in provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:201 [INFO] cancelling 16964348 on provider checkpoint proc_lsst.multi INFO: cancelling 16964348 on provider checkpoint 2024-03-09 15:01:38 proc_lsst.multi:146 [INFO] found job 16964026 in provider astro proc_lsst.multi INFO: found job 16964026 in provider astro 2024-03-09 15:01:38 proc_lsst.multi:201 [INFO] cancelling 16964026 on provider astro proc_lsst.multi INFO: cancelling 16964026 on provider astro 2024-03-09 15:01:39 proc_lsst.multi:146 [INFO] found job 16964027 in provider astro proc_lsst.multi INFO: found job 16964027 in provider astro 2024-03-09 15:01:39 proc_lsst.multi:201 [INFO] cancelling 16964027 on provider astro proc_lsst.multi INFO: cancelling 16964027 on provider astro 2024-03-09 15:01:39 proc_lsst.multi:146 [INFO] found job 16964028 in provider astro proc_lsst.multi INFO: found job 16964028 in provider astro 2024-03-09 15:01:39 proc_lsst.multi:201 [INFO] cancelling 16964028 on provider astro proc_lsst.multi INFO: cancelling 16964028 on provider astro 2024-03-09 15:01:39 proc_lsst.multi:146 [INFO] found job 16964029 in provider astro proc_lsst.multi INFO: found job 16964029 in provider astro 2024-03-09 15:01:39 proc_lsst.multi:201 [INFO] cancelling 16964029 on provider astro proc_lsst.multi INFO: cancelling 16964029 on provider astro 2024-03-09 15:01:39 proc_lsst.multi:146 [INFO] found job 16964030 in provider astro proc_lsst.multi INFO: found job 16964030 in provider astro 2024-03-09 15:01:39 proc_lsst.multi:201 [INFO] cancelling 16964030 on provider astro proc_lsst.multi INFO: cancelling 16964030 on provider astro 2024-03-09 15:01:39 proc_lsst.multi:146 [INFO] found job 16964031 in provider astro proc_lsst.multi INFO: found job 16964031 in provider astro 2024-03-09 15:01:39 proc_lsst.multi:201 [INFO] cancelling 16964031 on provider astro proc_lsst.multi INFO: cancelling 16964031 on provider astro 2024-03-09 15:01:40 proc_lsst.multi:146 [INFO] found job 16964032 in provider astro proc_lsst.multi INFO: found job 16964032 in provider astro 2024-03-09 15:01:40 proc_lsst.multi:201 [INFO] cancelling 16964032 on provider astro proc_lsst.multi INFO: cancelling 16964032 on provider astro 2024-03-09 15:01:40 proc_lsst.multi:146 [INFO] found job 16964033 in provider astro proc_lsst.multi INFO: found job 16964033 in provider astro 2024-03-09 15:01:40 proc_lsst.multi:201 [INFO] cancelling 16964033 on provider astro proc_lsst.multi INFO: cancelling 16964033 on provider astro 2024-03-09 15:01:40 proc_lsst.multi:146 [INFO] found job 16964034 in provider astro proc_lsst.multi INFO: found job 16964034 in provider astro 2024-03-09 15:01:40 proc_lsst.multi:201 [INFO] cancelling 16964034 on provider astro proc_lsst.multi INFO: cancelling 16964034 on provider astro 2024-03-09 15:01:40 proc_lsst.multi:146 [INFO] found job 16964035 in provider astro proc_lsst.multi INFO: found job 16964035 in provider astro 2024-03-09 15:01:40 proc_lsst.multi:201 [INFO] cancelling 16964035 on provider astro proc_lsst.multi INFO: cancelling 16964035 on provider astro 2024-03-09 15:01:40 proc_lsst.multi:146 [INFO] found job 16964036 in provider astro proc_lsst.multi INFO: found job 16964036 in provider astro 2024-03-09 15:01:40 proc_lsst.multi:201 [INFO] cancelling 16964036 on provider astro proc_lsst.multi INFO: cancelling 16964036 on provider astro 2024-03-09 15:01:40 proc_lsst.multi:146 [INFO] found job 16964037 in provider astro proc_lsst.multi INFO: found job 16964037 in provider astro 2024-03-09 15:01:40 proc_lsst.multi:201 [INFO] cancelling 16964037 on provider astro proc_lsst.multi INFO: cancelling 16964037 on provider astro 2024-03-09 15:01:41 proc_lsst.multi:146 [INFO] found job 16964038 in provider astro proc_lsst.multi INFO: found job 16964038 in provider astro 2024-03-09 15:01:41 proc_lsst.multi:201 [INFO] cancelling 16964038 on provider astro proc_lsst.multi INFO: cancelling 16964038 on provider astro 2024-03-09 15:01:41 proc_lsst.multi:146 [INFO] found job 16964039 in provider astro proc_lsst.multi INFO: found job 16964039 in provider astro 2024-03-09 15:01:41 proc_lsst.multi:201 [INFO] cancelling 16964039 on provider astro proc_lsst.multi INFO: cancelling 16964039 on provider astro 2024-03-09 15:01:41 proc_lsst.multi:146 [INFO] found job 16964040 in provider astro proc_lsst.multi INFO: found job 16964040 in provider astro 2024-03-09 15:01:41 proc_lsst.multi:201 [INFO] cancelling 16964040 on provider astro proc_lsst.multi INFO: cancelling 16964040 on provider astro 2024-03-09 15:01:41 proc_lsst.multi:146 [INFO] found job 16964041 in provider astro proc_lsst.multi INFO: found job 16964041 in provider astro 2024-03-09 15:01:41 proc_lsst.multi:201 [INFO] cancelling 16964041 on provider astro proc_lsst.multi INFO: cancelling 16964041 on provider astro 2024-03-09 15:01:41 proc_lsst.multi:146 [INFO] found job 16964042 in provider astro proc_lsst.multi INFO: found job 16964042 in provider astro 2024-03-09 15:01:41 proc_lsst.multi:201 [INFO] cancelling 16964042 on provider astro proc_lsst.multi INFO: cancelling 16964042 on provider astro 2024-03-09 15:01:41 proc_lsst.multi:146 [INFO] found job 16964043 in provider astro proc_lsst.multi INFO: found job 16964043 in provider astro 2024-03-09 15:01:41 proc_lsst.multi:201 [INFO] cancelling 16964043 on provider astro proc_lsst.multi INFO: cancelling 16964043 on provider astro 2024-03-09 15:01:42 proc_lsst.multi:146 [INFO] found job 16964044 in provider astro proc_lsst.multi INFO: found job 16964044 in provider astro 2024-03-09 15:01:42 proc_lsst.multi:201 [INFO] cancelling 16964044 on provider astro proc_lsst.multi INFO: cancelling 16964044 on provider astro 2024-03-09 15:01:42 proc_lsst.multi:146 [INFO] found job 16964045 in provider astro proc_lsst.multi INFO: found job 16964045 in provider astro 2024-03-09 15:01:42 proc_lsst.multi:201 [INFO] cancelling 16964045 on provider astro proc_lsst.multi INFO: cancelling 16964045 on provider astro 2024-03-09 15:01:42 proc_lsst.multi:146 [INFO] found job 16964046 in provider astro proc_lsst.multi INFO: found job 16964046 in provider astro 2024-03-09 15:01:42 proc_lsst.multi:201 [INFO] cancelling 16964046 on provider astro proc_lsst.multi INFO: cancelling 16964046 on provider astro 2024-03-09 15:01:42 proc_lsst.multi:146 [INFO] found job 16964047 in provider astro proc_lsst.multi INFO: found job 16964047 in provider astro 2024-03-09 15:01:42 proc_lsst.multi:201 [INFO] cancelling 16964047 on provider astro proc_lsst.multi INFO: cancelling 16964047 on provider astro 2024-03-09 15:01:42 proc_lsst.multi:146 [INFO] found job 16964048 in provider astro proc_lsst.multi INFO: found job 16964048 in provider astro 2024-03-09 15:01:42 proc_lsst.multi:201 [INFO] cancelling 16964048 on provider astro proc_lsst.multi INFO: cancelling 16964048 on provider astro 2024-03-09 15:01:42 proc_lsst.multi:146 [INFO] found job 16964049 in provider astro proc_lsst.multi INFO: found job 16964049 in provider astro 2024-03-09 15:01:42 proc_lsst.multi:201 [INFO] cancelling 16964049 on provider astro proc_lsst.multi INFO: cancelling 16964049 on provider astro 2024-03-09 15:01:43 proc_lsst.multi:146 [INFO] found job 16964050 in provider astro proc_lsst.multi INFO: found job 16964050 in provider astro 2024-03-09 15:01:43 proc_lsst.multi:201 [INFO] cancelling 16964050 on provider astro proc_lsst.multi INFO: cancelling 16964050 on provider astro 2024-03-09 15:01:43 proc_lsst.multi:146 [INFO] found job 16964051 in provider astro proc_lsst.multi INFO: found job 16964051 in provider astro 2024-03-09 15:01:43 proc_lsst.multi:201 [INFO] cancelling 16964051 on provider astro proc_lsst.multi INFO: cancelling 16964051 on provider astro 2024-03-09 15:01:43 proc_lsst.multi:146 [INFO] found job 16964052 in provider astro proc_lsst.multi INFO: found job 16964052 in provider astro 2024-03-09 15:01:43 proc_lsst.multi:201 [INFO] cancelling 16964052 on provider astro proc_lsst.multi INFO: cancelling 16964052 on provider astro 2024-03-09 15:01:43 proc_lsst.multi:146 [INFO] found job 16964053 in provider astro proc_lsst.multi INFO: found job 16964053 in provider astro 2024-03-09 15:01:43 proc_lsst.multi:201 [INFO] cancelling 16964053 on provider astro proc_lsst.multi INFO: cancelling 16964053 on provider astro 2024-03-09 15:01:43 proc_lsst.multi:146 [INFO] found job 16964054 in provider astro proc_lsst.multi INFO: found job 16964054 in provider astro 2024-03-09 15:01:43 proc_lsst.multi:201 [INFO] cancelling 16964054 on provider astro proc_lsst.multi INFO: cancelling 16964054 on provider astro 2024-03-09 15:01:43 proc_lsst.multi:146 [INFO] found job 16964055 in provider astro proc_lsst.multi INFO: found job 16964055 in provider astro 2024-03-09 15:01:43 proc_lsst.multi:201 [INFO] cancelling 16964055 on provider astro proc_lsst.multi INFO: cancelling 16964055 on provider astro 2024-03-09 15:01:44 proc_lsst.multi:146 [INFO] found job 16964061 in provider escience_a40 proc_lsst.multi INFO: found job 16964061 in provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:201 [INFO] cancelling 16964061 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964061 on provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:146 [INFO] found job 16964062 in provider escience_a40 proc_lsst.multi INFO: found job 16964062 in provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:201 [INFO] cancelling 16964062 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964062 on provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:146 [INFO] found job 16964063 in provider escience_a40 proc_lsst.multi INFO: found job 16964063 in provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:201 [INFO] cancelling 16964063 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964063 on provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:146 [INFO] found job 16964064 in provider escience_a40 proc_lsst.multi INFO: found job 16964064 in provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:201 [INFO] cancelling 16964064 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964064 on provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:146 [INFO] found job 16964065 in provider escience_a40 proc_lsst.multi INFO: found job 16964065 in provider escience_a40 2024-03-09 15:01:44 proc_lsst.multi:201 [INFO] cancelling 16964065 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964065 on provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:146 [INFO] found job 16964066 in provider escience_a40 proc_lsst.multi INFO: found job 16964066 in provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:201 [INFO] cancelling 16964066 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964066 on provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:146 [INFO] found job 16964067 in provider escience_a40 proc_lsst.multi INFO: found job 16964067 in provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:201 [INFO] cancelling 16964067 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964067 on provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:146 [INFO] found job 16964068 in provider escience_a40 proc_lsst.multi INFO: found job 16964068 in provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:201 [INFO] cancelling 16964068 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964068 on provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:146 [INFO] found job 16964069 in provider escience_a40 proc_lsst.multi INFO: found job 16964069 in provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:201 [INFO] cancelling 16964069 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964069 on provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:146 [INFO] found job 16964070 in provider escience_a40 proc_lsst.multi INFO: found job 16964070 in provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:201 [INFO] cancelling 16964070 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964070 on provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:146 [INFO] found job 16964071 in provider escience_a40 proc_lsst.multi INFO: found job 16964071 in provider escience_a40 2024-03-09 15:01:45 proc_lsst.multi:201 [INFO] cancelling 16964071 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964071 on provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:146 [INFO] found job 16964072 in provider escience_a40 proc_lsst.multi INFO: found job 16964072 in provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:201 [INFO] cancelling 16964072 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964072 on provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:146 [INFO] found job 16964073 in provider escience_a40 proc_lsst.multi INFO: found job 16964073 in provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:201 [INFO] cancelling 16964073 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964073 on provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:146 [INFO] found job 16964074 in provider escience_a40 proc_lsst.multi INFO: found job 16964074 in provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:201 [INFO] cancelling 16964074 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964074 on provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:146 [INFO] found job 16964075 in provider escience_a40 proc_lsst.multi INFO: found job 16964075 in provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:201 [INFO] cancelling 16964075 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964075 on provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:146 [INFO] found job 16964076 in provider escience_a40 proc_lsst.multi INFO: found job 16964076 in provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:201 [INFO] cancelling 16964076 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964076 on provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:146 [INFO] found job 16964077 in provider escience_a40 proc_lsst.multi INFO: found job 16964077 in provider escience_a40 2024-03-09 15:01:46 proc_lsst.multi:201 [INFO] cancelling 16964077 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964077 on provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:146 [INFO] found job 16964078 in provider escience_a40 proc_lsst.multi INFO: found job 16964078 in provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:201 [INFO] cancelling 16964078 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964078 on provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:146 [INFO] found job 16964079 in provider escience_a40 proc_lsst.multi INFO: found job 16964079 in provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:201 [INFO] cancelling 16964079 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964079 on provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:146 [INFO] found job 16964080 in provider escience_a40 proc_lsst.multi INFO: found job 16964080 in provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:201 [INFO] cancelling 16964080 on provider escience_a40 proc_lsst.multi INFO: cancelling 16964080 on provider escience_a40 2024-03-09 15:01:47 proc_lsst.multi:146 [INFO] found job 16964056 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964056 in provider escience_rtx6k 2024-03-09 15:01:47 proc_lsst.multi:201 [INFO] cancelling 16964056 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964056 on provider escience_rtx6k 2024-03-09 15:01:47 proc_lsst.multi:146 [INFO] found job 16964057 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964057 in provider escience_rtx6k 2024-03-09 15:01:47 proc_lsst.multi:201 [INFO] cancelling 16964057 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964057 on provider escience_rtx6k 2024-03-09 15:01:47 proc_lsst.multi:146 [INFO] found job 16964058 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964058 in provider escience_rtx6k 2024-03-09 15:01:47 proc_lsst.multi:201 [INFO] cancelling 16964058 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964058 on provider escience_rtx6k 2024-03-09 15:01:48 proc_lsst.multi:146 [INFO] found job 16964059 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964059 in provider escience_rtx6k 2024-03-09 15:01:48 proc_lsst.multi:201 [INFO] cancelling 16964059 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964059 on provider escience_rtx6k 2024-03-09 15:01:48 proc_lsst.multi:146 [INFO] found job 16964060 in provider escience_rtx6k proc_lsst.multi INFO: found job 16964060 in provider escience_rtx6k 2024-03-09 15:01:48 proc_lsst.multi:201 [INFO] cancelling 16964060 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16964060 on provider escience_rtx6k 2024-03-09 15:01:48 proc_lsst.multi:146 [INFO] found job 77954 in provider local proc_lsst.multi INFO: found job 77954 in provider local 2024-03-09 15:01:48 proc_lsst.multi:201 [INFO] cancelling 77954 on provider local proc_lsst.multi INFO: cancelling 77954 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 77954 parsl.providers.local.local WARNING: Failed to kill PID: 77954 and child processes on local 2024-03-09 15:01:49 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 790.1784 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190505_A1c_science#step0_20240309T144841Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 790.1905 seconds; current memory usage: 0.365 Gibyte, delta: 0.023 Gibyte, peak delta: 0.024 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 822.0906 seconds; current memory usage: 0.365 Gibyte, delta: 0.227 Gibyte, peak delta: 0.228 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.366 Gibyte (main), 0.366 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190505_A1c_science#step0_20240309T144841Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 13m46.002s user 5m2.934s sys 2m27.012s